search

holzschu / a-shell

A terminal for iOS, with multiple windows
BSD 3-Clause "New" or "Revised" License
2.6k stars 117 forks source link

Escaping for a-Shell is broken? #564

Open Emasoft opened 1 year ago

Emasoft commented 1 year ago

I've found a strange issue on a-Shell when using ffmpeg. This command:

ffmpeg -y -loglevel repeat+info -i file:output.m4a -map 0 -dn -ignore_unknown -vn -acodec copy -write_id3v1 1 -metadata 'title=You'"'"'re Going to Miss It - Harry Gregson-Williams' -metadata date=20170728 -metadata 'description=Music composed by Harry Gregson-Williams for the movie Spy Game.' -metadata 'synopsis=Music composed by Harry Gregson-Williams for the movie Spy Game.' -metadata 'purl=https://www.youtube.com/watch?v=5jgQy5tMQQ8' -metadata 'comment=https://www.youtube.com/watch?v=5jgQy5tMQQ8' -metadata artist=Prologue -movflags +faststart file:output.temp.m4a

Is not parsed correctly by a-Shell, and returns the following ERROR:

[~/Documents]$ ffmpeg -y -loglevel repeat+info -i file:output.m4a -map 0 -dn -ignore_unknown -vn -acodec copy -write_id3v1 1 -metadata 'title=You'"'"'re Going to Miss It - Harry Gregson-Williams' -metadata date=20170728 -metadata 'description=Music composed by Harry Gregson-Williams for the movie Spy Game.' -metadata 'synopsis=Music composed by Harry Gregson-Williams for the movie Spy Game.' -metadata 'purl=https://www.youtube.com/watch?v=5jgQy5tMQQ8' -metadata 'comment=https://www.youtube.com/watch?v=5jgQy5tMQQ8' -metadata artist=Prologue -movflags +faststart file:output.temp.m4a
ffmpeg version n5.1.2-11-g30d432f205 Copyright (c) 2000-2022 the FFmpeg developers
  built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  configuration: --cc=clang --arch=arm64 --disable-asm --enable-cross-compile --enable-pthreads --enable-videotoolbox --disable-audiotoolbox --enable-openssl --enable-libfreetype --enable-libzimg --enable-network --disable-debug --disable-gpl --disable-nonfree --enable-libmp3lame
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file:output.m4a':
  Metadata:
    major_brand     : M4A 
    minor_version   : 512
    compatible_brands: M4A isomiso2
    encoder         : Lavf59.27.100
  Duration: 00:09:15.21, start: 0.000000, bitrate: 126 kb/s
  Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
[NULL @ 0x13901f0e0] Unable to find a suitable output format for '"'
": Invalid argument
[~/Documents]$ 
[~/Documents]$

It seems the escaping is not parsed correctly by the shell. '"'"' is the correct escaping for ' on UNIX. But a-Shell parses it wrong, and " ends being left out.

Moreover: this bug in the parsing code for escaping characters can cause a-Shell to crash. Here is a command that ALWAYS crashes a-Shell (at least on my iPad):

ffmpeg -y -loglevel repeat+info -i file:output.m4a -map 0 -dn -ignore_unknown -vn -acodec copy -write_id3v1 1 -metadata 'title=You'"'"'re Going to Miss It - Harry Gregson-Williams' -movflags +faststart file:output.temp.m4a

And this command is just a shortened version of the one above.

Also, those kind of commands are usually emitted by youtube-dl and yt-dlp (this is where I got them) for postprocessing videos. And this bug seems to make both programs crash with ERROR. No such crash happens on my Debian machine. In other words, at the moment many youtube URLs (all those with titles that need escaping) cannot be passed to those programs in a-Shell (unless you disable the postprocessing options).

You can reproduce the issue in yt-dlp with the following command:

[~/Documents]$ yt-dlp -x "https://www.youtube.com/watch?app=desktop&v=5jgQy5tMQQ8" --add-metadata -f m4a -o output.mp4
[youtube] Extracting URL: https://www.youtube.com/watch?app=desktop&v=5jgQy5tMQQ8
[youtube] 5jgQy5tMQQ8: Downloading webpage
[youtube] 5jgQy5tMQQ8: Downloading android player API JSON
[youtube] 5jgQy5tMQQ8: Downloading MPD manifest
[info] 5jgQy5tMQQ8: Downloading 1 format(s): 140
[download] Destination: output.mp4
[download] 100% of    8.42MiB in 00:00:00 at 25.51MiB/s
[FixupM4a] Correcting container of "output.mp4"
[ExtractAudio] Not converting audio output.mp4; the file is already in a common audio format
[Metadata] Adding metadata to "output.mp4"
ERROR: Postprocessing: Going: Invalid argument
[~/Documents]
[~/Documents]
[~/Documents]

And if you need the verbose output:

[~/Documents]$ yt-dlp -vU -x "https://www.youtube.com/watch?app=desktop&v=5jgQy5tMQQ8" --add-metadata -f m4a -o output.mp4 
[debug] Command-line config: ['-vU', '-x', 'https://www.youtube.com/watch?app=desktop&v=5jgQy5tMQQ8', '--add-metadata', '-f', 'm4a', '-o', 'output.mp4']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c01] (pip)
[debug] Python 3.11.0 (CPython iPad8,11 64bit) - macOS-16.3.1-iPad8,11-arm-64bit (OpenSSL 1.1.1i  8 Dec 2020)
[debug] exe versions: ffmpeg n5.1.2-11-g30d432f205 (setts), ffprobe n5.1.2-11-g30d432f205, phantomjs 
broken, rtmpdump broken
[debug] Optional libraries: Cryptodome-3.15.0, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, web
sockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.02.17, Current version: 2023.02.17
yt-dlp is up to date (2023.02.17)
[youtube] Extracting URL: https://www.youtube.com/watch?app=desktop&v=5jgQy5tMQQ8
[youtube] 5jgQy5tMQQ8: Downloading webpage
[youtube] 5jgQy5tMQQ8: Downloading android player API JSON
[youtube] 5jgQy5tMQQ8: Downloading MPD manifest
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] 5jgQy5tMQQ8: Downloading 1 format(s): 140
[debug] Invoking http downloader on "https://rr1---sn-45nufxc-4jve.googlevideo.com/videoplayback?expire=1676999680&ei=oKf0Y72jDdCy1gK9zYHACQ&ip=2001%3Ab07%3A646e%3A5dca%3A2980%3Acfa0%3Af7d0%3A412a&id=o-AHSlh_FSkKKzr4GcZ-0xyysNiZhoqnqIm7ZRJrx9-OkS&itag=140&source=youtube&requiressl=yes&mh=Yw&mm=31%2C26&mn=sn-45nufxc-4jve%2Csn-hgn7rnls&ms=au%2Conr&mv=m&mvi=1&pl=51&initcwndbps=1357500&spc=H3gIhnhBotRV-MRblM1PQUOSDm8Wo1Y&vprv=1&svpuc=1&mime=audio%2Fmp4&gir=yes&clen=8825280&dur=555.212&lmt=1520923364209603&mt=1676977531&fvip=5&keepalive=yes&fexp=24007246&beids=24472384&c=ANDROID&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAKhHMcH3qDi2dBgGJiM-hMZMw0SRacZlmpUZRtURmiVvAiBkAHPgjytupySQes61Wv05cMo0ujC3WBhgJALhzWt-xQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAP1XiDCvicFI5r9KlQPYs4xuWRBuoOl31P9Cwwo1OH45AiEA0xZ-AxKigWNTEVl5S5Syy8khyf9G2YKGnypejC7AUn8%3D"
[download] output.mp4 has already been downloaded
[download] 100% of    8.40MiB
[debug] ffmpeg command line: ffprobe -show_streams file:output.mp4
[ExtractAudio] Destination: output.m4a
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i file:output.mp4 -vn -acodec copy -bsf:a aac_adtstoasc -movflags +faststart file:output.m4a
Deleting original file output.mp4 (pass -k to keep)
[Metadata] Adding metadata to "output.m4a"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i file:output.m4a -map 0 -dn -ignore_unknown -vn -acodec copy -write_id3v1 1 -metadata 'title=You'"'"'re Going to Miss It - Harry Gregson-Williams' -metadata date=20170728 -metadata 'description=Music composed by Harry Gregson-Williams for the movie Spy Game.' -metadata 'synopsis=Music composed by Harry Gregson-Williams for the movie Spy Game.' -metadata 'purl=https://www.youtube.com/watch?v=5jgQy5tMQQ8' -metadata 'comment=https://www.youtube.com/watch?v=5jgQy5tMQQ8' -metadata artist=Prologue -movflags +faststart file:output.temp.m4a
[debug] ffmpeg version n5.1.2-11-g30d432f205 Copyright (c) 2000-2022 the FFmpeg developers
  built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  configuration: --cc=clang --arch=arm64 --disable-asm --enable-cross-compile --enable-pthreads --enable-videotoolbox --disable-audiotoolbox --enable-openssl --enable-libfreetype --enable-libzimg --enable-network --disable-debug --disable-gpl --disable-nonfree --enable-libmp3lame
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file:output.m4a':
  Metadata:
    major_brand     : M4A 
    minor_version   : 512
    compatible_brands: M4A isomiso2
    encoder         : Lavf59.27.100
  Duration: 00:09:15.21, start: 0.000000, bitrate: 126 kb/s
  Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
[NULL @ 0x104ee0b30] Unable to find a suitable output format for 'Going'
Going: Invalid argument

ERROR: Postprocessing: Going: Invalid argument
Traceback (most recent call last):
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3307, in process_info
    replace_info_dict(self.post_process(dl_filename, info_dict, files_to_move))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3486, in post_process
    info = self.run_all_pps('post_process', info, additional_pps=info.get('__postprocessors'))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3468, in run_all_pps
    info = self.run_pp(pp, info)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/YoutubeDL.py", line 3446, in run_pp
    files_to_delete, infodict = pp.run(infodict)
                                ^^^^^^^^^^^^^^^^
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/postprocessor/common.py", line 24, in run
    ret = func(self, info, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/postprocessor/common.py", line 129, in wrapper
    return func(self, info)
           ^^^^^^^^^^^^^^^^
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/postprocessor/ffmpeg.py", line 705, in run
    self.run_ffmpeg_multiple_files(
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/postprocessor/ffmpeg.py", line 324, in run_ffmpeg_multiple_files
    return self.real_run_ffmpeg(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/var/mobile/Containers/Data/Application/B3257773-9D80-4169-8C86-EA5754C70197/Library/lib/python3.11/site-packages/yt_dlp/postprocessor/ffmpeg.py", line 362, in real_run_ffmpeg
    raise FFmpegPostProcessorError(stderr.strip().splitlines()[-1])
yt_dlp.postprocessor.ffmpeg.FFmpegPostProcessorError: Going: Invalid argument
[~/Documents]$
[~/Documents]$

What do you think?

holzschu commented 1 year ago

It's very difficult to parse all possible combinations of quotes. a-Shell uses a minimalist parser that goes from one quote to the next unescaped quote. In your case, 'title=You\'re Going to Miss It - Harry Gregson-Williams' would (very likely) work. Using a curly apostrophe, such as ‘ would also work.

If you really need the full command parsing with all possible quotes, you can start dash inside a-Shell. It has the full command line parsing.

I see at the end that these commands are created automatically by yt-dlp, so you don't necessarily have an easy way to change the way they are written. I'll see if I can change the way the quotes are being parsed, so closing quote + no espace after is not the end of the argument.

Emasoft commented 1 year ago

I just discovered that yt-dlp returns error if I try to specify the subtitles languages to embed in a video. The error from yt-dlp and ffmpeg is:

error ValueError('Wrong regex for subtitlelangs: *')

But the string I used as parameter was:

'subtitleslangs': 'en.*,it.*,fr.*,ja.*,zh.*,-live_chat'

I tried various combinations, but it seems that selecting the languages for the subtitles in yt-dlp always returns an error.

au5ton commented 1 year ago

This won't apply to this scenario, but for some scenarios with yt-dlp this limitation may avoided by using the argument --restrict-filenames which will convert something like 'SHOUTOUT TO MY COUSIN'"'"'S KID [asdfghjkl].mp4' to 'SHOUTOUT_TO_MY_COUSIN_S_KID-[asdfghjkl].mp4'