Proper Way to Escape Special Character in a Variable

[zQueal]

What happened, and what did you expect to happen?

I'm using Elvish to accomplish a rather trivial feat--extract subtitles from video files using ffmpeg. I have something like this;

fd -e mkv -e mp4 -e avi | peach {|x| ffmpeg -y -i $x $x".srt" }

Which works about 90% of the time, however, if the filename has a special character in it such as ' or &, the command fails. I've done my best to search the website, but I can't find the proper way to avoid special characters breaking this command from executing correctly.

Output of "elvish -version"

0.21.0+official

Code of Conduct

• [X] I agree to follow Elvish's Code of Conduct.

[krader1961]

We're going to need more information. Elvish doesn't have the variable expansion problems of POSIX shells that require using double-quotes (or other mechanisms) to keep shell meta-characters, like spaces, in variable values from causing problems.

Here is a slightly hard to read example that illustrates that Elvish has no trouble passing values it reads from stdin that contain shell meta-characters to an external command:

> echo "abc def\nfgh'ijk\nlmn&opq" | peach {|x| bash -c 'echo "<$0>"' $x".srt"}
<fgh'ijk.srt>
<abc def.srt>
<lmn&opq.srt>

Notice that because peach processes its input in parallel that example echoed the values in a different order than the input to peach. You might want to simplify the problem, for debugging purposes, by using each rather than peach to avoid potential problems introduced by the peach parallelism. If that makes the problem disappear then that is a good indication you have some sort of race in your code.

What is the literal error you observe? What happens if you run var x = 'a&b'; ffmpeg -y -i $x $x".srt"? When I do that I get the expected ffmpeg error: Error opening input file a&b.

[zQueal]

What is the literal error you observe?

This:

❯ var x = 'a&b'; ffmpeg -y -i $x $x".srt"
ffmpeg version 2023-07-06-git-f00222e81f-full_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libvpl --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libcodec2 --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      58. 14.100 / 58. 14.100
  libavcodec     60. 22.100 / 60. 22.100
  libavformat    60. 10.100 / 60. 10.100
  libavdevice    60.  2.101 / 60.  2.101
  libavfilter     9.  8.102 /  9.  8.102
  libswscale      7.  3.100 /  7.  3.100
  libswresample   4. 11.100 /  4. 11.100
  libpostproc    57.  2.100 / 57.  2.100
[in#0 @ 000001961944b800] Error opening input: No such file or directory
Exception: ffmpeg exited with 1
  [tty 2]:1:16-39: var x = 'a&b'; ffmpeg -y -i $x $x".srt"

Also using peach was just for convenience--I have enough RAM to run 20-30 instances of ffmpeg to extract the subtitles, so doing it in parallel versus procedurally saves me a considerable amount of time.

[krader1961]

I have ffmpeg version 7.0.2 which includes an additional line telling me the file name which illustrates that a file name containing a special character is passed literally to the ffmpeg command:

[in#0 @ 0x600001e48100] Error opening input: No such file or directory
Error opening input file a&b.
Error opening input files: No such file or directory

That my example command produces a "no such file" error is, obviously, expected since you presumably don't have a file named a&b. What happens if you replace that file name with one that exists and does contain a special character? Can you update your ffmpeg? The version you're running is from a commit on 2023-07-04 which isn't ancient but there have been 5745 commits since then. I was also really asking what error you see from your script, not my example. Presumably it's the same "Error opening input: No such file or directory" but confirmation would be nice.

The suggestion to replace peach with each was meant only as a troubleshooting step to see if removing the parallelism changes the problem. It was not meant as a permanent change. Your script may accidentally be processing the same file more than once. Or perhaps there is a change to the current working directory. You wrote "I have something like this" which worries me. In your original problem statement is that the actual command that is failing or is the actual script doing something else?

[zQueal]

OK, so I've updated ffmpeg to the latest build essential ffmpeg version (2024-09-22-git-a577d313b2-essentials_build) and now I'm struggling to find an example where it fails now. It seems to even be able to process a file with ★ in its filename.

Looks like maybe this was ffmpeg's fault with shitty error messages.