external encoder: ffmpeg with video=true

[dhannyz]

I can't for the life of me get the input format correct for reading with ffmpeg. The following example in the documentation does not work for me.

  %external(process="ffmpeg -i pipe:0 -f avi pipe:1",video=true),
  "/tmp/test.avi", s)

Can I get a hint about how I should read in with ffmpeg? i have tried various combinations of input formats, avi demuxer, rawvideo, pcm_16le...

Also, is it possible to output.external with video, similarly to ho the external decoder works (just no pipe back in to liquidsoap) ? @smimram i've noticed you've been responsible for a lot of the video output stuff. Any hints?

[autonarcosis]

I haven't tested this since v1.3.3 but this hasn't worked for quite some time (I speculate it ever worked). From what I can remember, there's no way for ffmpeg to know what the input is and synchronize with it (specifically the synchronization). Use GStreamer instead. Vastly more complex but it works and is more stable than ffmpeg. The only consistent use of ffmpeg I've found is to use it to output to Icecast for vp8 video as the Icecast and GStreamer packages didn't properly support webm in Fedora.

[smimram]

I'd tend to agree with the GStreamer suggestion. The problem we have is that there is apparently no standard format for encoding video in raw format. We used to manage to get one working in video, but apparently it is not supported anymore (and we did not change anything in our code). If you have a suggestion for an easy supported way to encode raw video, I'd be glad to hear about it.

[autonarcosis]

I dug through my test scripts from last year and found this output that semi worked. The issue here is that it's Video Only, no audio tracks. Even though I did include the audio on the encoding side. From my testing it doesn't work properly currently, I can't test more because I am having issues with v1.3.7 and the liq lib scripts (currently the output.icecast doesn't function). Don't have six hours to figure out why just yet.

# Semi working raw video output to ffmpeg
output.external(%avi,"ffmpeg -re -f rawvideo -video_size 720x576 -pix_fmt rgba -i pipe:0 -f webm -content_type video/webm -c:v libvp
x -b:v 1500K -flags:v +global_header -cpu-used 0 -qmin 10 -qmax 42 -deadline realtime -quality realtime -auto-alt-ref 0 -c:a libvorb
is -flags:a +global_header icecast://source:yourpassword@www.yourdomaname.com:8000/test.webm",mksafe(input))

[dhannyz]

Hmm, that's a little unfortunate. I do like using ffmpeg for some tasks, for example, the frequency visualiser in this livestream https://www.youtube.com/watch?v=mNSTeO0Y_3M which has audio coming from my liquidsoap radio.

I had been trying with something like this on input ffmpeg -f avi -c:a pcm_s16le -ac 2 -ar 48000 -c:v rawvideo -framerate 30 -video_size 640x360 -i pipe:0 ... I might just have to ditch my ffmpeg stuff

[smimram]

The AVI header was broken, I just fixed it. Now I can at least use

output.external(%avi(), "ffplay -f avi -vcodec rawvideo -acodec pcm_s16le  pipe:0", s)

Don't have much time to try output.external right now, but please tell me if you make progress.

[smimram]

Ok, so the script

s = single("test.avi")
output.file(%external(process="ffmpeg -i pipe:0 -f avi pipe:1",video=true), "/tmp/test.avi", s)

results in

2019/05/22 13:28:36 [single_8310:3] "test.avi" is static, resolving once for all...
2019/05/22 13:28:36 [test(dot)avi:4] Content kind is {audio=2;video=1;midi=0}.
2019/05/22 13:28:36 [test(dot)avi:4] Activations changed: static=[/tmp/test(dot)avi:/tmp/test(dot)avi], dynamic=[].
2019/05/22 13:28:36 [decoder.gstreamer:4] Using GStreamer 1.14.4.
2019/05/22 13:28:36 [decoder.gstreamer:5] Decode A/V: true/true.
2019/05/22 13:28:36 [decoder.gstreamer:5] Gstreamer pipeline: filesrc location="test.avi" ! decodebin name=d d. ! queue ! audioconvert ! audioresample ! appsink max-buffers=10 drop=false sync=false name="audio_sink" caps="audio/x-raw,format=S16LE,layout=interleaved,channels=2,rate=44100" d. ! queue ! videoconvert ! videoscale add-borders=true ! videorate ! appsink name="video_sink" drop=false sync=false max-buffers=10 caps="video/x-raw,format=RGBA,width=320,height=240,framerate=25/1,pixel-aspect-ratio=1/1".
2019/05/22 13:28:36 [test(dot)avi:3] Prepared "test.avi" (RID 0).
2019/05/22 13:28:36 [/tmp/test(dot)avi:4] Activations changed: static=[/tmp/test(dot)avi], dynamic=[].
2019/05/22 13:28:36 [/tmp/test(dot)avi:4] Enabling caching mode: active source.
2019/05/22 13:28:36 [/tmp/test(dot)avi:3] Starting process: ffmpeg -i pipe:0 -f avi pipe:1
2019/05/22 13:28:36 [threads:3] Created thread "wallclock_main" (2 total).
2019/05/22 13:28:36 [clock:4] Main phase starts.
2019/05/22 13:28:36 [clock.wallclock_main:3] Streaming loop starts, synchronized with wallclock.
2019/05/22 13:28:37 [/tmp/test(dot)avi:5] stderr: ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-4)
  configuration: --disable-decoder=amrnb --disable-decoder=libopenjpeg --disable-libopencv --disable-outdev=sdl2 --disable-podpages --disable-sndio --disable-stripping --enable-libaom --enable-avfilter --enable-avresample --enable-gcrypt --enable-gnutls --enable-gpl --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libilbc --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-l
2019/05/22 13:28:37 [/tmp/test(dot)avi:5] stderr: ibx265 --enable-libxvid --enable-libzvbi --enable-nonfree --enable-opencl --enable-opengl --enable-postproc --enable-pthreads --enable-shared --enable-version3 --enable-libwebp --incdir=/usr/include/x86_64-linux-gnu --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --toolchain=hardened --enable-frei0r --enable-chromaprint --enable-libx264 --enable-libiec61883 --enable-libdc1394 --enable-vaapi --enable-libmfx --disable-altivec --shlibdir=/usr/lib/x86_64-linux-gnu
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100

2019/05/22 13:28:36 [/tmp/test(dot)avi:3] Closing process's stdin
2019/05/22 13:28:37 [clock.wallclock_main:2] Source /tmp/test(dot)avi failed while streaming: Process_handler.Finished!
2019/05/22 13:28:37 [clock.wallclock_main:3] Raised at file "tools/process_handler.ml", line 317, characters 34-48
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "tools/tutils.ml", line 80, characters 16-19
2019/05/22 13:28:37 [clock.wallclock_main:3] Re-raised at file "tools/tutils.ml", line 82, characters 33-40
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "tools/process_handler.ml", line 314, characters 16-202
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "encoder/external_encoder.ml", line 145, characters 8-87
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "tools/tutils.ml", line 80, characters 16-19
2019/05/22 13:28:37 [clock.wallclock_main:3] Re-raised at file "tools/tutils.ml", line 82, characters 33-40
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "encoder/external_encoder.ml", line 143, characters 4-239
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "outputs/output.ml", line 256, characters 10-21
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "outputs/output.ml", line 264, characters 35-47
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "outputs/output.ml", line 180, characters 6-22
2019/05/22 13:28:37 [clock.wallclock_main:3] Called from file "clock.ml", line 160, characters 17-25
2019/05/22 13:28:37 [/tmp/test(dot)avi:4] Activations changed: static=[], dynamic=[].
2019/05/22 13:28:37 [source:4] Source /tmp/test(dot)avi gets down.
2019/05/22 13:28:37 [/tmp/test(dot)avi:5] stderr: Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, avi, from 'pipe:0':
  Metadata:
    encoder         : Liquidsoap/1.4.0+scm (Unix; OCaml 4.07.1)
  Duration: 47721:51:31.80, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo, bgr24, 320x240, 25 fps, 25 tbr, 25 tbn, 25 tbc
    Stream #0:1: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, stereo, s16, 1411 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg4 (native))
  Stream #0:1 -> #0:1 (pcm_s16le (native) -> mp3 (libmp3lame))

2019/05/22 13:28:37 [/tmp/test(dot)avi:5] stderr: Output #0, avi, to 'pipe:1':
  Metadata:
    ISFT            : Lavf58.20.100
    Stream #0:0: Video: mpeg4 (FMP4 / 0x34504D46), yuv420p, 320x240, q=2-31, 200 kb/s, 25 fps, 25 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.35.100 mpeg4
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
    Stream #0:1: Audio: mp3 (libmp3lame) (U[0][0][0] / 0x0055), 44100 Hz, stereo, s16p
    Metadata:
      encoder         : Lavc58.35.100 libmp3lame

2019/05/22 13:28:37 [/tmp/test(dot)avi:5] stderr: frame=    0 fps=0.0 q=0.0 Lsize=       1kB time=00:00:00.00 bitrate=N/A speed=   0x    
video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used)

2019/05/22 13:28:36 [/tmp/test(dot)avi:3] Process exited with code 0
2019/05/22 13:28:36 [/tmp/test(dot)avi:3] Cleaning up process

So, it looks like Liquidsoap is shutting down the external encoder... @toots any idea why?

[toots]

The external encoder is shutdown at each end of track. If you want to avoid that you can use merge_tracks

[toots]

However, that is intended to make sure any needed header are added at track beginning. The error above seems like it shouldn't happen. I'll look at it.

[smimram]

Yes, here it's a single so I don't see why we should have tracks...

[toots]

single have tracks when they loop back on their file 🙂

[smimram]

Yes, of course, but here we are really at the very beginning of the file, far from a loop.

[autonarcosis]

Ah. I was going to point out that the issue was that this was in no way an AVI. Looks as if you figured that out by the time I woke up. I will give the update a try. With the header working correct I suspect things will fall into place.

@dhannyz

I figured out an ffmpeg visualization output last year. This is what I came up with. It's basic but you kind of get the idea. Not sure how stable it will be. I haven't tested it beyond maybe an hour or less.

#Super Fancy Streaming Visualizations
output.external(%wav,"ffmpeg -loglevel quiet -i pipe:0 -f webm -content_type 'video/webm' -filter_complex \"[0:a]avectorscope=s=360x
288,pad=720:576[vs]; [0:a]showspectrum=mode=separate:color=intensity:scale=cbrt:s=360x288[ss]; [0:a]showwaves=s=720x288:mode=line[sw
]; [vs][ss]overlay=w[bg]; [bg][sw]overlay=0:H-h,drawtext=fontfile=/usr/share/fonts/gnu-free/FreeSans.ttf:fontcolor=white:x=10:y=10:t
ext='Awesome Sauce'[out]\" -map \"[out]\" -map 0:a -c:v libvpx -b:v 1500K -flags:v +global_header -auto-alt-ref 0 -cpu-used 0 -qmin 
10 -qmax 42 -deadline realtime -quality realtime -c:a libvorbis -flags:a +global_header icecast://source:password@www.yerdomain.com:8000/test.webm",mksafe(mainsource))

[autonarcosis]

This works! FINALLY! A VP8 stream! And if I had $10k for an AVX512 cpu VP9. A couple million AV1! This is big as it finally goes from Theoras small framesize / quality to near DVD framesize / quality. GStreamer wouldn't work as some of the libshout's are compiled without webm support. Sadly the output.icecast(%external()) doesn't work. Same issue as what @smimram is reporting. Encodes nothing, immediately stops and locks up (have to hit Control-\ to stop Liquidsoap).

output.external(%avi,"ffmpeg -re -f avi -i pipe:0 -f webm -content_type video/webm -c:v libvpx -b:v 1500K -flags:v +global_header -
cpu-used 0 -qmin 10 -qmax 42 -deadline realtime -quality realtime -auto-alt-ref 0 -c:a libvorbis -flags:a +global_header icecast://s
ource:passwerd@www.blahblahblah.com:8000/test.webm",mksafe(input))

[dhannyz]

@autonarcosis awesome, ive been running my ffmpeg stream on a dedi without a GPU for about 5 days now, but was just a static video background.

until ffmpeg \
-thread_queue_size 512 -i https://stream.nightride.fm/nightride.m4a \
-thread_queue_size 512 -stream_loop -1 -i nr.mp4 \
-filter_complex \
"[0:a]showfreqs=s=960x500:mode=bar:win_func=blackman:win_size=w4096:averaging=0.88:ascale=lin:fscale=log:colors=white|white[base]; \
[base]split[topRight][bottomRightTemp]; \
[bottomRightTemp]vflip[bottomRight]; \
[topRight][bottomRight]vstack[rightTemp]; \
[rightTemp]split[right][leftTemp]; \
[leftTemp]hflip[left]; \
[left][right]hstack[final]; \
[1:v][final]overlay=0:H-(h-350),drawtext=fontfile=/usr/share/fonts/truetype/amigafonts/TopazPlus_a1200_v1.0.ttf:fontcolor=white:fontsize=40:x=10:y=10:textfile=/home/rekt/nightride.txt:reload=1[out]" \
-map "[out]" -map 0:a \
-c:v libx264 -x264opts keyint=60:no-scenecut -r 30 -preset medium -minrate 3000k -maxrate 6000k -bufsize 12000k -ar 44100 -c:a libfdk_aac -b:a 128k -bsf:a aac_adtstoasc \
-f flv rtmp://a.rtmp.youtube.com/live2/<token> ; do
        echo "restarting ffmpeg command..."
        sleep 2
done

But this now working on output.external is great! I'm building out a dedi with a GPU as we speak

[smimram]

@autonarcosis @dhannyz %external should be working now :)

[m4rcu5]

Hi,

I see that all the commits that are related to fix the %external encoder issue are in v1.4.3, however I am still able to reproduce the bug, that hangs liquidsoap when using the external encoder.

Test setup

#!/usr/bin/liquidsoap
video = noise()
audio = sine()

s = mux_audio(video, audio=audio)

codec = %external(process="ffmpeg -i pipe:0 -f avi pipe:1",video=true)

output.file(codec, "/tmp/test.avi", mksafe(s))

Console

2020/11/09 13:22:01 [gstreamer.loader:3] Loaded GStreamer 1.14.4 0
2020/11/09 13:22:01 [frame:3] Using 44100Hz audio, 25Hz video, 44100Hz master.
2020/11/09 13:22:01 [frame:3] Frame size must be a multiple of 1764 ticks = 1764 audio samples = 1 video samples.
2020/11/09 13:22:01 [frame:3] Targetting 'frame.duration': 0.04s = 1764 audio samples = 1764 ticks.
2020/11/09 13:22:01 [frame:3] Frames last 0.04s = 1764 audio samples = 1 video samples = 1764 ticks.
2020/11/09 13:22:01 [sandbox:3] Sandboxing disabled
2020/11/09 13:22:01 [video.converter:3] Using preferred video converter: gavl.
2020/11/09 13:22:01 [threads:4] Created thread "gstreamer_main_loop" (1 total).
2020/11/09 13:22:01 [audio.converter:3] Using samplerate converter: ffmpeg.
2020/11/09 13:22:01 [threads:4] Created thread "generic queue #1" (1 total).
2020/11/09 13:22:01 [threads:4] Created thread "generic queue #2" (2 total).
2020/11/09 13:22:01 [threads:4] Created thread "non-blocking queue #1" (3 total).
2020/11/09 13:22:01 [threads:4] Created thread "non-blocking queue #2" (4 total).
2020/11/09 13:22:01 [clock:4] Currently 1 clocks allocated.
2020/11/09 13:22:01 [clock.wallclock_main:4] Starting 1 sources...
2020/11/09 13:22:01 [source:4] Source output.file_9286 gets up.
2020/11/09 13:22:01 [source:4] Source mksafe gets up.
2020/11/09 13:22:01 [source:4] Source mux_9281 gets up.
2020/11/09 13:22:01 [mux_9281:4] Content kind is {audio=2;video=1;midi=0}.
2020/11/09 13:22:01 [source:4] Source noise_9275 gets up.
2020/11/09 13:22:01 [noise_9275:4] Content kind is {audio=0;video=1;midi=0}.
2020/11/09 13:22:01 [noise_9275:4] Activations changed: static=[mux_9281:mksafe:/tmp/test(dot)avi:/tmp/test(dot)avi], dynamic=[].
2020/11/09 13:22:01 [source:4] Source sine_9279 gets up.
2020/11/09 13:22:01 [sine_9279:4] Content kind is {audio=2;video=0;midi=0}.
2020/11/09 13:22:01 [sine_9279:4] Activations changed: static=[mux_9281:mksafe:/tmp/test(dot)avi:/tmp/test(dot)avi], dynamic=[].
2020/11/09 13:22:01 [mux_9281:4] Activations changed: static=[], dynamic=[mksafe:/tmp/test(dot)avi:/tmp/test(dot)avi].
2020/11/09 13:22:01 [source:4] Source safe_blank gets up.
2020/11/09 13:22:01 [safe_blank:4] Content kind is {audio=2;video=1;midi=0}.
2020/11/09 13:22:01 [safe_blank:4] Activations changed: static=[], dynamic=[mksafe:/tmp/test(dot)avi:/tmp/test(dot)avi].
2020/11/09 13:22:01 [mksafe:4] Activations changed: static=[/tmp/test(dot)avi:/tmp/test(dot)avi], dynamic=[].
2020/11/09 13:22:01 [/tmp/test(dot)avi:4] Activations changed: static=[/tmp/test(dot)avi], dynamic=[].
2020/11/09 13:22:01 [/tmp/test(dot)avi:4] Enabling caching mode: active source.
2020/11/09 13:22:01 [/tmp/test(dot)avi:3] Starting process
2020/11/09 13:22:01 [threads:4] Created thread "wallclock_main" (2 total).
2020/11/09 13:22:01 [clock:4] Main phase starts.
2020/11/09 13:22:01 [clock.wallclock_main:3] Streaming loop starts, synchronized with wallclock.
2020/11/09 13:22:01 [mksafe:3] Switch to mux_9281.
2020/11/09 13:22:01 [mux_9281:4] Activations changed: static=[mksafe:/tmp/test(dot)avi:/tmp/test(dot)avi], dynamic=[mksafe:/tmp/test(dot)avi:/tmp/test(dot)avi].
^C2020/11/09 13:22:09 [main:3] Shutdown started!
2020/11/09 13:22:09 [main:3] Waiting for main threads to terminate...
2020/11/09 13:22:09 [threads:4] Waiting for thread gstreamer_main_loop to shutdown
2020/11/09 13:22:09 [threads:4] Thread "gstreamer_main_loop" terminated (1 remaining).
2020/11/09 13:22:09 [threads:4] Waiting for thread wallclock_main to shutdown
2020/11/09 13:22:09 [/tmp/test(dot)avi:4] Activations changed: static=[], dynamic=[].
2020/11/09 13:22:09 [source:4] Source /tmp/test(dot)avi gets down.
2020/11/09 13:22:01 [/tmp/test(dot)avi:3] Closing process's stdin
2020/11/09 13:22:01 [/tmp/test(dot)avi:3] Process exited with code 0
2020/11/09 13:22:01 [/tmp/test(dot)avi:3] Cleaning up process
^\Quit

Using v1.4.3 (from liquidsoap apt) on Debian 10.6

[dikkedimi]

I'm wondering if this has anything to do with the syntax error the example from documentation gave me. I know encoding live video for youtube is a lot to ask of a Pi 3B+ , but I'd like to get it working anyway, and then decide if it's the way to go or not.