h264_v4l2m2m: "Failed to set gop size: Invalid argument"

[domsson]

I'm trying to stream from a Raspberry Pi 4 via RMTP, using hardware acceleration through h264_v4l2m2m. With this fork of ffmpeg, hardware acceleration works. However, the following warning can be observed:

[h264_v4l2m2m @ 0x55d588da10] Failed to set gop size: Invalid argument

I assume h264_v4l2m2m doesn't know what to do with the -g argument? The issue can be observed even with a simple use-case as the following:

ffmpeg -i big_buck_bunny_720p_10mb.mp4 -c:v h264_v4l2m2m -c:a copy out.mp4

The warning doesn't seem to be a problem when encoding to a file; everything works out okay in that case. However, when streaming to a platform like Twitch or YouTube, those platforms won't accept or be able to process the stream. I suspect this has to do with the above error, for two reasons:

1. When replacing -c:v h264_v4l2m2m with -c:v libx264, the warning disappears and the platforms will process and display the stream (but the Pi will be on the brink of dying due to the lack of hardware acceleration)
2. Twitch and YouTube both explicitly require a keyframe interval of 2, and if I understand correctly, that is directly related to gop/ the -g argument

Observe the difference in what Twitch makes of the incoming RMTP stream when using libx264, vs. h264_v4l2m2m, with all other settings/ arguments being the same:

libx264 (stream online, but Pi burns up and drops frames)

h264_v4l2m2m (stream offline, but Pi is happy and doesn't drop frames)

Is there any chance h264_v4l2m2m will be able to set the gop / keyframes in the future? This would be huge, as it would enable direct streaming from a Raspberry Pi.

Thank you for your work on this fork.

[kobuki]

Just encountered this problem while testing the v42l interface. The inability to set the GOP size makes it impossible to properly chunk live stream like HLS, for example. The old OMX interface didn't have this issue, so the hardware is capable of setting it.

EDIT As it appears, there's a workaround that guarantees key frames at certain intervals. For my use case of emitting 6 second chunks for HLS, it's a good fit. However, there are no guarantees that the GOP size aligns with this setting. The option I used: -force_key_frames 'expr:gte(t,n_forced*6)'

[domsson]

Thanks for the hint with -force_key_frames, @kobuki - unfortunately, it doesn't seem to make a difference in my use-case. The gop warning still shows, the stream codec still isn't recognized on the receiving and and the stream doesn't show.

[kobuki]

Did you try -force_key_frames 'expr:gte(t,n_forced*2)' to produce key frames every other sec? You don't need a GOP size of 2, you actually need a GOP size of either 60 or 120 (every 2 seconds), depending on the frame rate.

[domsson]

Did you try -force_key_frames 'expr:gte(t,n_forced*2)' to produce key frames every other sec?

Yep, that's exactly what I used. Unfortunately it didn't work. Maybe the keyframes aren't the only issue. Checking Twitch insights shows that it doesn't recognize the audio or video codec, for example, as can be seen on the screenshot above. But not sure.

[kobuki]

All methods I used for our streaming solution produced output that's rendered fine in all major browsers (libx264, omx, v4l2), so I suggest comparing the output in a bitstream analyzer. For one, the raspi hw encoder doesn't (can't) use b-frames in the bitstream. Maybe that's a problem? In any case, it's probably better to ask directly on twitch communities.

[stakach]

also have this issue:

as above, when switching to libx264 everything works fine. with those errors it only streams the audio

this is the version installed

ffmpeg version 4.3.5-0+deb11u1+rpt2 Copyright (c) 2000-2022 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)

[spookyfirehorse]

gop size working

now h264_v4l2m2m is working to stream over rtsp

and gop size is working

on 4.3.6 and 5.1.2

bullseye

32 bit FFMPEG

apt build-dep

git clone -b test/4.3.6/main https://github.com/jc-kynesim/rpi-ffmpeg.git

./configure --prefix=/usr --libdir=/usr/lib/arm-linux-gnueabihf/neon/vfp --incdir=/usr/include/arm-linux-gnueabihf --extra-cflags=-I/usr/include/libdrm --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --enable-nonfree --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libpulse --enable-nonfree --enable-libfdk-aac --enable-libx265 --enable-version3 --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --enable-vulkan --arch=armv7l --enable-mmal --enable-avresample --disable-filter=resample --enable-libwavpack --enable-libv4l2 --enable-sand --enable-vout-drm --enable-vout-egl --enable-v4l2-request --enable-libudev --enable-rpi --enable-neon --enable-v4l2-request --enable-sand --enable-pocketsphinx --disable-static --enable-opencl --enable-libsrt --enable-librabbitmq --enable-version3

make -j4

sudo make -j4 install

streaming

ffmpeg -hwaccel drm -hwaccel_output_format drm_prime -flags low_delay -hide_banner -f v4l2 -i /dev/video0 -c:v h264_v4l2m2m -pix_fmt yuv420p -b:v 1500k -threads 4 -g 25 -f rtsp -rtsp_transport tcp rtsp://localhost:8554/mystream

cheers