Trying to make RPi FFmpeg work on Ubuntu - has it been tried?

[Fredrum]

Nevermind! User error again! :)

[Fredrum]

Actually re-opening...did anyone make this work on Ubuntu-Rpi4b? If so, any advice?

[jc-kynesim]

Are you asking if my tree compiles on rpi ubuntu (The 'this' in the issue title is somewhat ambiguous)? I haven't tried. If you can characterise what fails I'll have a quick look.

[Fredrum]

HI can change the title when I know more what I'm asking! :)

My Setup: Linux pi-ubuntu 5.13.0-1026-raspi #28-Ubuntu SMP PREEMPT Mon Apr 25 15:22:19 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux OS: Ubuntu 21.10 aarch64

Actually after thinking a bit clearer this morning it seems like I'm either missing or the configure script is not finding EGL.

It builds fine using this config line, ./configure --enable-shared --disable-static --enable-sand --enable-v4l2-request --enable-libdrm --enable-libudev --enable-opengl --enable-vout-egl --enable-vout-drm

But the out devices are missing your egl one,

Enabled outdevs:
alsa                    oss                     v4l2
fbdev                   sdl2                    vout_drm
opengl                  sndio                   xv

I did installed these, sudo apt install libglfw3-dev libgles2-mesa-dev

Ubuntu comes with, OpenGL version string: 2.1 Mesa 21.2.6

And the raw ffmpeg seems to work as this runs ok: ./ffmpeg -no_cvt_hw -hwaccel drm -vcodec h264 -i ~/Videos/jellyfish-50-mbps-hd-h264.mkv -f null -

but this does not, since I have an EGL problem: ./ffmpeg -no_cvt_hw -hwaccel drm -vcodec h264 -i ~/Videos/jellyfish-50-mbps-hd-h264.mkv -f vout_egl /dev/null

Probably this does not actually have to do with ffmpeg as such it feels more like I've got something missning somewhere. So feel free to leave me to it. :)

EDIT: I just did sudo find -name *egl*.so* and the only matches I get are with a wayland egl.

./usr/lib/xorg/modules/libglamoregl.so
./usr/lib/aarch64-linux-gnu/libwayland-egl.so.1
./usr/lib/aarch64-linux-gnu/libwayland-egl.so.1.0.0
./usr/lib/aarch64-linux-gnu/libwayland-egl.so

I'm reading in the 'configure' script that,

vout_egl_outdev_deps="xlib"
vout_egl_outdev_select="epoxy"

Maybe Ubuntu Rpi is now also using Wayland as a default and that uses different files for its egl which are not identified by 'configure' ?

EDIT2: Sorry scratch that I can also find these,

./usr/lib/aarch64-linux-gnu/libEGL.so.1
./usr/lib/aarch64-linux-gnu/libEGL.so.1.1.0
./usr/lib/aarch64-linux-gnu/libEGL.so

I'm going to try this now, sudo apt install libx11-dev Nope, that's already installed.

So I guess maybe the question is, why is 'configure' not finding EGL?

[jc-kynesim]

I'd go though the headers included in libavdevice/egl_vout.c and see what is missing on your system. If none are missing then I've got a config problem. If I had to guess a missing package I'd look at libepoxy-dev (I'm guessing the name) as that's not a default package for most things. (Used by me 'cos it was used by the original author of the code I copied.)

[Fredrum]

I looked in egl_vout.c and compared the includes to the ones in the enabled drm_vout.c and the suspect differences were,

#include <epoxy/gl.h>
#include <epoxy/egl.h>
...
#include <X11/Xlib.h>
#include <X11/Xutil.h>

so I ran apt on libepoxy-dev but that was already installed and latest version.

One other actually was #include <stdatomic.h> but I'm assuming that comes with gnu c/c++.

libx11-dev was also already installed as I mentioned earlier.

Soo...not sure?

[jc-kynesim]

vout_egl requires xlib & epoxy in configure, xlib appears to require Xvlib.h which is in libxv-dev. Maybe my dependency was a little wide but as I've never had to add that manually myself I didn't notice. If you don't already know about it may I recommend apt-file as a stunningly useful utility for tracking down which apt package contains a file. (sudo apt install apt-file)

If that doesn't help, zip up and send me ffbuild/config.log from whatever your build dir is.

[Fredrum]

Thanks for the tip I think I used something like apt-file a while ago but then forgot about it again. libxv-dev was already installed on my system. The problem is that I don't know if I can find an output or record of why vout_egl got rejected? I also can't find where to read the rule that decides if its ok to build it?

Xvlib.h was found here, /usr/include/X11/extensions/Xvlib.h

config.h.gz config.log.gz

[jc-kynesim]

Add --enable-epoxy to your config. It is arguably a bug you need it (and I'll have a go at fixing it) as it should be a dependency not something you have to have set but I think that will fix your problem. Tip: when your custom config fails have a look at what I'm using in pi-util/conf_native.sh as that is what I use to configure stuff. ffmpeg configure is something I'm still confused by.

[jc-kynesim]

I've pushed a new configure script to dev/4.4/rpi_import_1 which shouldn't need the enable-epoxy or enable-libudev flags and should complain if it is missing something it needs for vout_egl.

[Fredrum]

Nice one thanks I'll try these steps out tonight! I think it makes sense to fix in the config like you said although for me I was going to write an install script for 'my' users to make it super easy. I'll let you know how it works out with --enable-epoxy or your new config.

[Fredrum]

Btw is the dev/4.4/rpi_import_1 your new main branch to use for your rpi-ffmpeg? Iv'e been using test/4.3.3/rpi_main so far.

[jc-kynesim]

dev/4.4 is used by Kodi and generally maintained in sync with test/4.3.2/rpi_main which is the Pi (debian) distribution. In this case I've just pushed to 4.4 so I can get your feedback before I commit the changes to the other branch.

[Fredrum]

Alright that got us a step further so that's great! But I now get this error message, [vout_egl @ 0xaaab12410620] Frame (format=0) not DRM_PRiME I'll add the whole chunk at the end here. This was doing both vout_egl on the desktop and vout_drm in the tty. Same message. I checked ffmpeg,

ldd ffmpeg | grep drm
libdrm.so.2 => /lib/aarch64-linux-gnu/libdrm.so.2 (0x0000ffffa14ec000)

ldd ffmpeg | grep dri
ldd ffmpeg | grep gbm

both returned empty but I'd have to check on my RaspiOS Bullseye to see if I should expect any of those.

I might have to let this simmer a bit and taking a look at libnraries attached on the RaspberryOS version and see if I can find some differences.

I my program at least it returns that setting up av codec context was successful. No errors from the below.

    codec_context->pix_fmt = AV_PIX_FMT_DRM_PRIME;  /// request a DRM frame

    // test
    codec_context->thread_count = 1;

    if (avcodec_open2(codec_context, av_codec, nullptr) < 0) {
        printf("Could not open codec\n");
        return 1;
    }

    /// Must have gotten drm_prime
    /// If you get 182 or other you are sourcing the wrong header file
    printf("Actual Fmt: %d\n", codec_context->pix_fmt); // 181

    if(codec_context->pix_fmt != AV_PIX_FMT_DRM_PRIME) {
        printf("Couldn't negoitiate AV_PIX_FMT_DRM_PRIME\n");
        return 1;
    }

FULL ERROR from the run.

built with gcc 11 (Ubuntu 11.2.0-7ubuntu2) configuration: --enable-shared --disable-static --enable-sand --enable-v4l2-request --enable-libdrm --enable-libudev --enable-opengl --enable-epoxy --enable-vout-egl --enable-vout-drm libavutil 56. 51.100 / 56. 51.100 libavcodec 58. 91.100 / 58. 91.100 libavformat 58. 45.100 / 58. 45.100 libavdevice 58. 10.100 / 58. 10.100 libavfilter 7. 85.100 / 7. 85.100 libswscale 5. 7.100 / 5. 7.100 libswresample 3. 7.100 / 3. 7.100 Input #0, matroska,webm, from '/home/fred/Videos/jellyfish-50-mbps-hd-h264.mkv': Metadata: encoder : libebml v1.2.0 + libmatroska v1.1.0 creation_time : 2016-02-06T04:00:55.000000Z Duration: 00:00:30.03, start: 0.000000, bitrate: 49901 kb/s Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default) Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native)) Press [q] to stop, [?] for help [vout_egl @ 0xaaab12410620] EGL version 1.4 Output #0, vout_egl, to '/dev/null': Metadata: encoder : Lavf58.45.100 Stream #0:0(eng): Video: wrapped_avframe, yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn, 29.97 tbc (default) Metadata: encoder : Lavc58.91.100 wrapped_avframe [vout_egl @ 0xaaab12410620] Frame (format=0) not DRM_PRiME av_interleaved_write_frame(): Invalid argument frame= 1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.03 bitrate=N/A speed=0.0429x
video:1kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown Conversion failed!

[jc-kynesim]

What was your command line? I see no evidence of accelerated decode there and the vout_egl/drm devices are very stupid such that they only accept DRM_PRIME frames (and the error says they aren't getting that). I'd be using something like: ./ffmpeg -no_cvt_hw -c:v h264_v4l2m2m -i ~/rpi/streams/jellyfish-3-mbps-hd-h264.mkv -f vout_egl /dev/null Maybe -v trace would shed light on what's being tried (or maybe hide it in too much debug)

[doesnotexist]

I successfully built dev/4.4/rpi_import_1 on the rpi ubuntu 22.04 (jammy) after installing libxv-dev libglfw3-dev libgles2-mesa-dev libdrm-dev libudev-dev . Although, libx11-dev was already installed pkg-config complained about no xlib until I installed the libxv-dev like you mentioned above. For configuring I did pi-util/conf_native.sh --noshared and then make -j8 -C out/arm64-jammy-4.4-static-rel

The problem I have now is finding a command line that works to confirm that the hardware accelerated decode is working. I tried: .out/arm64-jammy-4.4-static-rel/ffmpeg -no_cvt_hw -c:v h264_v4l2m2m -i ~/jellyfish-3-mbps-hd-h264.mkv vout_egl /dev/null

but that errors with

[NULL @ 0xaaab29c26ee0] Unable to find a suitable output format for 'vout_egl'
vout_egl: Invalid argument

analogous error if I change that to vout_drm

Not super familiar with ffmpeg or what the the various vout_* arguments mean but perhaps this could be something to do with the fact that Ubuntu is running gnome 3 on wayland? If you're testing on raspian I think they haven't moved to wayland yet.

Took a shot in the dark (removed -f vout and changed /dev/null to a mp4 output file) and didn't get too far, but it does appear the h264_v4l2m2m code is starting up before it fails: without -v trace it errors with

Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented

so I added -v trace to see if that might help (apologies for the large output, I truncated all the stuff prior to the [h264_v4l2m2m] point:

~/src/rpi-ffmpeg/out/arm64-jammy-4.4-static-rel$ ./ffmpeg -no_cvt_hw -c:v h264_v4l2m2m -i ~/jellyfish-3-mbps-hd-h264.mkv -v trace ~/test-out.mp4
ffmpeg version n4.4-128-g357c22f19e-rpi Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
  configuration: --prefix=/home/omar/src/rpi-ffmpeg/out/arm64-jammy-4.4-static-rel/install --libdir=/home/omar/src/rpi-ffmpeg/out/arm64-jammy-4.4-static-rel/install/lib/aarch64-linux-gnu --incdir=/home/omar/src/rpi-ffmpeg/out/arm64-jammy-4.4-static-rel/install/include/aarch64-linux-gnu --disable-stripping --disable-thumb --enable-v4l2-request --enable-libdrm --enable-vout-egl --enable-vout-drm --disable-shared --disable-mmal --enable-sand --extra-cflags='-ggdb ' --extra-cxxflags=' ' --extra-ldflags= --extra-libs= --extra-version=rpi
 ...
[h264_v4l2m2m @ 0xaaaaef1deb00] <<< v4l2_decode_init
[h264_v4l2m2m @ 0xaaaaef1deb00] level=40
[h264_v4l2m2m @ 0xaaaaef1deb00] no_cvt_hw so accepting pix_fmt 181 with codec internal hwaccel
[h264_v4l2m2m @ 0xaaaaef1deb00] Format drm_prime chosen by get_format().
[h264_v4l2m2m @ 0xaaaaef1deb00] avctx requested=0 (yuv420p) 1920x1080; get_format requested=181 (drm_prime)
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video1
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'uvcvideo' on card 'HD Pro Webcam C920' in splane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video0
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'uvcvideo' on card 'HD Pro Webcam C920' in splane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video31
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-codec' on card 'bcm2835-codec-encode_image' in mplane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video18
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-codec' on card 'bcm2835-codec-image_fx' in mplane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video23
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-isp' on card 'bcm2835-isp' in unknown mode
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video22
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video21
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video20
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video16
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-isp' on card 'bcm2835-isp' in unknown mode
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video12
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video15
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video14
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-isp' on card 'bcm2835-isp' in splane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video11
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] v4l2 output format not supported
[h264_v4l2m2m @ 0xaaaaef1deb00] probing device /dev/video10
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] Using device /dev/video10
[h264_v4l2m2m @ 0xaaaaef1deb00] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
[h264_v4l2m2m @ 0xaaaaef1deb00] requesting formats: output=H264 capture=YU12
[h264_v4l2m2m @ 0xaaaaef1deb00] output: H264 16 buffers initialized: 0000x0000, sizeimage 01620736, bytesperline 00000000
[h264_v4l2m2m @ 0xaaaaef1deb00] Driver 'bcm2835-codec': Quirks=0
[h264_v4l2m2m @ 0xaaaaef1deb00] check_size[0]: Stepwise: Min: 32x32 Max: 1920x1920, Step: 2x2
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> mpeg4 (native))
Press [q] to stop, [?] for help
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[h264_v4l2m2m @ 0xaaaaef1deb00] In PTS=0, DTS=-33, track=1, n=1
[h264_v4l2m2m @ 0xaaaaef1deb00] output set status 1074026002 (ON) OK
[h264_v4l2m2m @ 0xaaaaef1deb00] VIDIOC_DECODER_CMD start OK
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output empty
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output pre VIDIOC_QBUF: index 0, ts=0.000001 count=0
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output VIDIOC_QBUF: index 0, ts=0.000001 count=1
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=0, events=0x2, revents=0
    Last message repeated 1 times
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[h264_v4l2m2m @ 0xaaaaef1deb00] In PTS=100, DTS=0, track=2, n=2
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output ret=0, timeout=0, events=0x104, revents=0
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output pre VIDIOC_QBUF: index 1, ts=0.000002 count=1
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output VIDIOC_QBUF: index 1, ts=0.000002 count=2
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=0, events=0x2, revents=0
    Last message repeated 1 times
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[h264_v4l2m2m @ 0xaaaaef1deb00] In PTS=33, DTS=33, track=3, n=3
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output ret=0, timeout=0, events=0x104, revents=0
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output pre VIDIOC_QBUF: index 2, ts=0.000003 count=2
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output VIDIOC_QBUF: index 2, ts=0.000003 count=3
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=0, events=0x2, revents=0
    Last message repeated 1 times
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[h264_v4l2m2m @ 0xaaaaef1deb00] In PTS=67, DTS=67, track=4, n=4
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output ret=1, timeout=0, events=0x104, revents=0x104
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output ret=0, timeout=0, events=0x104, revents=0
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output pre VIDIOC_QBUF: index 0, ts=0.000004 count=2
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output VIDIOC_QBUF: index 0, ts=0.000004 count=3
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=0, events=0x2, revents=0
    Last message repeated 1 times
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[h264_v4l2m2m @ 0xaaaaef1deb00] In PTS=167, DTS=100, track=5, n=5
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output ret=1, timeout=0, events=0x104, revents=0x104
    Last message repeated 2 times
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output empty
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output pre VIDIOC_QBUF: index 0, ts=0.000005 count=0
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output VIDIOC_QBUF: index 0, ts=0.000005 count=1
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=0, events=0x2, revents=0
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=5, events=0x2, revents=0
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=5, events=0x2, revents=0
[h264_v4l2m2m @ 0xaaaaef1deb00] In PTS=133, DTS=133, track=6, n=6
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output ret=1, timeout=0, events=0x104, revents=0x104
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output empty
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output pre VIDIOC_QBUF: index 0, ts=0.000006 count=0
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output VIDIOC_QBUF: index 0, ts=0.000006 count=1
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=5, events=0x2, revents=0
    Last message repeated 1 times
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=5, events=0x2, revents=0
[h264_v4l2m2m @ 0xaaaaef1deb00] In PTS=267, DTS=167, track=7, n=7
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output ret=1, timeout=0, events=0x104, revents=0x104
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll output empty
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output pre VIDIOC_QBUF: index 0, ts=0.000007 count=0
[h264_v4l2m2m @ 0xaaaaef1deb00] --- output VIDIOC_QBUF: index 0, ts=0.000007 count=1
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=1, timeout=5, events=0x2, revents=0x2
[h264_v4l2m2m @ 0xaaaaef1deb00] Dq event 5
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 capture changed: alloc=0 (32x32) -> (1920x1088)
[h264_v4l2m2m @ 0xaaaaef1deb00] Source change: SAR: 1/1, wxh 1920x1088 crop 1920x1080 @ 0,0, reinit=1
[h264_v4l2m2m @ 0xaaaaef1deb00] reinit context
[h264_v4l2m2m @ 0xaaaaef1deb00] capture: YU12 20 buffers initialized: 1920x1088, sizeimage 03133440, bytesperline 00001920
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 0, ts=0.000000 count=1
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 1, ts=0.000000 count=2
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 2, ts=0.000000 count=3
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 3, ts=0.000000 count=4
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 4, ts=0.000000 count=5
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 5, ts=0.000000 count=6
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 6, ts=0.000000 count=7
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 7, ts=0.000000 count=8
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 8, ts=0.000000 count=9
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 9, ts=0.000000 count=10
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 10, ts=0.000000 count=11
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 11, ts=0.000000 count=12
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 12, ts=0.000000 count=13
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 13, ts=0.000000 count=14
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 14, ts=0.000000 count=15
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 15, ts=0.000000 count=16
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 16, ts=0.000000 count=17
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 17, ts=0.000000 count=18
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 18, ts=0.000000 count=19
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 19, ts=0.000000 count=20
[h264_v4l2m2m @ 0xaaaaef1deb00] capture set status 1074026002 (ON) OK
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=0, timeout=5, events=0x41, revents=0
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 poll capture ret=1, timeout=5, events=0x41, revents=0x41
[h264_v4l2m2m @ 0xaaaaef1deb00] Out PTS=0/-9223372036854775808, DTS=0, track=1, n=1
detected 4 logical cores
[graph 0 input from stream 0:0 @ 0xaaaaef205ec0] Setting 'video_size' to value '1920x1080'
[graph 0 input from stream 0:0 @ 0xaaaaef205ec0] Setting 'pix_fmt' to value '181'
[graph 0 input from stream 0:0 @ 0xaaaaef205ec0] Setting 'time_base' to value '1/1000'
[graph 0 input from stream 0:0 @ 0xaaaaef205ec0] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 0:0 @ 0xaaaaef205ec0] Setting 'frame_rate' to value '19001/634'
[graph 0 input from stream 0:0 @ 0xaaaaef205ec0] w:1920 h:1080 pixfmt:drm_prime tb:1/1000 fr:19001/634 sar:1/1
[format @ 0xaaaaef206810] Setting 'pix_fmts' to value 'yuv420p'
[auto_scaler_0 @ 0xaaaaef207ed0] Setting 'flags' to value 'bicubic'
[auto_scaler_0 @ 0xaaaaef207ed0] w:iw h:ih flags:'bicubic' interl:0
[format @ 0xaaaaef206810] auto-inserting filter 'auto_scaler_0' between the filter 'Parsed_null_0' and the filter 'format'
Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
[h264_v4l2m2m @ 0xaaaaef1deb00] capture: Buffer requeue
[h264_v4l2m2m @ 0xaaaaef1deb00] --- capture VIDIOC_QBUF: index 0, ts=0.000001 count=20
Error while processing the decoded data for stream #0:0
[AVIOContext @ 0xaaaaef1df380] Statistics: 0 seeks, 0 writeouts
[AVIOContext @ 0xaaaaef1e30d0] Statistics: 163840 bytes read, 0 seeks
[h264_v4l2m2m @ 0xaaaaef1deb00] <<< v4l2_decode_close
[h264_v4l2m2m @ 0xaaaaef1deb00] V4L2 Codec end
[h264_v4l2m2m @ 0xaaaaef1deb00] output set status 1074026003 (OFF) OK
[h264_v4l2m2m @ 0xaaaaef1deb00] capture set status 1074026003 (OFF) OK
V4L2 Context destroyed
[h264_v4l2m2m @ 0xaaaaef1deb00] >>> v4l2_decode_close: rv=0
Conversion failed!

[Fredrum]

I'll take a look at this again on Sat evening as I'm tied up until then. Iv'e been using these command lines, ./ffmpeg -no_cvt_hw -hwaccel drm -vcodec h264 -i ~/Videos/jellyfish-50-mbps-hd-h264.mkv -f vout_egl /dev/null that I must have gotten from earlier testing. Is -no_cvt_hw -hwaccel drm not telling it to do hw accell?

@doesnotexist Is it because you forgot the -f in-front of 'vout_egl /dev/null' ? I see youv'e got -j8 on your make what hardware are you running on?

[doesnotexist]

I built it on the raspberry pi 4 running ubuntu jammy, I probably shouldn't have done -j4 but I was just copy/pasting what was in the pi-utils/BUILD.txt

@Fredrum Thanks for catching that missing -f, that was it! So it appears to work now with this: .out/arm64-jammy-4.4-static-rel/ffmpeg -no_cvt_hw -c:v h264_v4l2m2m -i ~/jellyfish-3-mbps-hd-h264.mkv -f vout_egl /dev/null

Using vout_egl, displaying the decoded images still requires quite a bit of CPU from what I can tell. All 4 cores go to 100%. This doesn't seem to be the case if I do it with vout_drm from a regular (no X) terminal / tty. So I think that confirms it works at least the h264 part. Haven't tried the stateless hevc decoding that is more specific to this fork.

[Fredrum]

@jc-kynesim Arr... yes you're right I see now that in my old notes/posts I was using h264_v4l2m2mbut at some point for this round of testing that became h264! Ha that might mean it's already actually working fine!

[Fredrum]

Sorry for the multiple posts. @doesnotexist
It shouldn't take all that cpu with EGL. I get around 18-25% cpu in resource mon (which I think adds up all threads) and definitely nowhere near 100%. EDIT: Numbers from when I use Raspberry OS.

Maybe try adding -hwaccel drm to your command line?

[doesnotexist]

@Fredrum -hwaccel drm has no effect for me when using vout_egl from gnome/wayland. My guess is that the cpu spike is happening outside of ffmpeg's direct control and perhaps has more to do wayland or with the compositing and scaling (full screen in vout_drm I'm assuming does no scaling) and perhaps some extra copying of the decoded frames is happening over in the wayland side of things. Watching bpytop while the video decodes, I never see the ffmpeg process uses more than 11% of cpu.

[jc-kynesim]

-hwaccel drm is what you need for h265 (V4L2 stateless), -c:v h264_v4l2m2m is what you need for H264 (V4L2 stateful). Note that with a simple -f vout_egl (or drm) ffmpeg will decode as fast as it can, dropping frames if display can't keep up and so you will get inflated (vs. real-time playback) CPU usage. If you want something a little closer to reality add-show_all 1 before the-f which means that all decoded frames should be displayed (i.e. you get 1 frame per vsync), but if your display rate doesn't match the video rate you will get fast/slow playback. Just for clarity I'll repeat my standard disclaimers: vout_drm/egl are test/example output devices, come as-is, and have known issues, any real-world usage is at your own risk.

[Fredrum]

Now I'm trying with, ./ffmpeg -no_cvt_hw -vcodec h264_v4l2m2m -i ~/Videos/jellyfish-50-mbps-hd-h264.mkv -f vout_egl /dev/null also tried ./ffmpeg -no_cvt_hw -hwaccel drm -vcodec h264_v4l2m2m -i ~/Videos/jellyfish-50-mbps-hd-h264.mkv -f vout_egl /dev/null

It's not complaining about a DRM Frame anymore but instead I get this, (It seems to do some work but no image or window appears)

Input #0, matroska,webm, from '/home/fred/Videos/jellyfish-50-mbps-hd-h264.mkv':
  Metadata:
    encoder         : libebml v1.2.0 + libmatroska v1.1.0
    creation_time   : 2016-02-06T04:00:55.000000Z
  Duration: 00:00:30.03, start: 0.000000, bitrate: 49901 kb/s
    Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 1k tbn, 59.94 tbc (default)
[h264_v4l2m2m @ 0xaaab051443d0] level=42
[h264_v4l2m2m @ 0xaaab051443d0] Using device /dev/video10
[h264_v4l2m2m @ 0xaaab051443d0] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
[h264_v4l2m2m @ 0xaaab051443d0] requesting formats: output=H264 capture=YU12
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0xaaab051443d0] V4L2 capture poll unexpected timeout: events=0x106
    Last message repeated 1 times

I run with -v trace and you can see the full output here,

https://pastebin.com/FMa2Nw4a

This opens a window with the video playing but is of course not the wanted pipeline, ./ffmpeg -vcodec h264 -i ~/Videos/jellyfish-50-mbps-hd-h264.mkv -f opengl /dev/null

EDIT: Hmm this returns empty, ldd ffmpeg | grep EGL I'm going to load up my raspi OS card and see if its the same. ...Nahhh just checked Rapsberrry OS's ffmpeg and its the same.

EDIT2: I just did ldd on ffmpeg on both OS's and the difference is ridiculous. Not sure if it is possible to use that. https://pastebin.com/H2snqjKw

[jc-kynesim]

Well the unexpected timeout thing is almost certainly 'cos the V4L2 stateful driver in the Ubuntu kernel (that is what you are using isn't it?) is somewhat behind the PiOS one. There was a bug in our driver some time ago where it didn't deal with the startup sequence correctly, and I'm pretty sure that what you are seeing here is that.

[Fredrum]

Ok yes I was starting to think that maybe there might be some kernel differences. Do you know what files/edits are involved or would it be better to ask that on the Raspberry OS kernel git issues area?

[jc-kynesim]

I could make educated guesses but you are probably better off asking. I don't know how Ubuntu & Pi deal with patches.

[Fredrum]

Hi I just overnight updated my SD card from Raspi Ubuntu 21.04->22.04 and now this command worked! ./ffmpeg -no_cvt_hw -vcodec h264_v4l2m2m -i ~/Videos/jellyfish-50-mbps-hd-h264.mkv -f vout_egl /dev/null

I haven't had time to test the kmsdrm or hevc commands yet as I had to rush but this seems promising!

[Fredrum]

Testing more now on my lunch hour! :)

h264 -> egl on Desktop worked good. h264 -> drm on tty worked good. but hevc -> drm on tty failed!

With -v trace I noticed it saying it couldn't open 'dmabufs'. It also reported hevc successfully decoding two frames.

So I tried to run the command with 'sudo' and then it worked! hevc jellyfish played on the tty.

Anyone's got an idea why that would be different? ls -lrt /dev/video* shows all devices having the same permissions. crw-rw----+ 1 root video 81, 0 Jun 1 08:17 /dev/video19

EDIT: There was errors opening these /dev/dma_heap/system /dev/dma_heap/linux,cma

ls -lrt /dev/dma_heap/*

crw-rw---- 1 root video 249, 0 Apr 21 05:42 /dev/dma_heap/system
crw-rw---- 1 root video 249, 1 Apr 21 05:42 /dev/dma_heap/linux,cma

EDIT2: Is this one of those things where devices need to be part of certain groups? I just learned about 'getfacl' (Get F**ck All? :) ) getfacl /dev/video19

# file: dev/video19
# owner: root
# group: video
user::rw-
user:fred:rw-
group::rw-
mask::rw-
other::--

getfacl /dev/dma_heap/system

# file: dev/dma_heap/system
# owner: root
# group: video
user::rw-
group::rw-
other::---

The second one is missing my name on the 'user:'.

EDIT3: Aha this made it work with out the need for super user.

sudo chown fred /dev/dma_heap/system
sudo chown fred /dev/dma_heap/linux,cma

Somewhat unfortunate that is needed but I don't know how to get that fixed/changed. Maybe it had something to do with me upgrading Ubuntu in place instead of flashing a fresh SD card. I have no idea.

UPDATE: When I re-did the process with a newly flashed Raspberry Ubuntu 22.04 SD card (as opposed to upgrade Ubuntu in place) I did not have the permissions problems.

[jc-kynesim]

linux,cma is the device that I use for getting dmabufs (I don't know if system needs better perms). It is possible that under unbuntu it has different default perms to PiOS where it has group video rw. Edit: Having actually read your post it seems more likely that you want o make user fred part of group video (i.e. people who want to see videos)

[jc-kynesim]

Maybe what you actually need is "sudo adduser fred video"?

[Fredrum]

Thanks! I will try that method instead on my next run! Now that's its clear that I can get this to work on RPi Ubuntu I will make my 'ffmpeg' install script for the Chiaki repo and then flash a fresh Ubuntu SD card to check all the steps.