Trying to get rpi-ffmpeg working under pure Wayland, no Xlib (almost there!)

[vanfanel]

Hi there,

I could FINALLY get hw-accelerated H264 decoding on the Raspberry Pi 4 with this ffmpeg fork, after years of trying. Yay!

So far, I have it working on the TTY, using drm video output like this (CPU usage with a 1080p clip is ~8%, incredible stuff) :

ffmpeg -no_cvt_hw -c:v h264_v4l2m2m -i video.mkv -f vout_drm /dev/null

However, obviously this won't run from a terminal emulator inside a Wayland compositor (no XWayland, no X-related libs are installed or needed on the system).

I had to build rpi-ffmpeg with this configuration:

./configure --disable-network --disable-debug --disable-muxers --disable-indevs \
--disable-outdev=fbdev --disable-outdev=oss --disable-doc --disable-bsfs --disable-ffprobe --disable-sdl2 \
--disable-stripping \
--disable-thumb \
--disable-mmal \
--enable-sand \
--enable-v4l2-request \
--enable-libdrm \
--enable-epoxy \
--enable-libudev \
--enable-vout-drm

In other words, I had to build it without --enable-vout-egl because --enable-vout-egl needs Xlib, wich is part of the X11 libs, hence not present here (remember, I am trying to make it work under a pure Wayland enviroment).

After looking at libavdevice/egl_vout.c, it seems that only an X11 EGL context is created there: no Wayland EGL context is created, there's no code for it.

So, what would be next? Should it be possible to have, let's say, mpv play accelerated H264 content using this libffmpeg build? Or, even if using mpv, it would require the creation of a Wayland EGL context in libavdevice/egl_vout.c?

My understanding is that mpv should take care of the Wayland context, while rpi-ffmpeg would provide the hardware decoding via v4l2m2m as in the vout_drm example I copied before, right?

Thanks!

[jc-kynesim]

I am working on this. If you happen to have a bit of example code lying around that gets me a wayland window and an egl context to draw into then I'll be happy to use it. Or better (though I believe more complex) some, not too involved, example code that takes YUV dmabufs and plonks them on the screen then that would be great. There are various places this can be nicked from (gstreamer, vlc probably) but their window handling is a bunch more involved than I'd like for a simple example driver.

[vanfanel]

I am working on this. If you happen to have a bit of example code lying around that gets me a wayland window and an egl context to draw into then I'll be happy to use it. Or better (though I believe more complex) some, not too involved, example code that takes YUV dmabufs and plonks them on the screen then that would be great. There are various places this can be nicked from (gstreamer, vlc probably) but their window handling is a bunch more involved than I'd like for a simple example driver.

Great to know you're working on this, Wayland is the way to go :) I have never worked on a Wayland context initialization, but maybe take a look at RetroArch Wayland backend, which I believe is pretty simple (and the context creation is very well isolated): https://github.com/libretro/RetroArch/blob/master/gfx/drivers_context/wayland_ctx.c

Also, there's SDL2 Wayland context creation, which seems a bit more complicated because it manages video modes, etc: https://github.com/libsdl-org/SDL/blob/main/src/video/wayland/SDL_waylandvideo.c

Hope it helps!

BTW: Does your answer mean that MPV can't play accelerated video on Wayland until you complete the context creation on ffmpeg's side??

[EmperorPenguin18]

I am working on this. If you happen to have a bit of example code lying around that gets me a wayland window and an egl context to draw into then I'll be happy to use it. Or better (though I believe more complex) some, not too involved, example code that takes YUV dmabufs and plonks them on the screen then that would be great. There are various places this can be nicked from (gstreamer, vlc probably) but their window handling is a bunch more involved than I'd like for a simple example driver.

I think I did this in mpv? This stuff is pretty complicated so maybe my implementation is wrong but I think I have YUV dmabufs working with mpv under wayland.

[jc-kynesim]

Yeah - I've now got a hello_eglwayland working (no repo yet) by cobbling together someone elses easy wayland window example, the egl device and the player from another hello project. Direct wayland output is clearly doable (gstreamer waylandsink does it) but I've not got there yet.

[wyup]

Hello jc-kynesim, I'm a linux enthusiast. Is it necessary today to build mpv and rpi-ffmpeg to enjoy H264/HEVC HD/4k/10-bit acceleration?

Is your rpi-ffmpeg fork eventually merged into Raspi OS or other PI distrubutions? If not, why don't they patch support for acceleration on their own platform?

Thanks for your work

[popcornmix]

The ffmpeg tree here is used by RPiOS (and has been for years). There is no reason to build it yourself.

[jc-kynesim]

What he said...

[wyup]

What formats from these have hardware acceleration enabled as of today with mpv player (both regular ffmpeg and mpv packages? h264, h265 - HD, 4k, 10-bit, 4:2:0 SDR/HDR?

[jc-kynesim]

On a Pi4 H264 8-bit 4:2:0 up to hd via h264_v4l2m2m, H265 up to 4k 8 & 10-bit 4:2:0 via hwaccel drm (V4l2 stateless). If you are on a Pi3 and have licences for MPEG2 etc those should also work (no H265 on Pi3 in the current world).

Edit: Sorry - no idea about mpv but that is what my ffmpeg supports

[wyup]

Good, thanks. It is because I've seen a github issue from mpv saying that mpv is built on/linked to ffmpeg, and that manual mpv and rpi-ffmpeg was necessary to have HEVC acceleration on RPi4, because Broadcom doesn't provide a hardware acceleration api that follows ffmpeg or so.

https://github.com/mpv-player/mpv/issues/10773#issuecomment-1288000201

BTW, is Wayland supported aswell?

[jc-kynesim]

Wayland is entirely mpvs problem in this scenario - ffmpeg only provides decode.

[jc-kynesim]

Having said that, and given that I do VLC for Pi as well I can comment that Wayland output sort of works - output via EGL works up to HD but you aren't going to get 4k output that way the Pi GL processing is just too slow, direct planar output is possible & arguably should work but I don't believe there is a current Wayland compositor that will just take the plane and send it to HVS which is what you need. I know there is work happening in this area so it may yet happen. Currently DRM output is the only way if you want working 4k.

[wyup]

I'm glad it's coming..

Is it true that Broadcom does not provide a standard API for hardware acceleration and that it requires low level work to make it happen?

[jc-kynesim]

Depends exactly what you mean by that and there is a lot of history to work through. But on a Pi most of the h/w accel drivers are written by Pi for V4L2 which is definitely a linux standard.

[wyup]

GBM/V4L2 is what LibreELEC uses, isn't it? (I think they are drivers built into the kernel). Can you inspire from their work to make rpi-ffmpeg?

Also, is there any vulkan support for hardware decoding instead of GL? I think Libreelec builds for RPI4 include some vulkan drivers (and mpv has just implemented it on linux/windows and intel/amd/nvidia)

[jc-kynesim]

GBM is a memory allocation library, V4L2 is video stuff for linux, rpi-ffmpeg uses V4L2 for h/w decode. GL doesn't do video decode (AFAIK), vulkan decode is quite similar to V4L2 stateless (and VAAPI).

[popcornmix]

GBM/V4L2 is what LibreELEC uses, isn't it? (I think they are drivers built into the kernel). Can you inspire from their work to make rpi-ffmpeg?

I think you have this the wrong way round. LibreELEC uses this repo for the ffmpeg part of V4L2.

LibreELEC doesn't include vulkan, and no, vulkan won't help with video decode(*) (but it will draw triangles fast).

(*) Yes, I know it's possible to accelerate video with vulkan, but it won't be nearly as fast as using hardware that was designed specifically for video decoding, like the hevc and h264 blocks that you can use through V4L2.

[jc-kynesim]

@popcornmix I think Vulkan has invented a dedicated video decode protocol these days. I haven't really looked at it but it looked a bit like all the others from what I recall.

[popcornmix]

Yes, but not something we support (or have plans to).

[vanfanel]

Having said that, and given that I do VLC for Pi as well I can comment that Wayland output sort of works - output via EGL works up to HD but you aren't going to get 4k output that way the Pi GL processing is just too slow, direct planar output is possible & arguably should work but I don't believe there is a current Wayland compositor that will just take the plane and send it to HVS which is what you need. I know there is work happening in this area so it may yet happen. Currently DRM output is the only way if you want working 4k.

From what I understand, that means that, on the Pi4, optimal v4l2m2m performance right now is achieved using rpi-ffmpeg for decoding + mpv for on-screen display on KMS/DRM (ie: running MPV on the TTY console) and NOT on Wayland, right?

In other words, it means that there's currently a performance penalty of ffmpeg(v4l2m2m)+mpv on Wayland, right?

[jc-kynesim]

Anything that isn't direct output to HVS imposes a performance penalty for any video display. The only way to get direct HVS output at the moment is via DRM so assuming you want to use mpv and that has DRM as an output option - yes.

[vanfanel]

Ah, thanks. Well, for me only the mistery of the dropped frames with v4l2m2+display_resample remains... Happens the same on KMS/DRM and on Wayland.

[jc-kynesim]

I'm going to regret this but "display_resample"? I assume that's an mpv option?

[vanfanel]

I'm going to regret this but "display_resample"? I assume that's an mpv option?

Yes and no. The mpv option would be video-sync=display-resample, being display-resample a possible value (the best one IMHO as it makes video playback perfectly smooth).

I know ffmpeg and mpv are different projects, but on Wayland with rpi-ffmpeg + current mpv git code: -Using the ffmpeg v4l2m2 decoder (via hwdec=v4l2m2) and video-sync=display-resample causes continuously dropped frames while CPU usage ~15%. (Strange isn't it?) -Using the ffmpeg v4l2m2-copy decoder (via hwdec=v4l2m2-copy) and video-sync=display-resample there are no dropped frames for hours and CPU usage is ~50%.

This is a mistery to me because, as you can see, the slower decoder (hwdec=v4l2m2-copy) has no frame drops even if CPU usage is higher. For me, it works great as it is and I don't mind the higher CPU usage in hwdec=v4l2m2-copy, so it's just a curiosity.

[jc-kynesim]

What I was actually trying to ask was - "what does that option do?"

[popcornmix]

display-resample = sync video display to vsync and resample audio to fit (i.e. lets you play 23.97fps video at 24Hz)

I wouldn't think that should be useful on a Pi, as hdmi video and hdmi audio are clocked from same PLL so won't drift, and ntsc versions of modes are supported. It may be useful with a different audio (e.g. USB) device which could drift.

[jc-kynesim]

Ah! OK - dunno why that doesn't work unless it is something to do with the longer & somewhat variable latency through v4l2 than you get with s/w.

[vanfanel]

@popcornmix It does make a very noticeable difference on the Pi4 with mpv+rpi-ffmpeg: it clearly prevents micro-stuttering which is noticeable in horizontal panning scenes.

@jc-kynesim works with both v4l2m2 and with v4l2m2-copy, but causes continuous frame dropping with v4l2m2, which are both using v4l2 if I'm not mistaken.

[wyup]

Hello @popcornmix and @jc-kynesim . Is there now any Wayland compositor that takes the plane and competes with V4L2/DRM/KMS? I've read Wayland can work alongside KMS. Is it true that Vulkan with its dedicated video decode on top of Wayland will never catch up with V4L2 KMS in RPI4/5? I'm thinking of either LibreElec or RPiOS Kodi and mpv with libplacebo enabled ffmpeg.

[jc-kynesim]

If you have a single wayland surface then labwc + wlroots + libliftoff can be persuaded to "composite" that plane directly to display but there are so many caveats on getting that working correctly that I'm not going to suggest it as a currently viable solution. But having said that there is actually hope for something vaguely plausible within the next year. (You are still stuffed if you want HDR though)