vf_scale_v4l2m2m: effective HWA scaling for raspberry-pi

[JohannesBe]

Dear Sir or Madam,

One of the avid users of the rpi-ffmpeg package is of course jellyfin. For quite some time, HWA using V4L2 on a RPI has been in the making. Currently it seems it kind of works, but performance is bad.

The reason for this seems to be a missing HWA scaling filter (vf_scale_v4l2m2m) in ffmpeg. After some digging it seems that a patch was created to add this filter to ffmpeg quite some time ago: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190903010230.96236-21-ffmpeg@tmm1.net/

Would it be possible to implement and/or review this patch, such that we can have a fully hardware-accelerated pipeline in jellyfin?

Thanks in advance!

Kindest regards, Johannes

[nyanmisaka]

I was told that the ISP on RPi4 cannot handle SAND30(10bit)=> SAND8(8bit) conversion but this filter indeed works for 8bit to 8bit scaling in https://getchannels.com/raspberry-pi/

[jc-kynesim]

Indeed the ISP can't handle SAND30, you need the HVS (though that only gets you RGB out) but that is harder to get at though I am working on that (via DRM rather than V4L2). I'll look at this patch when I have a bit of free time - it is something that we should have.

[Darkyere]

that seems great. I would find it usefull if HWA would get HWA encoding for H264. Its the only reason i am not using that on my RPI4 atm.

[nyanmisaka]

Indeed the ISP can't handle SAND30, you need the HVS (though that only gets you RGB out) but that is harder to get at though I am working on that (via DRM rather than V4L2). I'll look at this patch when I have a bit of free time - it is something that we should have.

That's great!

Since the decoder is using DRM, is it feasible to interop with the existing Vulkan filters in ffmpeg via drm-buf import/export functions to avoid extra copy-back?

https://github.com/FFmpeg/FFmpeg/blob/4fbf3c828b63a782bd6730427f376f10809b99a8/libavutil/hwcontext_vulkan.c#L347-L352

    /* Imports/exports */
    { VK_KHR_EXTERNAL_MEMORY_FD_EXTENSION_NAME,               FF_VK_EXT_EXTERNAL_FD_MEMORY     },
    { VK_EXT_EXTERNAL_MEMORY_DMA_BUF_EXTENSION_NAME,          FF_VK_EXT_EXTERNAL_DMABUF_MEMORY },
    { VK_EXT_IMAGE_DRM_FORMAT_MODIFIER_EXTENSION_NAME,        FF_VK_EXT_DRM_MODIFIER_FLAGS     },
    { VK_KHR_EXTERNAL_SEMAPHORE_FD_EXTENSION_NAME,            FF_VK_EXT_EXTERNAL_FD_SEM        },
    { VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME,             FF_VK_EXT_EXTERNAL_HOST_MEMORY   },

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15426

[jc-kynesim]

If there was Vulcan support on the Pi then that would work. But there isn't (if I'm wrong on that point I would be delighted).

[nyanmisaka]

If there was Vulcan support on the Pi then that would work. But there isn't (if I'm wrong on that point I would be delighted).

IIRC the v3dv vulkan 1.1 driver is supported on Pi 4. https://www.raspberrypi.com/news/vulkan-update-version-1-1-conformance-for-raspberry-pi-4/ https://www.phoronix.com/scan.php?page=search&q=V3DV

[chuma9615]

Hey there! I'm in real need of this, how can I help to see this implemented?

[noaho]

I don't know if this helps us? https://arstechnica.com/gadgets/2022/08/raspberry-pi-4-expands-3d-potential-with-vulkan-update/

[nyanmisaka]

Thank to @jc-kynesim, the v4l2 scaler seems to have progressed recently, where I found the SAND30/V4L2_PIX_FMT_NV12_10_COL128 aka 10bit.

Does this mean we can now convert from HEVC 10bit to H.264 8bit using hardware?

[jc-kynesim]

No - I'm afraid not - the V4L2 scaler is the ISP and that doesn't accept SAND30. The only bit of h/w on the board that can accept SAND30 is the HVS and that has 2 problems (1) it is already claimed by DRM and (2) it can only output RGB which I suspect doesn't help you.

[jc-kynesim]

Also beware - the heads of nearly all my branches currently have bust h/w encode - so don't use them right now if you need that. Fix coming today or tomorrow (with any luck).

[Superminaren]

Hello, what's the current status for this?

[EliteElectronic]

With hardware acceleration enabled, I found PGS Subtitles ONLY work on the Jellyfin Media Player APP Windows version. All others does not work! I also tried Jellyfin Media Player APP on Roku DOES NOT work. Why only Jellyfin Media Player APP Windows version work ??? !!! My main player is Jellyfin Media Player APP on Roku and all of my movies are Bluray rip with PGS Subtitles. Jellyfin team, please look into this and make hardware acceleration PGS Subtitles work everywhere ! Thank you !!!

[Arthedian]

@jc-kynesim Did anything change? Can I custom compile some branch which will enable H265 10bit decoding on RPI4?

[nyanmisaka]

You can decode it but not transcode it. Transcoding is much more compute intensive and the CPU of Pi 4 can not handle the scaling and pixel format conversion effectively.

[Arthedian]

I am not sure if I understand you correctly. Does that mean, that transcoding HEVC 10bit on RPI4 in Jellyfin won't ever be possible?

[nyanmisaka]

It will be insanely slow before this issue is resolved.

[jc-kynesim]

Nothing has changed recently. H265 h/w decode works fine (8 & 10 bit). However from the thread above it looks like you want transcode to H264 and there is no h/w path for 10bit to do that. This is a h/w limitation and so isn't going to be fixed because it can't be. If you are asking a different question then please clarify.

[Arthedian]

My usage is media server (Jellyfin), where I have HEVC HDR 10 bit files and I want to watch them in a browser which isn't compatible with HEVC. For me it would be ok to transcode from HEVC 10bit to h264 8bit.

[jc-kynesim]

Yup - so as above - cannot work. You can get quite close in h/w + s/w if you don't need more than HD, but if you want 4k then you are just doomed.

[nyanmisaka]

Better to use Pi 4 as a media player or file server instead of transcoding server. It performs well in both 4k and HDR when acting as a player such as LibreELEC.

[RicardoCst]

Sad that we are still at the point where we dont have support like AMD or Intel Quick Sync, I will assume this will forever be the case for ARM.