Feature request: enable/integrate rockchip hwaccel support

[dev-zero]

It would be really nice if the jellyfin-ffmpeg would support the rockchip hardware acceleration.

For 4.1 there's a maintained ffmpeg-fork here with additional patches it seems: https://github.com/rockchip-linux/ffmpeg/tree/rockchip/release/4.1 but basic rockchip acceleration support was already included via the RK-MPP in ffmpeg-3.

This library is needed to enable support: https://github.com/rockchip-linux/mpp

Communication with the kernel is then through a /dev/videoN device if I understood it correctly.

[nyanmisaka]

@HeyMeco I can't reproduce this on the RK3588. I'm afraid this is a hardware limitation on the RK356x and older chips. According to the datasheet, it only states H264 High Profile@Level 4.2 support. This specification cannot carry such a high luma sample rate: 3840x2160x24fps (199,065,600 > 133,693,440 = 522,240x256). Therefore my previous estimate of 4k@25 does not apply to this case.

Strictly speaking the HEVC encoder cannot do this too, as the value also exceeds the maximum value given in the datasheet (4096x4096x10=167,772,160), so the behavior can be undefined. I will check with Rockchip staffs and confirm with them.

[HeyMeco]

@nyanmisaka Strictly speaking the HEVC encoder cannot do this too, as the value also exceeds the maximum value given in the datasheet (4096x4096x10=167,772,160), so the behavior can be undefined. I will check with Rockchip staffs and confirm with them.

RK3568-MPP-Experience.md I'm collecting my results here. While H264 really should be kept at 1080p (above that its just a broken mess) HEVC is way more capable than I thought on this SoC (still not practical for above 1080P). I Always used Level 5.1 in my tests and got around 18fps in 4k and 60fps in 1080p.

Wish we could add custom resolutions and mbps to the Jellyfin WebUi

[CodeChenL]

@nyanmisaka Good morning, I'm trying to open the HDMI IN of the RK3588 with your jellfin-ffmpeg, but it reports an error:

rock@rock-5b:~$ sudo /lib/jellyfin-ffmpeg/ffmpeg -re -f v4l2 -i /dev/video0 -c:v hevc_rkmpp ./output.mp4
[sudo] password for rock: 
ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --enable-libv4l2 --arch=arm64 --cross-prefix=/usr/bin/aarch64-linux-gnu- --toolchain=hardened --enable-cross-compile --enable-rkmpp --enable-rkrga
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[video4linux2,v4l2 @ 0xaaaae15e9320] Not a video capture device.
/dev/video0: No such device
rock@rock-5b:~$ 

[nyanmisaka]

@CodeChenL Please apply these (avdevice/v4l2) patches and try again.

[CodeChenL]

@CodeChenL Please apply these (avdevice/v4l2) patches and try again.

Thank,i'll try it.

[nyanmisaka]

@HeyMeco

RK3568-MPP-Experience.md I'm collecting my results here. While H264 really should be kept at 1080p (above that its just a broken mess) HEVC is way more capable than I thought on this SoC (still not practical for above 1080P). I Always used Level 5.1 in my tests and got around 18fps in 4k and 60fps in 1080p.

Wish we could add custom resolutions and mbps to the Jellyfin WebUi

You can indeed set a maximum resolution and bitrate limit in Jellyfin Web.

[HeyMeco]

@nyanmisaka You can indeed set a maximum resolution and bitrate limit in Jellyfin Web.

Yes thank you, I just meant that setting those manually would be something I'd look forward to. To have 4k 20mbps for example or whatever someone thinks would be suitable for their HW encoder

[CodeChenL]

@CodeChenL Please apply these (avdevice/v4l2) patches and try again.

There have been some changes, but it's still not working

rock@rock-5b:~$ sudo /lib/jellyfin-ffmpeg/ffmpeg -re -f v4l2 -i /dev/video0 -c:v hevc_rkmpp ./output.mp4
ffmpeg version 6.0.1-Jellyfin Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --enable-libv4l2 --arch=arm64 --cross-prefix=/usr/bin/aarch64-linux-gnu- --toolchain=hardened --enable-cross-compile --enable-rkmpp --enable-rkrga
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[video4linux2,v4l2 @ 0xaaaad33d9320] ioctl(VIDIOC_G_INPUT): Inappropriate ioctl for device
/dev/video0: Inappropriate ioctl for device

[nyanmisaka]

@CodeChenL One more patch https://github.com/hbiyik/FFmpeg/commit/48d00fc6bfb3b1db3bd53eb9378616a520af0b0f.patch

[CodeChenL]

@CodeChenL One more patch https://github.com/hbiyik/FFmpeg/commit/48d00fc6bfb3b1db3bd53eb9378616a520af0b0f.patch

Just now I found this patch too, thanks for the help!

[CodeChenL]

@CodeChenL One more patch https://github.com/hbiyik/FFmpeg/commit/48d00fc6bfb3b1db3bd53eb9378616a520af0b0f.patch

It‘s work,thank for your help

[tandy-1000]

Has anyone tried this on the RK3399 yet? EDIT: I tried the Docker image, enabled with the following settings:

According to this datasheet: https://opensource.rock-chips.com/images/d/d7/Rockchip_RK3399_Datasheet_V2.1-20200323.pdf

But I get this error: [h264_rkmpp @ 0xaaaaee08e940] Failed to init MPP context: -1

[dmarkey]

Sorry to come late to the game, I've installed jellyfin-ffmpeg6_6.0.1-2-jammy_arm64.deb, is there any more configuration to enable the HW acceleration in the jellyfin UI needed or is it just magic

[HeyMeco]

@dmarkey Sorry to come late to the game, I've installed jellyfin-ffmpeg6_6.0.1-2-jammy_arm64.deb, is there any more configuration to enable the HW acceleration in the jellyfin UI needed or is it just magic

Rather use the Docker Run command above to get Jellyfin running until it gets pulled into the master. Then you can set the HW Accelerator to RKMPP (in the selector for VAAPI/QuickSyncVideo/...) and then it is "like magic"

I choose these as per the Datasheet of the SoC

[nyanmisaka]

@tandy-1000

But I get this error: [h264_rkmpp @ 0xaaaaee08e940] Failed to init MPP context: -1

MPP is only available in vendor BSP kernels. Not available in mainline.

[RicardoCst]

@nyanmisaka I was wondering (maybe off-topic) about amlogic, is there even hope? It seems they dont think linux support is necessary.

[tandy-1000]

@tandy-1000

But I get this error: [h264_rkmpp @ 0xaaaaee08e940] Failed to init MPP context: -1

MPP is only available in vendor BSP kernels. Not available in mainline.

Now I need to figure out how to do that on NixOS... EDIT: I wonder where I should go with the friendlyelec or radxa 6.1 kernel...

[HeyMeco]

@tandy-1000 I wonder where I should go with the friendlyelec or radxa 6.1 kernel...

There is also this Mainline6.7_RC1-Panthor-MPP-RGA option

[jjm2473]

Here's a docker image for testing. Feedback is welcome.

docker run -d \
 --name jellyfin \
 --privileged \
 --net=host \
 --restart=unless-stopped \
 --volume /path/to/config:/config \
 --volume /path/to/cache:/cache \
 --volume /path/to/media:/media \
 `for dev in dri dma_heap mali0 rga mpp_service \
    iep mpp-service vpu_service vpu-service \
    hevc_service hevc-service rkvdec rkvenc vepu h265e ; do \
   [ -e "/dev/$dev" ] && echo " --device /dev/$dev"; \
  done` \
 nyanmisaka/jellyfin:latest-rockchip

@nyanmisaka

I noticed that the GPU device path of RK3528 is /dev/mali instead of /dev/mali0 (this may be the case for all mali-400). It may need to be compatible.

[nyanmisaka]

I noticed that the GPU device path of RK3528 is /dev/mali instead of /dev/mali0 (this may be the case for all mali-400). It may need to be compatible.

Only RK3588 supports tone-mapping. Therefore this device is optional.

[nyanmisaka]

@RicardoCst

@nyanmisaka I was wondering (maybe off-topic) about amlogic, is there even hope? It seems they dont think linux support is necessary.

This depends on amlogic and SBC vendors. You can get a RK3588S board with decent performance at a very cost-effective price, but it is difficult to find a competing product using amlogic at a similar price.

[wnpllrzodiac]

Has anyone tried this on the RK3399 yet? EDIT: I tried the Docker image, enabled with the following settings: According to this datasheet: https://opensource.rock-chips.com/images/d/d7/Rockchip_RK3399_Datasheet_V2.1-20200323.pdf

But I get this error: [h264_rkmpp @ 0xaaaaee08e940] Failed to init MPP context: -1

you should make sure /dev/mpp_service and /dev/rga is available, it came from rockchip kernel source code.

[dmarkey]

Finding a few breakages in the web player when using that docker image.. complete UI stalls especially around playing x265 HVEC. Anyone seen similar?

[HeyMeco]

Finding a few breakages in the web player when using that docker image.. complete UI stalls especially around playing x265 HVEC. Anyone seen similar?

Are you trying x265 on the RK SoC's? That encoder can cripple modern consumer x86 CPU's let alone current RK CPU's. If you check with htop I would almost guarantee your cpu usage is at 100% and that's the reason. That's why getting HW Acceleration is the goal

[nyanmisaka]

nyanmisaka/ffmpeg-rockchip will be used to store the Wiki and my fork. @CodeChenL

[ikapo]

Is the nyanmisaka/jellyfin:latest-rockchip docker image supposed to work with x265 HEVC? I tried using it with the docker run command @nyanmisaka posted above but also got the [h264_rkmpp @ 0xaaaaee08e940] Failed to init MPP context: -1.

Will using the official orange pi image fix this? It uses BSP kernel (I'm currently on Armbian with Orange Pi 5). If not how do I go about using the Mainline6.7_RC1-Panthor-MPP-RGA option @HeyMeco commented? I'm available to test this and provide feedback.

Thank you everyone for the awesome work!

[nyanmisaka]

Is the nyanmisaka/jellyfin:latest-rockchip docker image supposed to work with x265 HEVC?

Sure.

I tried using it with the docker run command @nyanmisaka posted above but also got the [h264_rkmpp @ 0xaaaaee08e940] Failed to init MPP context: -1.

Will using the official orange pi image fix this? It uses BSP kernel (I'm currently on Armbian with Orange Pi 5). If not how do I go about using the Mainline6.7_RC1-Panthor-MPP-RGA option @HeyMeco commented? I'm available to test this and provide feedback.

Thank you everyone for the awesome work!

Currently, Rockchip do not have a driver for encoding video in the Linux mainline. MPP and RGA require the BSP kernel from Rockchip, so the docker image is supported using images from OrangePi, Armbian legacy or Ubuntu-Rockchip.

[CodeChenL]

nyanmisaka/ffmpeg-rockchip will be used to store the Wiki and my fork. @CodeChenL

Good work！

[ikapo]

@nyanmisaka Weird, I'm on Armbian bookworm with 5.10.160-legacy-rk35xx kernel (Orange pi 5), yet I still get this error: [hevc_rkmpp @ 0xaaab0839a150] Failed to init MPP context: -1 Stream mapping: Stream #0:0 -> #0:0 (hevc (hevc_rkmpp) -> h264 (h264_rkmpp)) Stream #0:1 -> #0:1 (aac (native) -> aac (libfdk_aac)) Error while opening decoder for input stream #0:0 : Generic error in an external library

Is there any other command I need to run other than the docker run one you posted? Do I need to install the 3D acceleration packages posted here? All the /dev/ devices are properly exposed to the container

[nyanmisaka]

@ikapo Can you try with sudo and --privileged?

sudo docker run -d \
 --name jellyfin \
 --privileged \
 --net=host \
 --restart=unless-stopped \
 --volume /path/to/config:/config \
 --volume /path/to/cache:/cache \
 --volume /path/to/media:/media \
 `for dev in dri dma_heap mali0 rga mpp_service \
    iep mpp-service vpu_service vpu-service \
    hevc_service hevc-service rkvdec rkvenc vepu h265e ; do \
   [ -e "/dev/$dev" ] && echo " --device /dev/$dev"; \
  done` \
 nyanmisaka/jellyfin:latest-rockchip

[ikapo]

@nyanmisaka It's working great now! Thank you!!!

[HeyMeco]

@nyanmisaka Only RK3588 supports tone-mapping. Therefore this device is optional.

I've got OpenCL working on Rk3568 / Mali G52 with the libmali-bifrost-g52-g2p0-x11-gbm_1.9-1_arm64.deb. But tonemapping doesn't work with anything besides RK3588?

Any idea if we can get tonemapping on other RK SoC's too?

I tried

/usr/lib/jellyfin-ffmpeg/ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime \
-i input_hdr10.mkv -vf scale_rkrga=format=p010:transpose=vflip, \
hwmap=derive_device=opencl,tonemap_opencl=format=nv12, \
hwmap=derive_device=rkmpp:reverse=1,format=drm_prime -f null -

and get

Unable to find a suitable output format for 'hwmap=derive_device=opencl,tonemap_opencl=format=nv12,'

[nyanmisaka]

@HeyMeco OpenCL expects P010 format but the RGA2 hardware on 356x and older cannot handle P010 output (RK_FORMAT_YCbCr_420_SP_10B). That's why only the 3588 with RGA3 hardware supports tone-mapping in OpenCL.

[RicardoCst]

@RicardoCst

@nyanmisaka I was wondering (maybe off-topic) about amlogic, is there even hope? It seems they dont think linux support is necessary.

This depends on amlogic and SBC vendors. You can get a RK3588S board with decent performance at a very cost-effective price, but it is difficult to find a competing product using amlogic at a similar price.

Yeah and even hw support, amlogic is very unfriendly in that regard for linux.

[mrodriguez3313]

Im kind of a novice with Linux. I was excited to see that HW accel on RK3588 boards was becoming a thing. I was referred here to check out how to run this. Im open to test things out if needed.

When I run uname -a on my system I get

Linux fulatower 5.10.160-legacy-rk35xx #1 SMP Tue Nov 28 02:45:16 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

This told me, this could be compatible with my device. I ran the docker command with sudo. and I believe this is working properly. I made sure to select "Rockchip MPP (RKMPP)" under Dashboard -> Playback -> Transcoding. I kept "h264, HEVC 10bit, vp9 10bit" enabled by default. I made sure to turn on "Enable HW encoding, allow encoding in HEVC format, allow encoding in AV1 format, and Enable Tone Mapping" I left everything else the same.

after I run htop, I can see that cpu usage is lower than it was before, but other than that idk what else I should be testing to confirm it is working.

[HeyMeco]

@mrodriguez3313 allow encoding in AV1 format

RK3588 can only decode AV1 and encode H264 & HEVC

[mrodriguez3313]

Got it. So even though the options are there, some options aren't supported. What else can I do to test that HWA is working on my machine?

[nyanmisaka]

As long as you can find *_rkmpp from the stream mapping in FFmpeg logs, it works.

Stream mapping:
  Stream #0:0 (hevc_rkmpp) -> setparams:default
  overlay_rkrga:default -> Stream #0:0 (hevc_rkmpp)
  Stream #0:1 -> #0:1 (copy)

[HeyMeco]

@nyanmisaka I finally got my hands on the RK3588 (Rock 5B) and I got everything working besides Tone-Mapping. For some reason I get 1-2 fps (if I even get it to play a few secs). I was using armbian legacy with this Mali OpenCL blob libmali-valhall-g610-g13p0-gbm_1.9-1_arm64.deb. Do you have a recommened image / kernel I can reference?

EDIT: This was a result of using a power supply with only 9v 2a, switching to 20v 3a fixed it.

[jepperc]

Just want to say this is amazing work 🚀

man. 29. jan. 2024 08.50 skrev Nyanmisaka @.***>:

Closed #34 https://github.com/jellyfin/jellyfin-ffmpeg/issues/34 as completed via #318 https://github.com/jellyfin/jellyfin-ffmpeg/pull/318.

— Reply to this email directly, view it on GitHub https://github.com/jellyfin/jellyfin-ffmpeg/issues/34#event-11623797678, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA7FVLADOTB3WXDSV43OBTYQ5IFFAVCNFSM4MGPHSRKU5DIOJSWCZC7NNSXTWQAEJEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW4OZRGE3DEMZXHE3TMNZY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[RicardoCst]

Good job! 👍 🥇

[HeyMeco]

From which Jellyfin version on can we expect to use the main docker image? As far as I can tell the WebUI branch hasn’t been updated yet to have the option available but I could be wrong

[nyanmisaka]

From which Jellyfin version on can we expect to use the main docker image? As far as I can tell the WebUI branch hasn’t been updated yet to have the option available but I could be wrong

JF 10.9

Merge order: ffmpeg -> server -> web

[Qwertylala1]

Transcoding HDR movies with tone mapping doesn't seem to work with the latest Rockchip 6.1.43 kernel. I have a test install of Armbian with the new kernel and transcoding with Jellyfin works with SDR movies. Everything else should be identical to my normal Armbian install with the old 5.10.160 kernel where HDR transcoding works great.

I have the latest version of the libmali drivers installed (v1.9-1-b9619b9).

Log.txt

[HeyMeco]

@Qwertylala1 maybe attach clinfo output too.

[Qwertylala1]

Of course, attached here.

clinfo.txt

[HeyMeco]

@Qwertylala1 latest driver requires DDK g21p0-01eac0 and 6.1 currently is reverted back to g18p0-01eac0 because g21 is broken with RK6.1 today. Maybe try with an older driver build before the g21 DDK requirement

[Qwertylala1]

Yes, that did it! I tried v1.9-1-2d267b0 from November since I know that I have used that version before without any problems and now transcoding works fine with HDR movies. Thank you!

[jwsky]

Yes, that did it! I tried v1.9-1-2d267b0 from November since I know that I have used that version before without any problems and now transcoding works fine with HDR movies. Thank you!

the same issue like yours,i'm using orange pi 5 plus,rk3588 @HeyMeco @Qwertylala1 ,i've already installed my lib to the correct version v1.9-1-2d267b0 and such HWA settings by the official tutorial,and the hwa works，but the tone mapping does not work.it's a dovi movie file ,the color seems a little like to purple and green.

i‘ve’ research this for days，but it still didn't work. It'll be so appreciate if you can have some time looking up this issue，thank you

i'm using the OS: 1,Official Images of ubuntu focal ,downloaded from the official website. not work 2,ubuntu 24 from https://github.com/Joshua-Riek/ubuntu-rockchip . not work

uname -a Linux ubuntu 6.1.0-1013-rockchip #13-Ubuntu SMP Thu May 9 21:20:56 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

log11.txt clinfo_output.txt

video file info:
Video Title: 4K HEVC HDR Codec: HEVC AVC: No Profile: Main 10 Level: 150 Resolution: 3840x2160 Aspect ratio: 16:9 Anamorphic: No Interlaced: No Framerate: 23.976025 Bitrate: 24998 kbps Bit depth: 10 bit Video range: HDR Video range type: DOVI DV title: DV Profile 5 DV version major: 1 DV version minor: 0 DV profile: 5 DV level: 6 DV rpu preset flag: 1 DV el preset flag: 0 DV bl preset flag: 1 DV bl signal compatibility id: 0 Pixel format: yuv420p10le Ref frames: 1

[nyanmisaka]

https://jellyfin.org/docs/general/administration/hardware-acceleration/#hardware-accelerated-tone-mapping

Rockchip RKMPP currently only support HDR10 and HLG tone-mapping.

@jwsky Dolby Vision metadata is not properly supported by RKMPP. Please open an issue there to let them know.

https://github.com/rockchip-linux/mpp/issues