Add presets for hardware acceleration on Rockchip devices

[MarcA711]

I guess there is a considerable number of users that use Rockchip hardware. The rk3588 seems to be widely used and the new Home Assistant Green uses a rk3566.

There is a FFmpeg fork that supports hardware de- and encoding on Rockchip devices. A static binary is available from my repo.

Using this FFmpeg version, the presets for hardware encoding are:

"h264/rk": "-c:v h264_rkmpp_encoder -g 50 -bf 0",
"h265/rk": "-c:v hevc_rkmpp_encoder -g 50 -bf 0",

Is there any chance that go2rtc includes support for the Rockchip platform?

In any case, great project. Many thanks!

[AlexxIT]

Do you know if Hass developers uses forked ffmpeg version? These presets are useless, until most users will have forked ffmpeg version.

[MarcA711]

No, I don't know what ffmpeg version is used by hass. Yes, these presets will only work with the forked FFmpeg version.

Maybe the user can provide a static binary himself if he wants to use hw accel on rockchip. Or go2rtc can download a binary, if it detects rockchip hardware.

[AlexxIT]

Is this chip support only arm64 arch?

[MarcA711]

The bigger, modern Rockchip SoCs (RK3588/s and RK356X) support arm64. These are the most widespread SoCs as far as I know.

But there are also older SoCs that support armhf.

But I don't know for sure, if you need this information, I will do some research.

[AlexxIT]

I wander if this ffmpeg version can automatically use hardware decoder without choosing it manually.

[MarcA711]

Yes, it does

[AlexxIT]

Don't know about including this binary to go2rtc docker, because this project was closed due to license issues https://github.com/JeffyCN/FFmpeg

And there is some another project with rkmpp support: https://github.com/jjm2473/ffmpeg-rk

[MarcA711]

My binary was built from this project: https://github.com/hbiyik/FFmpeg/

It is actively maintained.

[StuartIanNaylor]

Don't know about including this binary to go2rtc docker, because this project was closed due to license issues

Its not that project its FFmpeg & Hevc and why Youtube is moving to AV1 a lot of current developement is dodging ffmpeg and focusing on gstreamer AV1. There will not be any official rockchip Fmpeg H264/H265 or any mainline inclusions unless the argument about licensing that I don't fully understand is resolved.

Its a bit of a problem as Rockchip historically is heavily based on ffmpeg, but all work is with gstreamer and av1 hence the only submission was a AV1 stateless codecs for rk3588 & Mediatec SoCs. To be honest I am as confused as anyone but the above was the proposed Rockchip ffmpeg and a few others have forked as the only work being done is those backporting to solve the obvious problems no ffmpeg causes but the above is jeffy.chen@rock-chips.com

A lot of the BSP images contain the above or forks of and they are being backported whilst ignoring this licence as problem as no ffmpeg h264/h254 is seen as a bigger problem. The decoders in the rk35xx are pretty good and can do up x16 simultaneous 1080p streams, or at least the one in the RK3588 can, but think they are the same.

[AlexxIT]

@MarcA711 now I see, you made special Frigate distrib with Rockchip support https://github.com/blakeblackshear/frigate/blob/dev/docker/rockchip/Dockerfile So FFmpeg inside Frigate will have hardware support for this chips.

How user can select right docker container version?

[NickM-27]

There are multiple frigate container variants, all listed at https://deploy-preview-6262--frigate-docs.netlify.app/frigate/installation#docker

[AlexxIT]

@NickM-27 I can see that Orange Pi 5 with RK3588S, has some NPU on board. Did Frigate try to work with it in any way?

[MarcA711]

Yes, I added support for Object Detection using the NPU.

[AlexxIT]

@MarcA711 I do some tests with your ffmpeg without any success. Maybe you can help me.

Welcome to Orange Pi 1.0.2 Bullseye with Linux 5.10.110-rockchip-rk3588

Test1

alexxit@orangepi5b:/tmp$ ./ffmpeg -i bbb.mp4 out.mp4
ffmpeg version e243e8d001-20231113 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.2.0 (crosstool-NG 1.25.0.232_c175b21)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-nonfree --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-openssl --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --disable-libdavs2 --enable-libdrm --enable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --disable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --disable-libvpx --enable-libwebp --enable-lv2 --enable-rkmpp --disable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --disable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lstdc++ -lgomp' --extra-version=20231113
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: mp41
    creation_time   : 1970-01-01T00:00:00.000000Z
    title           : Big Buck Bunny
    artist          : Blender Foundation
    composer        : Blender Foundation
    date            : 2008
    encoder         : Lavf52.14.0
  Duration: 00:09:56.46, start: 0.000000, bitrate: 867 kb/s
  Stream #0:0[0x1](und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(progressive), 320x180 [SAR 1:1 DAR 16:9], 702 kb/s, 24 fps, 24 tbr, 24 tbn (default)
    Metadata:
      creation_time   : 1970-01-01T00:00:00.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 159 kb/s (default)
    Metadata:
      creation_time   : 1970-01-01T00:00:00.000000Z
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
[h264_rkmpp_decoder @ 0x55a56d0ba0] Picture format is nv12.
Segmentation fault

Test2

alexxit@orangepi5b:/tmp$ ./ffmpeg -f lavfi -i testsrc2 -t 1 -c h264_rkmpp_encoder -f null -
ffmpeg version e243e8d001-20231113 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 13.2.0 (crosstool-NG 1.25.0.232_c175b21)
  configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-nonfree --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-openssl --enable-lzma --enable-fontconfig --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --disable-libdavs2 --enable-libdrm --enable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libass --enable-libbluray --disable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --disable-libvpx --enable-libwebp --enable-lv2 --enable-rkmpp --disable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --disable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lstdc++ -lgomp' --extra-version=20231113
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
Input #0, lavfi, from 'testsrc2':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, yuv420p, 320x240 [SAR 1:1 DAR 4:3], 25 fps, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_rkmpp_encoder))
Press [q] to stop, [?] for help
[h264_rkmpp_encoder @ 0x55916d3850] Picture format is yuv420p.
[h264_rkmpp_encoder @ 0x55916d3850] Rate Control mode is set to CBR
[h264_rkmpp_encoder @ 0x55916d3850] Bitrate Target/Min/Max is set to 6000000/5625000/6375000
[h264_rkmpp_encoder @ 0x55916d3850] Profile is set to HIGH
[h264_rkmpp_encoder @ 0x55916d3850] 8x8 Transform is enabled
[h264_rkmpp_encoder @ 0x55916d3850] Level is set to 0
[h264_rkmpp_encoder @ 0x55916d3850] Coder is set to CABAC
[h264_rkmpp_encoder @ 0x55916d3850] Quality Min/Max is set to 50%(Quant=30) / 100%(Quant=10)
Segmentation fault

[MarcA711]

You either need sudo or you need to add our user to some group (that I don't remember at the moment) in order to get access to the hardware.

Try it with sudo ./ffmpeg -i ...

[AlexxIT]

Leave it here

alexxit@orangepi5b:/tmp$ sudo ./ffmpeg -hide_banner -decoders | grep rkmpp
 V..... av1_rkmpp_decoder    av1 (rkmpp decoder ) (codec av1)
 V..... h263_rkmpp_decoder   h263 (rkmpp decoder ) (codec h263)
 V..... h264_rkmpp_decoder   h264 (rkmpp decoder ) (codec h264)
 V..... hevc_rkmpp_decoder   hevc (rkmpp decoder ) (codec hevc)
 V..... mpeg1_rkmpp_decoder  mpeg1 (rkmpp decoder ) (codec mpeg1video)
 V..... mpeg2_rkmpp_decoder  mpeg2 (rkmpp decoder ) (codec mpeg2video)
 V..... mpeg4_rkmpp_decoder  mpeg4 (rkmpp decoder ) (codec mpeg4)
 V..... vp8_rkmpp_decoder    vp8 (rkmpp decoder ) (codec vp8)
 V..... vp9_rkmpp_decoder    vp9 (rkmpp decoder ) (codec vp9)

alexxit@orangepi5b:/tmp$ sudo ./ffmpeg -hide_banner -encoders | grep rkmpp
 V..... h264_rkmpp_encoder   h264 (rkmpp encoder ) (codec h264)
 V..... hevc_rkmpp_encoder   hevc (rkmpp encoder ) (codec hevc)
 V..... vp8_rkmpp_encoder    vp8 (rkmpp encoder ) (codec vp8)

alexxit@orangepi5b:/tmp$ sudo ./ffmpeg -hide_banner -h encoder=h264_rkmpp_encoder
Encoder h264_rkmpp_encoder [h264 (rkmpp encoder )]:
    General capabilities: hardware 
    Threading capabilities: none
    Supported hardware devices: 
    Supported pixel formats: nv24 yuv444p nv16 yuv422p bgr24 yuyv422 uyvy422 bgra bgr0 nv12 yuv420p drm_prime
rkmpp_h264_encoder AVOptions:
  -rc_mode           <int>        E..V....... Set rate control mode (from 0 to 4) (default CBR)
     VBR             0            E..V.......
     CBR             1            E..V.......
     CQP             2            E..V.......
     AVBR            3            E..V.......
  -quality_min       <int>        E..V....... Minimum Quality (from 0 to 100) (default 50)
  -quality_max       <int>        E..V....... Maximum Quality (from 0 to 100) (default 100)
  -width             <int>        E..V....... scale to Width (from 0 to 4096) (default 0)
  -height            <int>        E..V....... scale to Height (from 0 to 4096) (default 0)
  -profile           <int>        E..V....... Set profile restrictions (from -1 to 100) (default high)
     baseline        66           E..V.......
     main            77           E..V.......
     high            100          E..V.......
  -level             <int>        E..V....... Compression Level (from -99 to 255) (default 0)
     1               10           E..V.......
     1.1             11           E..V.......
     1.2             12           E..V.......
     1.3             13           E..V.......
     2               20           E..V.......
     2.1             21           E..V.......
     2.2             22           E..V.......
     3               30           E..V.......
     3.1             31           E..V.......
     3.2             32           E..V.......
     4               40           E..V.......
     4.1             41           E..V.......
     4.2             42           E..V.......
     5               50           E..V.......
     5.1             51           E..V.......
     5.2             52           E..V.......
     6               60           E..V.......
     6.1             61           E..V.......
     6.2             62           E..V.......
  -coder             <int>        E..V....... Entropy coder type (from 0 to 1) (default cabac) (from 0 to 1) (default cabac)
     cavlc           0            E..V.......
     cabac           1            E..V.......
  -8x8dct            <boolean>    E..V....... High profile 8x8 transform. (default true)

alexxit@orangepi5b:/tmp$ sudo ./ffmpeg -hide_banner -h encoder=hevc_rkmpp_encoder
Encoder hevc_rkmpp_encoder [hevc (rkmpp encoder )]:
    General capabilities: hardware 
    Threading capabilities: none
    Supported hardware devices: 
    Supported pixel formats: nv24 yuv444p nv16 yuv422p bgr24 yuyv422 uyvy422 bgra bgr0 nv12 yuv420p drm_prime
rkmpp_hevc_encoder AVOptions:
  -rc_mode           <int>        E..V....... Set rate control mode (from 0 to 4) (default CBR)
     VBR             0            E..V.......
     CBR             1            E..V.......
     CQP             2            E..V.......
     AVBR            3            E..V.......
  -quality_min       <int>        E..V....... Minimum Quality (from 0 to 100) (default 50)
  -quality_max       <int>        E..V....... Maximum Quality (from 0 to 100) (default 100)
  -width             <int>        E..V....... scale to Width (from 0 to 4096) (default 0)
  -height            <int>        E..V....... scale to Height (from 0 to 4096) (default 0)
  -level             <int>        E..V....... Compression Level (from -99 to 255) (default 0)
     1               30           E..V.......
     2               60           E..V.......
     2.1             63           E..V.......
     3               90           E..V.......
     3.1             93           E..V.......
     4               120          E..V.......
     4.1             123          E..V.......
     5               150          E..V.......
     5.1             153          E..V.......
     5.2             156          E..V.......
     6               180          E..V.......
     6.1             183          E..V.......
     6.2             186          E..V.......
     8.5             255          E..V.......

[AlexxIT]

@MarcA711 you can try latest master version

[MarcA711]

Works fine! Thank you for implementing this!

One more thing, the encoders also support hardware scaling using the -width and -height properties like this: ffmpeg -i input.mp4 -c:v rkmpp_h264_encoder -width 1280 -height 720 out.mp4

[AlexxIT]

I have tested width/height param. Looks like it works fine via filters: https://github.com/AlexxIT/go2rtc/blob/eceb4a476f86dc979352f9af4a4deaab478d5279/internal/ffmpeg/ffmpeg_test.go#L154-L164

[MarcA711]

Yes, filters work, but it is worse this way. Using -vf "scale=-1:320" the scaling part is done in software and results in a higher CPU load. Using -width and -height scaling will be done in hardware (in the Raster Graphics Accelerator - RGA) and results in a lower CPU load.

[AlexxIT]

Are you sure? Have you compare result? I saw something about new size in logs. And it was from rkmpp encoder.

[MarcA711]

Yes, the difference is big:

$ time sudo ./ffmpeg -i test.mp4 -c:v h264_rkmpp_encoder -vf "scale=1920:1080" out.mp4
[... FFmpeg output...]

real    0m12.228s
user    0m0.000s
sys     0m0.023s

$ time sudo ./ffmpeg -i test.mp4 -c:v h264_rkmpp_encoder -width 1920 -height 1080 out1.mp4
[...]

real    0m2.538s
user    0m0.015s
sys     0m0.007s

So, transcoding and rescaling took 12.3s vs 2.5 seconds.

[MarcA711]

I saw something about new size in logs. And it was from rkmpp encoder.

Where did you see this? hbiyik/FFmpeg has two branches:

• encoder is kind of the stable branch
• in exp_refactor_all he reimplements everything including new features

Maybe what you read is true for the newer exp_refactor_all branch. My ffmpeg was built from encoder. I plan to build a binary from the exp_refactor_all branch within the next couple of days, then I can test.

[AlexxIT]

I don't have access to OrangePi any more. Will this work?

sudo ./ffmpeg -i test.mp4 -c:v h264_rkmpp_encoder -width:v 1920 -height:v 1080 out1.mp4

[MarcA711]

yes, this works fine.

[AlexxIT]

https://github.com/AlexxIT/go2rtc/releases/tag/v1.8.5