Support H.264 and H.265 (HEVC) ffmpeg hardware decode/encode on OrangePi 5

artem-zinnatullin commented 1 year ago

Hi!

I'm trying to get ffmpeg -hwaccel drm to work so both decoding and encoding of H.264 and H.265 (HEVC) videos can happen with hardware acceleration on OrangePi 5 with its Mali G610 GPU.

References:

My understanding is that Mali G610 supports hardware accelerated encoding/decoding for H.265 and H.264, however I can't seem to get it fully working.

apt-get update
apt-get install ffmpeg
# This brings tons of packages including mali-g610-firmware

Now let's list relevant available HW decoders:

ffmpeg -decoders | grep -i rkmpp
 V..... av1_rkmpp            av1 (rkmpp) (codec av1)
 V..... h263_rkmpp           h263 (rkmpp) (codec h263)
 V..... h264_rkmpp           h264 (rkmpp) (codec h264)
 V..... hevc_rkmpp           hevc (rkmpp) (codec hevc)
 V..... mpeg1_rkmpp          mpeg1 (rkmpp) (codec mpeg1video)
 V..... mpeg2_rkmpp          mpeg2 (rkmpp) (codec mpeg2video)
 V..... mpeg4_rkmpp          mpeg4 (rkmpp) (codec mpeg4)
 V..... vp8_rkmpp            vp8 (rkmpp) (codec vp8)
 V..... vp9_rkmpp            vp9 (rkmpp) (codec vp9)

Now let's list relevant available HW encoders:

ffmpeg -encoders | grep -i omx
 V..... h264_omx             OpenMAX IL H.264 video encoder (codec h264)
 V..... mpeg4_omx            OpenMAX IL MPEG-4 video encoder (codec mpeg4)

So it appears we have HW H.265 (decoder only??) and HW H.264 decoder and encoder for Mali G610.

Let's try to convert a video from H.265 to H.264:

Input test video: https://test-videos.co.uk/bigbuckbunny/mp4-h265 (I use first 1080p 1MB video)

ffmpeg -hwaccel drm -hwaccel_device /dev/dri/renderD128 -i test-h265.mp4 -c:v h264_omx test-out-h264.mp4

# Some output omitted, here are important bits

# The input stream is H.265 HEVC
Stream #0:0(und): Video: hevc (hev1 / 0x31766568), yuv420p(progressive), 1920x1080, 834 kb/s

# Input decoder picked is HW accelerated and output encoder is also HW accelerated implementations
# otherwise it would've been (hevc (native) -> h264 (libx264)) where "native" and "libx264" indicates CPU implementations
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_rkmpp) -> h264 (h264_omx))

# !!! H.264 output encoder fails due to missing libOMX_Core and ffmpeg errors out !!!
[h264_omx @ 0xaaaac0d2c700] libOMX_Core.so not found
[h264_omx @ 0xaaaac0d2c700] libOmxCore.so not found
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

If I remove HW output encoder, it will successfully run on SW encoder (eating all CPU) and HW input decoder:

ffmpeg -hwaccel drm -hwaccel_device /dev/dri/renderD128 -i test-h265.mp4 test-h264.mp4

So the HW H.265 input decoder seems to work, but HW H.264 output encoder doesn't!

Now let's try to convert a video from H.264 to H.265

Input test video: https://test-videos.co.uk/bigbuckbunny/mp4-h264 (I use first 1080p 1MB video)

ffmpeg -hwaccel drm -hwaccel_device /dev/dri/renderD128 -i test-h264.mp4 -c:v libx265 test-out-h265.mp4

# Some output omitted, here are important bits

# The input stream is H.264
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 832 kb/s

# Input decoder picked is HW accelerated however output  encoder is SW because we don't seem to have HW output encoder built in this ffmpeg build
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_rkmpp) -> hevc (libx265))

# Conversion is very slow and eats all available CPU due to SF output encoding
frame=  167 fps=1.5 q=28.1 size=     256kB time=00:00:02.26 bitrate= 925.3kbits/s dup=0 drop=1 speed=0.02x

TL;TR: as of Joshua-Riek/ubuntu-rockchip:v1.19 HW input decoding of H.264 and H.265 (HEVC) works, but HW output encoding of H.264 and H.265 doesn't I think.

It seems that for H.264 HW output encoding we need libOMX_Core.so in the distro and it's unclear (to me) where can one get it
It seems that for H.265 (HEVC) HW output encoding we need ffmpeg built with that encoder support (another apt-get ppa for ffmpeg?)

HW decoding/encoding is essential for popular use case of OrangePi 5 boards — Frigate NVR and other similar cases when the board needs to efficiently encode/decode video streams.

Joshua-Riek commented 1 year ago

I don't think video encoding works with rkmpp provided by Rockchip. However, I could be wrong about this. My primary knowledge base is with the kernel and on the os level.

The source for ffmpeg can be found at the below ppa. https://launchpad.net/~liujianfeng1994/+archive/ubuntu/rockchip-multimedia/+packages

Joshua-Riek commented 1 year ago

I found some information that might be of some help. Also, some ffmpeg forks might have working HW encoding.

https://github.com/hbiyik/FFmpeg/tree/encoder https://github.com/jjm2473/ffmpeg-rk https://www.reddit.com/r/OrangePI/comments/14f725v/has_anybody_managed_to_build_ffmpeg_with_hw/

colornoise42 commented 1 year ago

Hi!

I'm trying to get ffmpeg -hwaccel drm to work so both decoding and encoding of H.264 and H.265 (HEVC) videos can happen with hardware acceleration on OrangePi 5 with its Mali G610 GPU.

HW H.265 and H.264 encoding now works with parameters: -c:v h264_rkmpp_encoder and -c:v hevc_rkmpp_encoder

StuartIanNaylor commented 11 months ago

HW H.265 and H.264 encoding now works with parameters: -c:v h264_rkmpp_encoder and -c:v hevc_rkmpp_encoder

@colornoise42 is that without adding further packages?

kaleb-adcellgroup commented 11 months ago

Can anyone advise on encode performance for a camera connected to OPi5+? System usage, 1080p60, 4k

seler69 commented 10 months ago

Hi! I'm trying to get ffmpeg -hwaccel drm to work so both decoding and encoding of H.264 and H.265 (HEVC) videos can happen with hardware acceleration on OrangePi 5 with its Mali G610 GPU.

HW H.265 and H.264 encoding now works with parameters: -c:v h264_rkmpp_encoder and -c:v hevc_rkmpp_encoder

i have bad experience with armbian and adding RK3588 - it didnt work so i have instaled Joshua Riek Ubuntu witk Rk3588 implementation

i get this root@confirm:/home/confirm# ffmpeg -codecs | grep h264 ffmpeg version 6.0-5+git230804.e243e8d001~j1 Copyright (c) 2000-2023 the FFmpeg developers built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04) configuration: --prefix=/usr --extra-version='5+git230804.e243e8d001~j1' --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-version3 --enable-rkmpp --enable-sdl2 --disable-sndio --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-shared libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 DEV.LS h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264_rkmpp_decoder h264 h264_v4l2m2m ) (encoders: h264_rkmpp_encoder libx264 libx264rgb h264_omx h264_v4l2m2m h264_vaapi )

but i can't configure properly frigate.yml

ffmpeg: hwaccel_args: -hwaccel drm -hwaccel_device -c:v h264_v4l2m2m h264_vaapi

please enlighten me!

StuartIanNaylor commented 10 months ago

Have a look at https://github.com/blakeblackshear/frigate/issues/6914#issuecomment-1607953573

seler69 commented 10 months ago

Have a look at blakeblackshear/frigate#6914 (comment)

root@confirm:/dev/dri# ls by-path card0 card1 renderD128 renderD129 i tried with his encoders and mine with renderD128 and renderD129 i'm stuck

here is my encoder dump `root@confirm:~# ffmpeg -codecs | grep rkmpp ffmpeg version 6.0-5+git230804.e243e8d001~j1 Copyright (c) 2000-2023 the FFmpeg developers built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04) configuration: --prefix=/usr --extra-version='5+git230804.e243e8d001~j1' --toolchain=hardened --libdir=/us r/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-strippin g --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enab le-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig -- enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-li bpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --e nable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enab le-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --ena ble-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --e nable-openal --enable-opencl --enable-opengl --enable-version3 --enable-rkmpp --enable-sdl2 --disable-sndio --enable-pocketsphinx --enable-librsvg --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chro maprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-shared libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 DEV.L. av1 Alliance for Open Media AV1 (decoders: av1_rkmpp_decoder libdav1d libaom-av1 av1 ) (encoders: libaom-av1 libsvtav1 ) DEV.L. h263 H.263 / H.263-1996, H.263+ / H.263-1998 / H.263 version 2 (decoders: h263_rkmpp _decoder h263 h263_v4l2m2m ) (encoders: h263 h263_v4l2m2m )

DEV.LS h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264_rkmpp_decoder h264 h2 64_v4l2m2m ) (encoders: h264_rkmpp_encoder libx264 libx264rgb h264_omx h264_v4l2m2m h264_vaapi )

DEV.L. hevc H.265 / HEVC (High Efficiency Video Coding) (decoders: hevc_rkmpp_decoder hevc hevc_v4l2m2m ) (encoders: hevc_rkmpp_encoder libx265 hevc_v4l2m2m hevc_vaapi )

DEV.L. mpeg1video MPEG-1 video (decoders: mpeg1_rkmpp_decoder mpeg1video mpeg1_v4l2m2m )

DEV.L. mpeg2video MPEG-2 video (decoders: mpeg2_rkmpp_decoder mpeg2video mpegvideo mpeg2_v4l2m2m ) (encoders: mpeg2video mpeg2_vaapi )

DEV.L. mpeg4 MPEG-4 part 2 (decoders: mpeg4_rkmpp_decoder mpeg4 mpeg4_v4l2m2m ) (encoders: m peg4 libxvid mpeg4_omx mpeg4_v4l2m2m )

DEV.L. vp8 On2 VP8 (decoders: vp8_rkmpp_decoder vp8 vp8_v4l2m2m libvpx ) (encoders: vp8_rk mpp_encoder libvpx vp8_v4l2m2m vp8_vaapi )

DEV.L. vp9 Google VP9 (decoders: vp9_rkmpp_decoder vp9 vp9_v4l2m2m libvpx-vp9 ) (encoders: libvpx-vp9 vp9_vaapi )`

so it look like @artem-zinnatullin

log from frigate "ERROR : Unknown decoder 'h264_rkmpp'" but its not working

StuartIanNaylor commented 10 months ago

So if you ffmpeg -decoders | grep rkmpp you get nothing ? If not then likely better asking on https://github.com/blakeblackshear/frigate/issues/6914#issuecomment-1607953573

Joshua-Riek / ubuntu-rockchip

Support H.264 and H.265 (HEVC) ffmpeg hardware decode/encode on OrangePi 5 #246