Can this branch do drm prime and AV_PIX_FMT_DRM_PRIME pix_fmt on rk3588 (Orange Pi 5)

[Fredrum]

Hello!!

I'm trying to build a zero copy hevc decode pipeline in ffmpeg to run on an Orange Pi 5 but the distro bundled (Armbian + mpp additions) ffmpeg does not seem to work with AV_PIX_FMT_DRM_PRIME as I'm used to on an Rasberry Pi 4.

I can HW decode using _avcodec_find_decoder_by_name("hevcrkmpp"); but I then have to co through a conversion to display and when doing a codec_context->pix_fmt = AV_PIX_FMT_DRM_PRIME; // for zero copy video path it doesn't work and I get a pix_fmt code of 0 when it needs to be 181.

Thats using the libavcode-dev that comes with the distro.

Can this be made to work using your ffmpeg? With this kernel, Linux orangepi5 5.10.110-rockchip-rk3588 #23.02.2 SMP Fri Feb 17 23:59:20 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Cheers and thank you for your work!

[JeffyCN]

the drm prime format was tested with ffmpeg and mpv(drm vo, in my github)

you might need to do some hw format setup to get drm prime format, you can take mpv as an example

[Fredrum]

Thank you very much for your reply! I'll take a look at the code you suggest.

I forgot to mention that I also used drm prime for zero copy to GL like this:

const EGLImage image = eglCreateImageKHR( egl_display,
                                              EGL_NO_CONTEXT,
                                              EGL_LINUX_DMA_BUF_EXT,
                                              NULL, attribs);

Should that also work do you think?

Cheers!

[JeffyCN]

for egl and zero-copy, it's been verified with ffmpeg mpv(xv vo) xserver(glamor), with my github version

[Fredrum]

Thank you for confirming that's half the battle won! :)

I wonder if I shouldn't get a 'drm' out dev from after the ffmpeg config run?

I'm running, ./configure --enable-shared --disable-static --enable-libdrm --enable-opengl --enable-rkmpp --enable-version3 --extra-libs=-lrockchip_mpp

But only getting:

Enabled outdevs:
alsa                    opengl                  sdl2                    v4l2
fbdev                   oss                     sndio                   xv

UPDATE: I'm trying running mpv under the tty on jellyfish hevc video files and I get image but this error messge:

[vo/gpu/opengl] Cannot retrieve DRM resources: Operation not supported
[vo/gpu/opengl] Failed to create KMS.
[vo/sdl] Using opengl
[vo/sdl] Warning: this legacy VO has bad performance. Consider fixing your graphics drivers, or not forcing the sdl VO.
No video PTS! Making something up. Using 29.970000 FPS.
VO: [sdl] 1920x1080 yuv420p 

I then tried running like this instead, > mpv --vo=drm but then I got,

(+) Video --vid=1 (*) (hevc 1920x1080 29.970fps)
[vo/drm] Cannot retrieve DRM resources: Operation not supported
[vo/drm] Failed to create KMS.
Error opening/initializing the selected video_out (--vo) device.
Video: no video
No video or audio streams selected.

Neither which sound like its working as it should?

Are those messages what you would expect?

[JeffyCN]

try to run it as video user(or root).

the test scripts:

# xv vo
mpv --hwdec=rkmpp --vd-lavc-software-fallback=no --vo=xv test.mp4

# x11egl + drm overlay
mpv --hwdec=rkmpp --vo=opengl --gpu-hwdec-interop=drmprime-drm --gpu-context=x11egl test.mp4

related code: https://github.com/JeffyCN/mpv/blob/debian/0.29.1-1/video/decode/vd_lavc.c#L789 https://github.com/JeffyCN/xorg-xserver/blob/rockchip/debian/1.20.4/glamor/glamor_xv.c#L642

[Fredrum]

I'm not going to have time to look into this more for a few days but I feel something is still amiss with the drm prime zero copy path. Based on me comparing CPU useages. To get more answers I'd have to spend more time and maybe insert prints in the mpv code etc.

I'm also still not getting a good codec_context->pix_fmt = AV_PIX_FMT_DRM_PRIME; // for zero copy video path returned from the Orange Pi's avocdec negotiated codec context. I'm getting a 0(zero) instead of the usual 181.

[Fredrum]

Hi none of your above examples are capable of demonstrating direct-to-screen using drm prime as they rely on either xv or x11 on which you cannot do drm display. I still tried the top one under tty (as that's what I need drm prime for) and I get the message:

 (+) Video --vid=1 (*) (hevc 1920x1080 29.970fps)
[vo/xv/x11] couldn't open the X11 display ()!
Error opening/initializing the selected video_out (--vo) device.
Video: no video
No video or audio streams selected.

which makes sense since im in tty without x11 server. As I want to be.

I then started adding things to some of the files in this ffmpeg fork and they also indicate that there's no drm prime going on here.

in libavutil/hwcontext.c your code errors out at around the line 625:

    if (!device_ctx->internal->hw_type->device_create) {
        ret = AVERROR(ENOSYS);
        printf("error b\n");
        goto fail;
    }

Which supports my observation in my own code that I'm not able to get a drm hw decoder path.

I then added some printouts to libavcodec/rkmppdec.c around line 278:

    decoder->device_ref = av_hwdevice_ctx_alloc(AV_HWDEVICE_TYPE_DRM);
    if (!decoder->device_ref) {
        printf("MyPrint:  in rkmpp Error av_hwdevice_ctx_alloc(AV_HWDEVICE_TYPE_DRM)\n");
        ret = AVERROR(ENOMEM);
        goto fail;
    }
    ret = av_hwdevice_ctx_init(decoder->device_ref);
    if (ret < 0)
        goto fail;

    printf("MyPrint:  rkmpp resolved pix_fmt: %d\n", avctx->pix_fmt);  // again, prints 0(zero) should be 181

Those both print out errors.

If its indeed possible to do drm prime you should be able to tell me a ffmpeg command line that can play without egl or gl middle layer under tty.

Something like this, ffmpeg -vcodec hevc_rkmpp -hwaccel drm -i ~/Desktop/jellyfish-40-mbps-hd-hevc.mkv -f null /dev/null

but with a drm based display device (without egl/gl). The above command prints the error:

Device creation failed: -14.
No device available for decoder: device type drm needed for codec hevc_rkmpp.

I'm guessing that people are getting confused with 'hardware decoding' and 'drm display'. I can get fast hardware decoding with hevc_rkmpp. But I can't get that displayed using a drm path.

In my case that's something I'm looking for to lower latency as I'm doing a gaming app where every little latency improvement is noticeable.

But in the meantime I have it working just doing a regular copy-to-gl-texture and its not bad since the hardware is so capable.

Cheers!

[JeffyCN]

https://github.com/JeffyCN/mpv/commit/af32a0b5e7732caabc788374eb19edb4940d1060

[JeffyCN]

1/ the drm prime path was added by ffmpeg upstream in the first version, my branch is just about a few performance improving and adding sw formats. if you think it doesn't work, please report to ffmpeg upstream.

2/ the hw path been tested by lots of people, including me with custom mpv(xv + drm-prime format) and custom mpv(x11egl(just for getting window rectangle), drm overlay + drm-prime format), my tests is just about showing mpv is able to get drm-prime format.

3/ the example of hw path is mpv, including: a/ how to get drm-prime hw format b/ how to use it in drm display. if you don't know how to use mpv, please write your own code to do those. i've posted related code about a/ before.

[JeffyCN]

and the other option would be hard-coding pixel format to drm-prime here: https://github.com/JeffyCN/FFmpeg/blob/master/libavcodec/rkmppdec.c#L200

the sw format would be disabled after that.

[Fredrum]

I'm starting to think maybe the environment also needs to be at a higher Kernel version. Possibly 5.13 or higher. The Orange Pi 5 is locked to an old 5.10 kernel that I think is not supporting drm prime. maybe I can try to run a new mainline kernel even of it is not ready for rk3588.

[JeffyCN]

the drm-prime should be supported even in BSP kernel 4.4

to use zero-copy hw path, you need: 1/ get ffmpeg to provide drm-prime hw format, which needs format negotiation(i've provided related code in mpv), or hard-coding it(i've provided related code too)

2/ use that drm-prime hw format dma-buf to do drm direct display, check mpv drm related code or kodi or gst kmssink

[JeffyCN]

FYI, all of these are tested by lots of people

and i would prefer to use gstreamer for video decoding

[Fredrum]

Well as long as I don't get a AV_PIX_FMT_DRM_PRIME its not working. ffmpeg is the main video decode api used in an enormous amount of software.

[JeffyCN]

check: "1/ get ffmpeg to provide drm-prime hw format, which needs format negotiation(i've provided related code in mpv), or hard-coding it(i've provided related code too)" https://github.com/JeffyCN/mpv/blob/debian/0.29.1-1/video/decode/vd_lavc.c#L789 https://github.com/JeffyCN/FFmpeg/blob/master/libavcodec/rkmppdec.c#L200