Can we merge my version and your version of ffmpeg into 1 single version?

[hbiyik]

Hello

I also have a fork of FFmpeg supporting mpp at https://github.com/hbiyik/FFmpeg/tree/encoder and i think it makes more sense to have one single version of such a niche fork and considering both FFmpeg and Rockchip has abandoned support for mpp enabled Rockchip it is also reasonable have a joint effort to make best of the limited resources. (I am assuming you have no relateion to Rockchip :))

The main difference in my version and your version is:

Common: Your are using librga, i am using direct IOCTL calls. You are integrating RGA to scale filter, i am using it as an interface to do only format conversions (rkplane.c). I am using libyuv as a fallback in case RGA fails. (There is always some funny reason for librga to fail :))

Those whole RGA differences are because RGA is tricky and prune to error and always ready to fail in various different ways.

Decoder: I am also outputting YUV420P or NV12 addition to DRMPRime. This allows generic use on browsers like Firefox or VLC where DRMPrime is not there.

Encoder: I am also inputting YUYV422, UYVY422, NV16, YUV422P, BGR24, BGRA, BGR0, NV12, YUV420P addition to HW drimprime, which allows to interface several AV devices directly. I have also added support for multi planar V4L device support from @rigaya which allows hdmi input to be recorded.

What do you think, how can we merge those 2 projects?

[hbiyik]

@jjm2473

[jjm2473]

I'm hoping to merge as well, but I'm not an expert in image/video and probably won't be able to help.

The original purpose of my project is to run Jellyfin smoothly on the Rockchip platform, so the design tends to be modular, which is convenient to combine into different functions for Jellyfin to call. At least several combinations mentioned in the document https://github.com/jjm2473/ffmpeg-rk/wiki/%E7%BC%96%E8%A7%A3%E7%A0%81-%7C-Transcoding .

In order to reduce memory copying and achieve image scaling, I do not support the decoder to directly output software frames, but need to manually add filter processing, so I implemented the scale_rga filter, which may be useful to you, too.

If you just want to convert hardware frames to software frames, just add the 'hwdownload,scale' filter, so I don't think it's necessary for the decoder to output software frames.

(Translated by Google)

[jjm2473]

RGA should stable now, after some patches

[hbiyik]

hello @jjm2473

Thanks for the explanations, and i think your approach is much modular than the approach i took. However, i chose this path due to the fact tthat not all the applications utilize the filters (scale or hwdownload) automatically when decoding and encoding. Ie: Firefox when decoding only checks if the decoder outputs yuv420p or not, if not, it does not even use it. Same goes for most of players as well. Same situation when encoding, at least OBS has some options to provide extra command line arguments when encoding but it overall makes procedure more complicated.

What i think could be done is, scaling filter can still be implemented as you have done, but the decoder / encoder uses the rga_scale filter internally to convert the hw planes to sw planes without explicitly expecting user to activate them. This way best of both worlds can be implemented.

Notice that even librga is stable i am using rga and libyuv together, there are cases that RGA can not convert the plane at all. Ie: 8K planes to YUV420P, due to RGA1 limitated to 4K), or NV15 plane to YUV420P since RGA1 does not support NV15, the later can be achieved in 2 steps though. Therefore in some cases i still use libyuv. Or some whatever kernel RGA3 future problem that is going to happen.

Additional note, when i am using softframes, i do not copy, simply convert (RGA copy) the necessary plane to a new buffer, but use the existing buffer as well from DRM buffer. Ie: if the pixfmt is NV12, i simply use DRMbuffers assigned to AVFrame, no copy at all, if the output frame is YUV420P, then planes are converted to AVBuffer, but Y plane is still on DRM plane and UV is on the converted AVBuffer. So i took as much as optimizations as possible.

In future i might consider also using IEP hardware, but i did not experiment much about it as well. Have you ever played with IEP?

[jjm2473]

hello! I only considered the transcoding scene before, and did not consider the use of the desktop (firefox or something), so there is no need to output software frames. It would be great if the software and hardware frame conversion does not require memory copying! I don't know much about IEP either, because the project was aimed at rk3568 and above from the beginning. Just checked the documentation, IEP supports relatively few formats and functions. Regarding RGA, I think it is better to use librga, because the ABI may not be compatible between different versions of RGA, or even between different driver versions. Using ioctl directly requires a lot of judgment, and librga has already done this. (check this issue https://github.com/JeffyCN/mirrors/issues/11 )

[hbiyik]

ok, understood, i think i have never used that so called >1.9.0 version of the librga thats why may be i had some bias on it. so https://github.com/JeffyCN/mirrors/tree/linux-rga-multi this version 1.9.2 should cover all different RGA versions (1,2,3) on bsp kernels >4.4, correct? Is there any soc which should use another version of the librga?

[jjm2473]

@hbiyik As README said on https://github.com/JeffyCN/mirrors/tree/linux-rga-multi, it should support all popular SoCs. (But I don't test it on other than RK35xx)

[hbiyik]

after several work, i added following stuff:

• hevc encoder
• vp8 encoder
• move to librga
• scaling functionality

i think that should cover all the jellyfin things, may be outputing different fps the the output, and thats trivial to implment.

wiki is at: https://github.com/hbiyik/FFmpeg/wiki#compilation

[jjm2473]

Hello!

I add link to your project on https://github.com/jjm2473/ffmpeg-rk#readme

I will try to migrate Jellyfin when I have time