Use Fuse instead of libv4l-utils

hbiyik commented 1 year ago

@JeffyCN

https://github.com/libfuse/libfuse/blob/master/example/ioctl.c

I have briefly tested this idea. The idea is to use a libfuse filesystem. A deamon as in the example will create a filesystem with video-vepu580, video-vdpuxxx files in it.

After the daemon is mounted to a directory, the generated files will be symlinked to /dev/video-*.

When another client application opens this /dev/video-vepu580 ie, and calls V4L2 IOCTL like (VIDIOC_QUERYCAP) with this FD, the daemon can catch it and dispatch to mpp backend to handle. ie: https://github.com/libfuse/libfuse/blob/869a4a6fa550ae054df01f9d50db68871f88ca4f/example/ioctl.c#L204 handle it like: https://github.com/JeffyCN/libv4l-rkmpp/blob/38da407140d7de2f6cf70c613787ab44be2bbb3a/src/libv4l-rkmpp-enc.c#L1167

This allows to bind v4l2 backend to mpp genericly, this can be used with chromium Firefox since 116 https://bugzilla.mozilla.org/show_bug.cgi?id=1833354 FFmpeg https://github.com/FFmpeg/FFmpeg/blob/8028b8260e6d0a4a469529ec5c5d504782a5f875/libavcodec/v4l2_m2m.c#L75

literally anything that supports vl42_m2m

the main difference with v4l2-utils, is that the plugin only works if the client uses libv4l2, if direct IOCTL is used then this does not work. Also with libfuse approach you do not need to patch anything at all, it is straight forward.

One concern could be how much overhead that the libfuse would bring but i think it should be minuscule.

What do you think? i think most the code already avaibale in this repo already..

JeffyCN commented 1 year ago

it's doable, but: the v4l-utils can alse used directly with LD_LIBRARY_PRELOAD

the main reason of limiting this plugin on chromium is that it depends on some special flow of chromium v4l2 vda(some are original chromium's, some are custom patched)

hbiyik commented 1 year ago

the v4l-utils can alse used directly with LD_LIBRARY_PRELOAD

i did not understand this, can you give a simple example how this can be achieved with ffmpeg? ie for a command like below using v4l2_m2m encoder which searches /dev/video-* v4l2 devices

ffmpeg -t 5 -init_hw_device drm=dr:/dev/dri/renderD128 -filter_hw_device dr -f lavfi -i testsrc=s=1280x720,format=nv12 -vf hwupload,format=drm_prime -c:v h264_v4l2m2m out.mp4

i think LD_LIBRARY_PRELOAD could only be used if it had replaced ioctl function may be.

https://github.com/FFmpeg/FFmpeg/blob/8028b8260e6d0a4a469529ec5c5d504782a5f875/libavcodec/v4l2_m2m.c#L75 ioctl(s->fd, VIDIOC_QUERYCAP, &cap);

notice that ffmpeg is not using v4l2_ioctl

am i missing something?

JeffyCN commented 1 year ago

try google ld_preload v4l2convert.so

the v4l-utils would hook open ioctl and close

hbiyik commented 1 year ago

hah, i dont know how, but it works.. segfaults though, i can take further from here. thanks

hbiyik commented 1 year ago

Ah i see, ioctl is calling v4l2_ioctl, this is how it is working ld preload. Closing the issue

hbiyik commented 8 months ago

I want to revisit this idea. With some minor tweaks i managed to get some frames out of the v4l2m2m decoder of ffmpeg.

I think this approach is really feasible because:

Even vanilla FFmpeg directly benefit from mpp using v4l2 statefull interace. No need to modify much.
There is already a very advanced RGA scaler implemented in ffmpeg, when needed this can also be mainlined but this is rather a secondary issue.
A lot of players have v4l2m2m interface with drm_prime output that achieves 0 copy.
Recently firefox also implemented v4l2m2m and chromium has support for a long time.

Good news is that the plugin already is non blocking and using external buffers. Bad news is there is no AFBC support, not even sure v4l2 has a proper afbc definition or agreed upon modifiers

to test: https://github.com/hbiyik/libv4l-rkmpp https://github.com/hbiyik/FFmpeg/tree/exp_m2m

LIBV4L_RKMPP_LOG_LEVEL=3 LD_PRELOAD=/usr/lib/v4l2convert.so ffplay -loglevel trace -vcodec h264_v4l2m2m ~/hiresvids/jelly/jellyfish-200-mbps-4k-uhd-h264.mkv

hbiyik commented 8 months ago

https://github.com/hbiyik/mpp-v4l2m2m/

here is the initial project as a starter, currently only VIDIOC_QUERYCAP works, but hey, it is a start.

[alarm@alarm fuse3]$ sudo v4l2-ctl -D -d /dev/video0-mpp-dec 
[sudo] password for alarm: 
Driver Info:
    Driver name      : rkmpp
    Card type        : rkmpp
    Bus info         : platform: rkmpp
    Driver version   : 6.4.0
    Capabilities     : 0x84204000
        Video Memory-to-Memory Multiplanar
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04204000
        Video Memory-to-Memory Multiplanar
        Streaming
        Extended Pix Format

sudo ./buildarm64/mpp-v4l2m2m-dec -f -l 3
[sudo] password for alarm: 
[843.740] [RKMPP] [47697] context_init(106): ctx(0xffff8c000c00)): inited,
[843.741] [RKMPP] [47705] decoder_thread_fn(240): ctx(0xffff8c000c00): starting decoder thread
[843.741] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERY_EXT_CTRL
[843.741] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_TRY_EXT_CTRLS
[843.742] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.742] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_G_SELECTION
[843.742] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERY_EXT_CTRL
[843.742] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.742] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.742] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.743] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.744] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.745] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.746] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.747] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_QUERYCTRL
[843.748] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_G_SELECTION
[843.748] [RKMPP] [47696] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_G_SELECTION
[843.748] [RKMPP] [47697] codec_ioctl(84): ERR: Unsupported IOCTL: VIDIOC_ENUM_FMT
[843.749] [RKMPP] [47696] context_destroy(124): ctx(0xffff8c000c00): closing

JeffyCN commented 8 months ago

maybe wrapper to vaapi is another choice

hbiyik commented 8 months ago

Isnt vaapi slice based? Can it also be frame based? I have no clue on vaapi.

JeffyCN commented 8 months ago

the mpp has internal input splitter(disabled by default on linux platform), so it should be doable, but i don't know much about that too :)

hbiyik commented 8 months ago

Currently v4l2 is the low hanging fruit for me, i somehow can see how it should work (except gazillion future fuse problems, like mmap).

If someone would have an overall plan for vaapi i would just jump on it. I remember @nyanmisaka had sone plans to glue HAL interface of mpp to some vaapi userspace lib.

hbiyik commented 8 months ago

I have concluded that performane will be hit hard with fuse due to several copies needs to be involved. Therefore i concluded that this was a bad idea to begin with.

nyanmisaka commented 8 months ago

https://github.com/rockchip-linux/mpp/blob/develop/doc/Rockchip_Developer_Guide_MPP_EN.md#the-form-of-input-bit-stream-whole-frame-and-broken-frame

MPP_DEC_SET_PARSER_SPLIT_MODE

The command parameter is RK_U32*, which is used to enable the protocol parser in the MPP to process internal frame segmentation. The default bitstream input mode is whole frame mode and assume the input is frame segmented. This command is called before mpp_init.

If i understand correctly, this MPP_DEC cmd is to switch to the slice based?

The real hwaccels in ffmpeg are slice based, such as vaapi, d3d11va/dxva2 and nvdec. And hw decoders are frame based, such as qsv, cuvid and v4l2-stateful.

nyanmisaka commented 8 months ago

video: rockchip: mpp: Add uapi header

Btw there is a newly added uapi header for the mpp_service kernel driver. If it were not for upstreaming the driver, this change might not be necessary.

JeffyCN / libv4l-rkmpp

Use Fuse instead of libv4l-utils #15