please identify where to find the stateless v4l2 code here?

[fduncanh]

jc-kinesim writes:

I've written (or more accurate taken some existing code from the Kodi project and modified it out of almost all recognition - but the starting point should be acknowledged) a stateless V4L2 decode that works with the Pi4-5 H265 decode

Hi jc-kinesim

It would be very useful if you could point out where here to find the stateless v4l2 decode you wrote. to access pi5 (and 4?) hw h265 decoding

We would be interested in getting gstreamer to use it if at all possible.

[jc-kynesim]

libavcodec/v4l2_req*. Pick a branch say test/5.1.6/main.

[fduncanh]

@jc-kynesim

Are the files listed below all the relevant ones that you added?

will they show how the connection to the Broadcom gpu is made? (where?)

The interest here is using the stateless v4l2 support in gstreamer directly, rather than through ffmpeg. The previous R PI model 4b had working h264 gpu support using (stateful ?) v4l2 support in gstreamer-plugins-good. The stateless v4l2 support is in gstreamer-plugins-bad. ideally this could be got to use the h265 support in the Pi model 5 and maybe model 4b

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/v4l2codecs? ref_type=heads

-rw-r--r-- 1 x users 844 Aug 14 10:55 v4l2_req_utils.h
-rw-r--r-- 1 x users  3030 Aug 14 10:55 v4l2_request_hevc.h
-rw-r--r-- 1 x users 12774 Aug 14 10:55 v4l2_request_hevc.c
-rw-r--r-- 1 x users   560 Aug 14 10:55 v4l2_req_pollqueue.h
-rw-r--r-- 1 x users  8573 Aug 14 10:55 v4l2_req_pollqueue.c
-rw-r--r-- 1 x users  7668 Aug 14 10:55 v4l2_req_media.h
-rw-r--r-- 1 x users 49285 Aug 14 10:55 v4l2_req_media.c
-rw-r--r-- 1 x users 47656 Aug 14 10:55 v4l2_req_hevc_vx.c
-rw-r--r-- 1 x users    60 Aug 14 10:55 v4l2_req_hevc_v4.c
-rw-r--r-- 1 x users    60 Aug 14 10:55 v4l2_req_hevc_v3.c
-rw-r--r-- 1 x users    60 Aug 14 10:55 v4l2_req_hevc_v2.c
-rw-r--r-- 1 x users    60 Aug 14 10:55 v4l2_req_hevc_v1.c
-rw-r--r-- 1 x users  1547 Aug 14 10:55 v4l2_req_dmabufs.h
-rw-r--r-- 1 x users  8926 Aug 14 10:55 v4l2_req_dmabufs.c
-rw-r--r-- 1 x users   693 Aug 14 10:55 v4l2_req_devscan.h
-rw-r--r-- 1 x users 12395 Aug 14 10:55 v4l2_req_devscan.c
-rw-r--r-- 1 x users   745 Aug 14 10:55 v4l2_req_decode_q.h
-rw-r--r-- 1 x users  1750 Aug 14 10:55 v4l2_req_decode_q.c
``

[jc-kynesim]

Yup that is pretty much the set of files. It uses the standard V4L2 "stateless" interface, which is generally documented here https://www.linuxtv.org/downloads/v4l-dvb-apis/userspace-api/index.html - Part I Section 4.5.3 is probably the most help.

[fduncanh]

It would be very useful to get a few hints about how the connection to the R Pi Broadcom GPU in between the OUTPUT and CAPTURE queue is made. (and which drivers to connect to, presumably in the RPi kernel tree? the drivers are probably shims to Broadcom proprietary firmware?))

I did get issues with the older stateful V4l2 plugin in gstreamer fixed for our gstreamer-based Apple Airplay server so have some experience. I am hoping that most of the user-side stateless stuff is already done in gstreamer-plugins-bad code

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/v4l2codecs?ref_type=heads

[jc-kynesim]

The driver is in the PiOS linux tree - you can find it under driver/staging/media/rpivid/*. Hevc decode is a completely independent h/w block - GPU not involved. I'm sure the h/w is proprietary, I'm not completely sure to whom, so you are on your own for figuring out what is going on there (but honestly the h/w setup corresponds quite closely to the info passed in the V4L2 stateless structures so most of it is quite obvious).

[fduncanh]

@jc-kynesim

Thanks for your helpful responses:

I finally found the reason why rpivid driver is not recognized as a stateless V4l2 decoder by GStreamer v4l2slh265dec plugin: it's because its response to the VIDIOC_G_FMT and VIDIOC_ENUM_FMT ioctl calls reports unsupported fourcc pixelformats NC12 and NC30. ("SAND")

$v4l2-ctl --list-formats-ext -d 19 ioctl: VIDIOC_ENUM_FMT Type: Video Capture Multiplanar

    [0]: 'NC12' (Y/CbCr 4:2:0 (128b cols))
    [1]: 'NC30' (10-bit Y/CbCr 4:2:0 (128b cols))

(Is "sand" a reference to the HEVC block hardware name?)

I see two places its handled: https://github.com/jc-kynesim/rpi-ffmpeg/commit/c7d8474ffa5689abf99367c43ec2d39a1957f564 https://github.com/jc-kynesim/rpi-ffmpeg/commit/b6b137b1d039b42b15325f87f55cb7c38e2270b0 , is that all?

I'm wondering how big a task it is likely to be to get support of "sand" into gstreamer.

are there any sources of information about "sand" formats?

I couldn't find any other source than here.

[6by9]

AIUI Mainline GStreamer will only support formats that are defined in the mainline kernel, and upstreaming the HEVC decoder is still in progress (I need to talk to jc-kynesim about it this coming week).

Once the format and driver are in the mainline kernel, I wouldn't expect there to be any great obstacles to adding the format to GStreamer, but would ask Nicolas Dufresne as the person who knows that area best.

Sand was an internal Broadcom name for the format as it originally came from a company called Sand Video that was bought by them in 2004 - https://www.theregister.com/2004/04/13/broadcom_acquires_sand/

[fduncanh]

@6by9

Thanks: I edited my post on the GStreamer forum to include your comment:

https://discourse.gstreamer.org/t/v4l2codecs-feasibility-of-adding-support-for-raspberry-pi-hevc-sand-formats-nc12-nc30/3522

It would be great to get the format (and driver) into the mainstream kernel. I was looking into whether a patch to GStreamer could be developed in the meantime, so ready to go after acceptance into the mainstream kernel.

[fduncanh]

@6by9 @jc-kynesim

is there any possibility the format could get into the mainstream kernel before the rpivid driver (in case that is delayed?)

[fduncanh]

@6by9 @jc-kynesim

@6by9 writes:

AIUI Mainline GStreamer will only support formats that are defined in the mainline kernel, and upstreaming the HEVC decoder is still in progress (I need to talk to jc-kynesim about it this coming week).

In some sense, this suggests that upstreaming the format is even more urgent than upstreaming the driver, which will be present on R Pi OS whether or not it is upstreamed. Do Ubuntu for R Pi use the R Pi kernel tree as well (they have support for the non-mainline kernel stateful h264 RPi decoder)?

[ndufresne]

Posted a reply in https://discourse.gstreamer.org/t/v4l2codecs-feasibility-of-adding-support-for-raspberry-pi-hevc-sand-formats-nc12-nc30/3522/2?u=ndufresne

We don't usually accept formats without a driver using it in Linux Media. Most attempts stalled so far. The HEVC decoder is likely the easiest to upstream, but iirc needs some clean-up.

As explained, we are working to make it easier for vendors to add formats (in opaque form). This also enables the use of compression, which is very often needed for 4k60+ performance.

[jc-kynesim]

I had a go at just getting sand into the kernel a while ago but indeed I was told to go away and upstream a driver.

[fduncanh]

@jc-kynesim

Thanks for all your work on the rpivid driver. Hope it makes it into mainline. There seems to be quite a bit of the activity (experimenting with rpivid and DRM_FORMAT_MOD_BROADCOM_SAND128.) mentioned by @ndufresne going on at https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/7355