Orange pi 5 plus HDMI IN native V4L2 support really needed (OBS crashes)

[MaximKurakin]

Thank you for your tremendous efforts!!! I'm trying to build a re-streaming platform out of orange pi 5 plus with 24.04 Ubuntu from you. But there is a bummber issue like OBS and other apps cannot open HDMI in via v4l2 propertly due to driver IOCTL missing.

So i found another fork where it seems to me all fixed and hdmirx driver patched for rockchip and v4l2 compliance runs with no errors. https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/issues/4#note_152133 The core issue which crashes OBS is lack of VIDIOC_G_PARM ioctl.

Could you please merge those chages with you build?

Grabbing HDMI IN with gstreamer or ffmpeg is working but i'm getting 30 fps max at 4K resolution and cannot change it no matter what setting i'm trying to apply. I need to get 2K in 60 FPS. I think this orange pi is capable of doing it with no effors (spec says 4K 60fps but nobody can get it from HDMI IN so far). It would be great to just plug and play HDMI IN via V4L2 directly to OBS and check what fps will i get then.

Thank you!

[neofeo]

I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg.

I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the VIDIOC_G_PARM ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18

I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.

First, you need to enable hdmi in since it was disabled on opi 5. Check this entire conversation.

[MaximKurakin]

I ended up having to rebuild the kernel whenever I wanted to test different pixel formats

You can do better. The logic of HDMIRX driver as I see it: setup fmt and dv timings lined up to the INCOMING signal. Thats' why you cannot change the format. Driver simply droppes all you set_fmt to the current format. ENUM_FMT func is messy and gives you a false hope that you may use 4 formats. Actually you can but only by changing outgoing fmt at you SOURCE device.

I hope it will geve you a hint where to go next...

[dfloer]

I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg.

I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the VIDIOC_G_PARM ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18

I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.

This is excellent to see, thanks.

I'm also interested in figuring out how to make the driver show the proper pixel format. If I feed it a NV12, NV16 or NV24 signal, it still claims that the signal is RGB, resulting in glitchy colours if I take it at face value. Doing a format conversion to the expected pixel format allows me to fix this, but it'd be nice if the metadata was correct.

[ramiropolla]

I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg.

I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the VIDIOC_G_PARM ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18

I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.

This is excellent to see, thanks.

I'm also interested in figuring out how to make the driver show the proper pixel format. If I feed it a NV12, NV16 or NV24 signal, it still claims that the signal is RGB, resulting in glitchy colours if I take it at face value. Doing a format conversion to the expected pixel format allows me to fix this, but it'd be nice if the metadata was correct.

I'll have a look when I have the time again.

By the way, the hdmi driver uses the v4l2 multiplanar API, even though they only use one plane. Does anyone know of another device or SBC that also implements the multiplanar API, but that uses more than one plane?

[dfloer]

NV12/NV16/NV24 should have two planes, as it's a "semi-planar" pixel format. One for the Y (luma) and one for the CbCr (chroma), which are interleaved. See: https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html

[ramiropolla]

NV12/NV16/NV24 should have two planes, as it's a "semi-planar" pixel format. One for the Y (luma) and one for the CbCr (chroma), which are interleaved. See: https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html

They have two planes, but the data is provided only in one memory buffer (see cplanes and mplanes here): https://github.com/Joshua-Riek/linux-rockchip/blob/73c7c9a13e84d9b43f2cf083e757a528439e0965/drivers/media/platform/rockchip/hdmirx/rk_hdmirx.c#L390

So the v4l2 multiplanar API exposes only one plane here. I'd like to add proper support for multiplanar API for FFmpeg, and with this device I can't really test.

[ramiropolla]

I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg. I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the VIDIOC_G_PARM ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18 I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.

This is excellent to see, thanks. I'm also interested in figuring out how to make the driver show the proper pixel format. If I feed it a NV12, NV16 or NV24 signal, it still claims that the signal is RGB, resulting in glitchy colours if I take it at face value. Doing a format conversion to the expected pixel format allows me to fix this, but it'd be nice if the metadata was correct.

I'll have a look when I have the time again.

That's odd. When I plug an NV24 input I get this:

$ v4l2-ctl -all
Driver Info:
    Driver name      : rk_hdmirx
    Card type        : rk_hdmirx
    Bus info         : fdee0000.hdmirx-controller
    Driver version   : 6.1.43
    Capabilities     : 0x84201000
        Video Capture Multiplanar
        Streaming
        Extended Pix Format
        Device Capabilities
    Device Caps      : 0x04201000
        Video Capture Multiplanar
        Streaming
        Extended Pix Format
Priority: 2
Video input : 0 (hdmirx: ok)
DV timings:
    Active width: 1920
    Active height: 1080
    Total width: 2200
    Total height: 1125
    Frame format: progressive
    Polarities: -vsync -hsync
    Pixelclock: 148492000 Hz (60.00 frames per second)
    Horizontal frontporch: 88
    Horizontal sync: 44
    Horizontal backporch: 148
    Vertical frontporch: 4
    Vertical sync: 5
    Vertical backporch: 36
    Standards: 
    Flags: 
DV timings capabilities:
    Minimum Width: 640
    Maximum Width: 4096
    Minimum Height: 480
    Maximum Height: 2160
    Minimum PClock: 20000000
    Maximum PClock: 600000000
    Standards: CTA-861
    Capabilities: Interlaced, Progressive
Format Video Capture Multiplanar:
    Width/Height      : 1920/1080
    Pixel Format      : 'NV24' (Y/UV 4:4:4)
    Field             : None
    Number of planes  : 1
    Flags             : set-csc, 0x0000002c
    Colorspace        : Default
    Transfer Function : Unknown (0x00000020)
    YCbCr/HSV Encoding: xvYCC 601
    Quantization      : Default
    Plane 0           :
       Bytes per Line : 1920
       Size Image     : 6220800

User Controls

            audio_sampling_rate 0x00981a80 (int)    : min=0 max=768000 step=1 default=0 value=48000 flags=read-only, volatile
                  audio_present 0x00981a81 (bool)   : default=0 value=1 flags=read-only, volatile

Digital Video Controls

                  power_present 0x00a00964 (bitmask): max=0x00000001 default=0x00000000 value=1 flags=read-only

And FFmpeg is happy treating it as NV24.

[dfloer]

I'll need to test again with the new kernel. Maybe your recent patch fixed this issue...

[MaximKurakin]

@michelehokuto friend. Shortly - with current gstreamer condition any videoscale and videoconvert gives you a lot of CPU usage and you cannot get 60 fps in any resolution at orange pi 5 plus. For some reason gstreamer likes only NV12 format. Both NV24 and RGB will give you "not negotiated error" and you will have to use videoconvert. As of today I didn't manage to capture 1080p60 NV24 or RBG in OBS due to low performance issues.

The best way is to leave it in NV12 color space and 4K and scale it in OBS by changing the canvas size.
Try to launch this code and grab this window by OBS "windows capture" function. You will see good stable 4K 60 and bearable CPU load. FYI: DO NOT USE "fps-overlay" as is (on the screen) it kills the performance significantly! Use always: signal-fps-measurements=true text-overlay=false and check FPS in the console. Also keep in mind that glimagesink renders openGL window directly to you monitor in the current display resolution. It means if you have 4K in you HDMI IN but your monitor is 1080p and you are using glimagesink - you final resolution is 1080. You may use xvimagesink instead but it goes with windows header and borders:

GST_DEBUG=2,fpsdisplaysink:4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1,pixel-aspect-ratio=1/1,interlace-mode=progressive ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v

If you need only to record in 1080 then try ffmeg. It will also generates tons of CPU load but still can provide 60 FPS the trick is scaling with rkrga hardware support which you cannot set in gstreamer (i don't know how): ffmpeg -init_hw_device rkmpp -f v4l2 -i /dev/video0 -vf "format=nv12|nv16|bgr24,hwupload,scale_rkrga=w=1920:h=1080:format=nv12:afbc=1" -c:v hevc_rkmpp -qp_init 18 -g:v 120 -v verbose -f nut - | mpv -

If you still need NV24 then use GST_DEBUG=4 param to see what is actually breaking negotiation in gstreamer. Usually it's a wrong colorspace which forces you to use videoconvert.

I know this is ridiculous but rk3588 chip software support currenly in the very raw state. RK3588 spec says "hardware encoding and decoding support" but actually all needed software (ffmpeg, ffplay, vlc, gst, obs ... ) are not fully support our chip. THere are separeted efforts to enable our chip support from the hard working guys like Joshua(stitching all togather) and nyanmisaka (ffmpeg for rockchip HW encoding\decoding), collabora (https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md - attempt to enable ALL hardware in upstream kernel for rockchip) but all this attempts are still "work in progress". If you are good at codding - then hack into the source code and DYI -) but if you are just a normal user - just wait for panthor upstream support (6.10 kernel myabe?) and suffer.

I'm still working on HDMIRX driver by myself and managed to add VIDIOC_G_PARM to it and also fix 4 errors out of 9 but my work is still far from completion.

[dfloer]

For some reason gstreamer likes only NV12 format. Both NV24 and RGB will give you "not negotiated error" and you will have to use videoconvert.

This sounds like the issues I was describing, where the capture device doesn't signal to gstreamer what format it's using. If your input is NV12, you need to use NV12. The capture device doesn't do any pixel format conversion, so choosing the wrong format will have it fail to negotiate.

Shortly - with current gstreamer condition any videoscale and videoconvert gives you a lot of CPU usage and you cannot get 60 fps in any resolution at orange pi 5 plus.

Setting the environment variable GST_VIDEO_CONVERT_USE_RGA=1 should use the RGA for video conversion and scaling. It has worked previously with 4k video for me.

[MaximKurakin]

Setting the environment variable GST_VIDEO_CONVERT_USE_RGA=1 should use the RGA for video conversion and scaling. It has worked previously with 4k video for me.

Could prove it with this cmd (downscale 4k to FHD 60) ( to check RGA is actually involved I used this cmd sudo cat /sys/kernel/debug/rkrga/load): GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! videoconvert ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1 ! videoscale ! video/x-raw,format=NV12,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=xvimagesink signal-fps-measurements=true text-overlay=false sync=false -v

But my Laptop Nvidia with Optimus looks like try to play "smart" and if nothing is moving on the screen it drops fps down to 30 from time to time. Anyway, that should be irrelevant for your task.

But trying to do the same with glimagesink doesn't work... I think opengl accepts only RGB signal? Could anybody make it work? I cound't find how to convert NV12 to RGB or to NV24... GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! videoconvert ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1 ! videoscale ! video/x-raw,format=NV12,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v

[MaximKurakin]

Could anybody make it work? I cound't find how to convert NV12 to RGB or to NV24...

Hmmm maybe something is really wrong with 1080 NV24 format? I connected Orange Pi 5 Plus HDMI 2 to HDMI IN and set 1080p60 NV24.

Format Video Capture Multiplanar:
    Width/Height      : 1920/1080
    Pixel Format      : 'NV24' (Y/UV 4:4:4)
    Field             : None
    Number of planes  : 1
    Flags             : 
    Colorspace        : Rec. 709
    Transfer Function : Unknown (0x00000098)
    YCbCr/HSV Encoding: xvYCC 709
    Quantization      : Default
    Plane 0           :
       Bytes per Line : 1920
       Size Image     : 6220800

Then tried to capture it with gstreamer: GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV24,height=1080,width=1920,framerate=60/1 ! videoconvert ! video/x-raw,format=NV12,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=xvimagesink signal-fps-measurements=true text-overlay=false sync=false -v

And I'm getting only 30 stable FPS and zero RGA usage even with GST_VIDEO_CONVERT_USE_RGA=1 . WTF? Also I had to convert NV24 to NV12 because xvimagesink doesn't work with NV24 -(

Possible reason one CPU core is 60% loaded.

[MaximKurakin]

RGB capturing in 1080p60 is broken completely. 4 FPS only

Format Video Capture Multiplanar:
    Width/Height      : 1920/1080
    Pixel Format      : 'BGR3' (24-bit BGR 8-8-8)
    Field             : None
    Number of planes  : 1
    Flags             : 
    Colorspace        : sRGB
    Transfer Function : Unknown (0x00000098)
    YCbCr/HSV Encoding: Unknown (0x000000ff)
    Quantization      : Limited Range
    Plane 0           :
       Bytes per Line : 5760
       Size Image     : 6220800

GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=BGR,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v

Possible reason: One CPU core is always 100% loaded. Gstreamer issue? single core RGB encoding? hmmm I don't know...

[MaximKurakin]

Then only color format which gives stable 60 FPS for 1080p is NV16!

Format Video Capture Multiplanar:
    Width/Height      : 1920/1080
    Pixel Format      : 'NV16' (Y/UV 4:2:2)
    Field             : None
    Number of planes  : 1
    Flags             : 
    Colorspace        : Default
    Transfer Function : Unknown (0x00000098)
    YCbCr/HSV Encoding: xvYCC 601
    Quantization      : Default
    Plane 0           :
       Bytes per Line : 1920
       Size Image     : 4147200

GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV16,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v

CPU graph is fine and flat

[MaximKurakin]

Finally I could achieve my goal to stream in 60 fps!!!! but only in 1080p =((( OPI 5 PLUS can handle 2k streaming but fps will be jumping between 44 to 56. Not eyes pleasing I must say... *Also NV16 color space is limited and can cause broken color gradients if your source is already compressed.

Short instructions for those who wants to get the same 1080p60 with camera and gstreamer hw h264 encoding to youtube:

1. Build and install to OBS obs-gstreamer plugin and you will get hardware encoder h264 via gstreamer IN OBS

3. Build and install to OBS droid-cam-obs plugin and you will get 1080p60 smooth camera with minimum workload via Wifi

5. Setup your gaming PC HDMI to output 1080p60 in NV16 (YCbCr422) and start gstreamer on Orange PI 5 Plus:

GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV16,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v

6. Setup OBS canvas and video settings to 1080

7. Capture openGL window in OBS via "Window Capture"

8. Start streaming and enjoy stable 60 fps streaming!!! Orange Pi will need cooling to handle this workload (I have noctua 80mm cooler + huge cooper radiator + custom bash script daemon handling PWM Fan control. But GPIO and PWM is a big topic for other time. will not post it here)

@michelehokuto just tag you because you were asking for something like this in another thread.