Open MaximKurakin opened 3 months ago
I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg.
I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the
VIDIOC_G_PARM
ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.
First, you need to enable hdmi in since it was disabled on opi 5. Check this entire conversation.
I ended up having to rebuild the kernel whenever I wanted to test different pixel formats
You can do better. The logic of HDMIRX driver as I see it: setup fmt and dv timings lined up to the INCOMING signal. Thats' why you cannot change the format. Driver simply droppes all you set_fmt to the current format. ENUM_FMT func is messy and gives you a false hope that you may use 4 formats. Actually you can but only by changing outgoing fmt at you SOURCE device.
I hope it will geve you a hint where to go next...
I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg.
I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the
VIDIOC_G_PARM
ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.
This is excellent to see, thanks.
I'm also interested in figuring out how to make the driver show the proper pixel format. If I feed it a NV12, NV16 or NV24 signal, it still claims that the signal is RGB, resulting in glitchy colours if I take it at face value. Doing a format conversion to the expected pixel format allows me to fix this, but it'd be nice if the metadata was correct.
I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg.
I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the
VIDIOC_G_PARM
ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.
This is excellent to see, thanks.
I'm also interested in figuring out how to make the driver show the proper pixel format. If I feed it a NV12, NV16 or NV24 signal, it still claims that the signal is RGB, resulting in glitchy colours if I take it at face value. Doing a format conversion to the expected pixel format allows me to fix this, but it'd be nice if the metadata was correct.
I'll have a look when I have the time again.
By the way, the hdmi driver uses the v4l2 multiplanar API, even though they only use one plane. Does anyone know of another device or SBC that also implements the multiplanar API, but that uses more than one plane?
NV12/NV16/NV24 should have two planes, as it's a "semi-planar" pixel format. One for the Y (luma) and one for the CbCr (chroma), which are interleaved. See: https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html
NV12/NV16/NV24 should have two planes, as it's a "semi-planar" pixel format. One for the Y (luma) and one for the CbCr (chroma), which are interleaved. See: https://docs.kernel.org/userspace-api/media/v4l/pixfmt-yuv-planar.html
They have two planes, but the data is provided only in one memory buffer (see cplanes and mplanes here): https://github.com/Joshua-Riek/linux-rockchip/blob/73c7c9a13e84d9b43f2cf083e757a528439e0965/drivers/media/platform/rockchip/hdmirx/rk_hdmirx.c#L390
So the v4l2 multiplanar API exposes only one plane here. I'd like to add proper support for multiplanar API for FFmpeg, and with this device I can't really test.
I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg. I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the
VIDIOC_G_PARM
ioctl. I opened a PR here: Joshua-Riek/linux-rockchip#18 I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.This is excellent to see, thanks. I'm also interested in figuring out how to make the driver show the proper pixel format. If I feed it a NV12, NV16 or NV24 signal, it still claims that the signal is RGB, resulting in glitchy colours if I take it at face value. Doing a format conversion to the expected pixel format allows me to fix this, but it'd be nice if the metadata was correct.
I'll have a look when I have the time again.
That's odd. When I plug an NV24 input I get this:
$ v4l2-ctl -all
Driver Info:
Driver name : rk_hdmirx
Card type : rk_hdmirx
Bus info : fdee0000.hdmirx-controller
Driver version : 6.1.43
Capabilities : 0x84201000
Video Capture Multiplanar
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x04201000
Video Capture Multiplanar
Streaming
Extended Pix Format
Priority: 2
Video input : 0 (hdmirx: ok)
DV timings:
Active width: 1920
Active height: 1080
Total width: 2200
Total height: 1125
Frame format: progressive
Polarities: -vsync -hsync
Pixelclock: 148492000 Hz (60.00 frames per second)
Horizontal frontporch: 88
Horizontal sync: 44
Horizontal backporch: 148
Vertical frontporch: 4
Vertical sync: 5
Vertical backporch: 36
Standards:
Flags:
DV timings capabilities:
Minimum Width: 640
Maximum Width: 4096
Minimum Height: 480
Maximum Height: 2160
Minimum PClock: 20000000
Maximum PClock: 600000000
Standards: CTA-861
Capabilities: Interlaced, Progressive
Format Video Capture Multiplanar:
Width/Height : 1920/1080
Pixel Format : 'NV24' (Y/UV 4:4:4)
Field : None
Number of planes : 1
Flags : set-csc, 0x0000002c
Colorspace : Default
Transfer Function : Unknown (0x00000020)
YCbCr/HSV Encoding: xvYCC 601
Quantization : Default
Plane 0 :
Bytes per Line : 1920
Size Image : 6220800
User Controls
audio_sampling_rate 0x00981a80 (int) : min=0 max=768000 step=1 default=0 value=48000 flags=read-only, volatile
audio_present 0x00981a81 (bool) : default=0 value=1 flags=read-only, volatile
Digital Video Controls
power_present 0x00a00964 (bitmask): max=0x00000001 default=0x00000000 value=1 flags=read-only
And FFmpeg is happy treating it as NV24.
I'll need to test again with the new kernel. Maybe your recent patch fixed this issue...
@michelehokuto friend. Shortly - with current gstreamer
condition any videoscale
and videoconvert
gives you a lot of CPU usage and you cannot get 60 fps in any resolution at orange pi 5 plus
. For some reason gstreamer likes only NV12 format. Both NV24 and RGB will give you "not negotiated error" and you will have to use videoconvert. As of today I didn't manage to capture 1080p60 NV24 or RBG in OBS due to low performance issues.
The best way is to leave it in NV12 color space and 4K and scale it in OBS by changing the canvas size.
Try to launch this code and grab this window by OBS "windows capture" function. You will see good stable 4K 60 and bearable CPU load. FYI: DO NOT USE "fps-overlay"
as is (on the screen) it kills the performance significantly! Use always: signal-fps-measurements=true text-overlay=false
and check FPS in the console. Also keep in mind that glimagesink
renders openGL window directly to you monitor in the current display resolution. It means if you have 4K in you HDMI IN but your monitor is 1080p and you are using glimagesink
- you final resolution is 1080. You may use xvimagesink
instead but it goes with windows header and borders:
GST_DEBUG=2,fpsdisplaysink:4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1,pixel-aspect-ratio=1/1,interlace-mode=progressive ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v
If you need only to record in 1080 then try ffmeg. It will also generates tons of CPU load but still can provide 60 FPS the trick is scaling with rkrga hardware support which you cannot set in gstreamer (i don't know how):
ffmpeg -init_hw_device rkmpp -f v4l2 -i /dev/video0 -vf "format=nv12|nv16|bgr24,hwupload,scale_rkrga=w=1920:h=1080:format=nv12:afbc=1" -c:v hevc_rkmpp -qp_init 18 -g:v 120 -v verbose -f nut - | mpv -
If you still need NV24 then use GST_DEBUG=4
param to see what is actually breaking negotiation in gstreamer. Usually it's a wrong colorspace which forces you to use videoconvert.
I know this is ridiculous but rk3588 chip software support currenly in the very raw state. RK3588 spec says "hardware encoding and decoding support" but actually all needed software (ffmpeg, ffplay, vlc, gst, obs ... ) are not fully support our chip. THere are separeted efforts to enable our chip support from the hard working guys like Joshua(stitching all togather) and nyanmisaka (ffmpeg for rockchip HW encoding\decoding), collabora (https://gitlab.collabora.com/hardware-enablement/rockchip-3588/notes-for-rockchip-3588/-/blob/main/mainline-status.md - attempt to enable ALL hardware in upstream kernel for rockchip) but all this attempts are still "work in progress". If you are good at codding - then hack into the source code and DYI -) but if you are just a normal user - just wait for panthor upstream support (6.10 kernel myabe?) and suffer.
I'm still working on HDMIRX driver by myself and managed to add VIDIOC_G_PARM to it and also fix 4 errors out of 9 but my work is still far from completion.
For some reason gstreamer likes only NV12 format. Both NV24 and RGB will give you "not negotiated error" and you will have to use videoconvert.
This sounds like the issues I was describing, where the capture device doesn't signal to gstreamer what format it's using. If your input is NV12, you need to use NV12. The capture device doesn't do any pixel format conversion, so choosing the wrong format will have it fail to negotiate.
Shortly - with current
gstreamer
condition anyvideoscale
andvideoconvert
gives you a lot of CPU usage and you cannot get 60 fps in any resolution atorange pi 5 plus
.
Setting the environment variable GST_VIDEO_CONVERT_USE_RGA=1
should use the RGA for video conversion and scaling. It has worked previously with 4k video for me.
Setting the environment variable
GST_VIDEO_CONVERT_USE_RGA=1
should use the RGA for video conversion and scaling. It has worked previously with 4k video for me.
Could prove it with this cmd (downscale 4k to FHD 60) ( to check RGA is actually involved I used this cmd sudo cat /sys/kernel/debug/rkrga/load):
GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! videoconvert ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1 ! videoscale ! video/x-raw,format=NV12,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=xvimagesink signal-fps-measurements=true text-overlay=false sync=false -v
But my Laptop Nvidia with Optimus looks like try to play "smart" and if nothing is moving on the screen it drops fps down to 30 from time to time. Anyway, that should be irrelevant for your task.
But trying to do the same with glimagesink
doesn't work... I think opengl accepts only RGB signal? Could anybody make it work? I cound't find how to convert NV12 to RGB or to NV24...
GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 io-mode=4 ! videoconvert ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1 ! videoscale ! video/x-raw,format=NV12,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v
Could anybody make it work? I cound't find how to convert NV12 to RGB or to NV24...
Hmmm maybe something is really wrong with 1080 NV24 format? I connected Orange Pi 5 Plus HDMI 2 to HDMI IN and set 1080p60 NV24.
Format Video Capture Multiplanar:
Width/Height : 1920/1080
Pixel Format : 'NV24' (Y/UV 4:4:4)
Field : None
Number of planes : 1
Flags :
Colorspace : Rec. 709
Transfer Function : Unknown (0x00000098)
YCbCr/HSV Encoding: xvYCC 709
Quantization : Default
Plane 0 :
Bytes per Line : 1920
Size Image : 6220800
Then tried to capture it with gstreamer:
GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV24,height=1080,width=1920,framerate=60/1 ! videoconvert ! video/x-raw,format=NV12,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=xvimagesink signal-fps-measurements=true text-overlay=false sync=false -v
And I'm getting only 30 stable FPS and zero RGA usage even with GST_VIDEO_CONVERT_USE_RGA=1
. WTF? Also I had to convert NV24 to NV12 because xvimagesink doesn't work with NV24 -(
Possible reason one CPU core is 60% loaded.
RGB capturing in 1080p60 is broken completely. 4 FPS only
Format Video Capture Multiplanar:
Width/Height : 1920/1080
Pixel Format : 'BGR3' (24-bit BGR 8-8-8)
Field : None
Number of planes : 1
Flags :
Colorspace : sRGB
Transfer Function : Unknown (0x00000098)
YCbCr/HSV Encoding: Unknown (0x000000ff)
Quantization : Limited Range
Plane 0 :
Bytes per Line : 5760
Size Image : 6220800
GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=BGR,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v
Possible reason: One CPU core is always 100% loaded. Gstreamer issue? single core RGB encoding? hmmm I don't know...
Then only color format which gives stable 60 FPS for 1080p is NV16!
Format Video Capture Multiplanar:
Width/Height : 1920/1080
Pixel Format : 'NV16' (Y/UV 4:2:2)
Field : None
Number of planes : 1
Flags :
Colorspace : Default
Transfer Function : Unknown (0x00000098)
YCbCr/HSV Encoding: xvYCC 601
Quantization : Default
Plane 0 :
Bytes per Line : 1920
Size Image : 4147200
GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV16,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v
CPU graph is fine and flat
Finally I could achieve my goal to stream in 60 fps!!!! but only in 1080p =((( OPI 5 PLUS can handle 2k streaming but fps will be jumping between 44 to 56. Not eyes pleasing I must say... *Also NV16 color space is limited and can cause broken color gradients if your source is already compressed.
Short instructions for those who wants to get the same 1080p60 with camera and gstreamer hw h264 encoding to youtube:
GST_VIDEO_CONVERT_USE_RGA=1 GST_DEBUG=4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV16,height=1080,width=1920,framerate=60/1 ! fpsdisplaysink video-sink=glimagesink signal-fps-measurements=true text-overlay=false sync=false -v
@michelehokuto just tag you because you were asking for something like this in another thread.
Thank you for your tremendous efforts!!! I'm trying to build a re-streaming platform out of orange pi 5 plus with 24.04 Ubuntu from you. But there is a bummber issue like OBS and other apps cannot open HDMI in via v4l2 propertly due to driver IOCTL missing.
So i found another fork where it seems to me all fixed and hdmirx driver patched for rockchip and v4l2 compliance runs with no errors. https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/issues/4#note_152133 The core issue which crashes OBS is lack of VIDIOC_G_PARM ioctl.
Could you please merge those chages with you build?
Grabbing HDMI IN with gstreamer or ffmpeg is working but i'm getting 30 fps max at 4K resolution and cannot change it no matter what setting i'm trying to apply. I need to get 2K in 60 FPS. I think this orange pi is capable of doing it with no effors (spec says 4K 60fps but nobody can get it from HDMI IN so far). It would be great to just plug and play HDMI IN via V4L2 directly to OBS and check what fps will i get then.
Thank you!