Orange pi 5 plus HDMI IN native V4L2 support really needed (OBS crashes)

[MaximKurakin]

Thank you for your tremendous efforts!!! I'm trying to build a re-streaming platform out of orange pi 5 plus with 24.04 Ubuntu from you. But there is a bummber issue like OBS and other apps cannot open HDMI in via v4l2 propertly due to driver IOCTL missing.

So i found another fork where it seems to me all fixed and hdmirx driver patched for rockchip and v4l2 compliance runs with no errors. https://gitlab.collabora.com/hardware-enablement/rockchip-3588/linux/-/issues/4#note_152133 The core issue which crashes OBS is lack of VIDIOC_G_PARM ioctl.

Could you please merge those chages with you build?

Grabbing HDMI IN with gstreamer or ffmpeg is working but i'm getting 30 fps max at 4K resolution and cannot change it no matter what setting i'm trying to apply. I need to get 2K in 60 FPS. I think this orange pi is capable of doing it with no effors (spec says 4K 60fps but nobody can get it from HDMI IN so far). It would be great to just plug and play HDMI IN via V4L2 directly to OBS and check what fps will i get then.

Thank you!

[Joshua-Riek]

Good find, have you confirmed the kernel patch works?

[MaximKurakin]

I'm trying to build the media module right now and test in on my Orange PI 5 PLUS. But it might take a while as soon as i'm really new in kernel drivers building. Struggling with finding proper way to override kernel media module with my module at working ubuntu 24.04.

What did i do:

1. built a m2m-deinterlace.ko (i'm pretty sure this is not what needed). Added VIDIOC_G_PARM handler to drivers/media/platform/rockchip/hdmirx/rk_hdmirx.c.
2. installed it to /lib/modules/6.1.0-1013-rockchip/extra
3. reboot
4. nothing changed.

I really need a good push in right direction =)

[Joshua-Riek]

If you have the kernel build dependencies installed correctly cd into your kernel tree and enter the below to re-build the kernel. You can then install the deb package.

# Setup env
export $(dpkg-architecture -aarm64)
export CROSS_COMPILE=aarch64-linux-gnu-
export CC=aarch64-linux-gnu-gcc
export LANG=C

# Compile the kernel into a deb package
fakeroot debian/rules clean binary-headers binary-rockchip do_mainline_build=true

[Joshua-Riek]

I can try to test on my end in a few hours, i'm currently working on some U-Boot code for some new Radxa boards.

[neofeo]

I am quite sure that in order to get hdmi in to work in obs studio we would need panthor. This is what amazingfate claim. So, should be pinned here.

[MaximKurakin]

Guys. Sorry for delay. I had to do a slight detour to GPIO and FAN settings, because during kernel compilation my orange pi was trying to melt ... So, the progress exist. I'v managed to reduce amount of v4l2-compliance errors from 9 to 7 as of today... OBS still crashes though...

[MaximKurakin]

I am quite sure that in order to get hdmi in to work in obs studio we would need panthor. This is what amazingfate claim. So, should be pinned here.

Well according to the article new GPU driver will be merged with kernel 6.10 (we expect this driver to land in Linux v6.10). But we are sitting on 6.1. I think it will take years to get there... Do you think enabling HDMI IN v4l2 full support completerly impossible with current panfrost driver? Even if i'll fix hdmirx driver for rk3855?

just let it here for future reference https://www.collabora.com/news-and-blog/news-and-events/release-the-panthor.html

[neofeo]

I mean, currently, hdmi in performs really bad in obs wheb used with v4l2 (at least when it worked). The only real way to use it is with a gst pipeline, and that's inside a windowd, not in obs itself. Panthor is meant for mainline, but was backpoorted by boogie to bsp 6.1

[MaximKurakin]

Ok this is my bes effort after 2 weeks of sufferring without sleep, food and life itself ... v4l2-compliance shows 0 errors and 0 warnings! But driver still throttling and dropping frames -((( so i'm getting same 4k30 or 1080p30 no more... OBS still crashes also ... I really need someboody to look at my attemt here https://github.com/MaximKurakin/linux-rockchip.git

I was making test like this: gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! fpsdisplaysink showing only unstable 30 fps at 4K and even at 1080p60 this test also shows 30 FPS! why? external source gives solid 60 fps.

ffmpeg -f v4l2 -video_size 3920x2160 -i /dev/video0 -pix_fmt yuv420p -vf "format=yuv420p" -framerate 60 -c:v rawvideo -vsync 1 -f sdl "SDL output" shows broken picture repeting screen 3 time in a row with broken colors (Trying to reach 0 errors in v4l2-compliance I messed up with yuv420p colospace i think)

I have tried to merge synopsis driver with rockchip because the source code is almost the same in general... But i'm week in DMA flags, bits and fencing... I need somebody more talanted and experienced to look at it. Any help is very much appreciated! I think if we can solve DMA sync issues we will finally fix rockchip driver for good and will get 4k60 as entitled in orange pi 5 plus specification... Nobody touched this driver since 2021... Let's fix it up all together!

[neofeo]

I would love to help, but I am not capable. Did you try to contact amazingfate? The only way I got decent hdmi in was with ximagesynk streaming pipelines and obs recording my desktop, at even then, I never got 1080p60 out of it. He claimed tha tin order to really use it inside obs, so, as a v4l2 source, we had to use panthor and even then it would need more work. You should at least co tact him and boogie at radxa discord, or armbian one.

[dfloer]

@MaximKurakin Did you happen to test HDMI audio input with the Collabora code? It was my understanding that they didn't support it yet.

Also, you might want to look at the obs-gstreamer plugin. If performance is bad, you might be doing colour format conversion on the CPU rather than RGA. I've never tried it directly in OBS, just in gstreamer directly.

[MaximKurakin]

Collabora driver doesn't support audio and HDCP. Thats why I went into madness of merging current rockchip driver with it. I wanted to take only good working parts from both of them and create new complianced driver with full support of video, audio, CEC and HDCP support. I've tried everything I could google =) gstreamer fpsdisplaysink is my main testing tool. The problem is not in the external tools like ffplay, vlc, gstreamer ... the driver itself messes up with DMA timings and my 60 fps turns into 30 because drivers is shouting about massive frame drops internally.

In debug mode drivers gives something like:

[ 1632.249161] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.249164] fdee0000.hdmirx-controller: dma_idle_int_handler: next_buf NULL, skip vb_done!
[ 1632.257834] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1081
[ 1632.257844] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:0
[ 1632.257847] fdee0000.hdmirx-controller: line_flag_int_handler: next_fd:24887296
[ 1632.265836] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.265847] fdee0000.hdmirx-controller: vb_done fd:0
[ 1632.273395] fdee0000.hdmirx-controller: qbuf fd:37330944
[ 1632.274504] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1081
[ 1632.274510] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:24887296
[ 1632.274512] fdee0000.hdmirx-controller: line_flag_int_handler: next_fd:37330944
[ 1632.282495] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.282504] fdee0000.hdmirx-controller: vb_done fd:24887296
[ 1632.291167] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1081
[ 1632.291174] fdee0000.hdmirx-controller: line_flag_int_handler: next_buf NULL, drop the frame!
[ 1632.291176] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:37330944
[ 1632.299162] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.299165] fdee0000.hdmirx-controller: dma_idle_int_handler: next_buf NULL, skip vb_done!
[ 1632.303915] fdee0000.hdmirx-controller: qbuf fd:12443648
[ 1632.307829] fdee0000.hdmirx-controller: dma_irq st1:0x100, st13:1080
[ 1632.307832] fdee0000.hdmirx-controller: line_flag_int_handler: curr_fd:37330944
[ 1632.307834] fdee0000.hdmirx-controller: line_flag_int_handler: next_fd:12443648
[ 1632.315828] fdee0000.hdmirx-controller: dma_irq st1:0x80, st13:2160
[ 1632.315833] fdee0000.hdmirx-controller: vb_done fd:37330944

[MaximKurakin]

@MaximKurakin Did you happen to test HDMI audio input with the Collabora code? It was my understanding that they didn't support it yet.

Also, you might want to look at the obs-gstreamer plugin. If performance is bad, you might be doing colour format conversion on the CPU rather than RGA. I've never tried it directly in OBS, just in gstreamer directly.

I tried this plugin today (had a hard time with unclear dependencies during buld). yes i can capture gstreamer directly to obs now. but fps is still 30 and tonns of dropped frames -( Thank you for you advice anyway!

Also I tried obs beam plugin. (had very sophisticated intercource with build process for arm64). Yes it works but device cannot keep up with 2k 60fps at any jpeg compression quality even in quality 1!!!! black screen always flashing from time to time. All equipment has 2.5 Gbps caps but this orange pi 5 plus really cant keep up with the massive frames comming from network... I don't know if anybody even tried to launch obs-beam on orange pi ... but I did and it doesn't work as expected -((( platform simply doen't have enough performance

[MaximKurakin]

You should at least co tact him and boogie at radxa discord, or armbian one.

Contacted. He only introduced : media: rockchip: hdmirx: add VIDIOC_G_PARM hack fix for Armbian. BUt this fix doesn' work for Jushua Riek build. I tested it already. https://github.com/armbian/linux-rockchip/commit/390072479592339b732a0aa52bdaa14381f19116

But he gave me hope with armbian embedded panthor driver support out of the box. SO, i'm going to give it a spin later today. Maybe with panthor driver overal graphical performance will be better? https://github.com/armbian/linux-rockchip/pull/171

[Joshua-Riek]

Panthor does not have better performance and has quite a few bugs and problems when playing games.

[neofeo]

To me, it has better performance,.overall. but even so, it has better features, and will have much better ones down the road. This is not about performance, but feautres

[MaximKurakin]

Ok. No more ARMBIAN for me!!!! It freezed 3 times in 30 minuts. HDMIRX driver exactly the sam as in Josua build ... 30 fps max. 9 compliance errors an 6 warnings. Morover! OBS doesn't recognize new panthor graphics and doesn't start . THis is a dead end to me. I want to say great thank you to Joshua for this stable and glitch free build. I'll stick with it for a while and will try to contact rk chip developers directly. Because fore the fu8ck sake how the f888ck do you pass unit test for hdmirx driver if it throttles and v4l2-compliance shows 9 errors ? Damn...

[neofeo]

Obs doesnt start bc it needs PAN_MESA_DEBUG=gl3, it's probably forced on panfork by default

[neofeo]

I understand your suffering! Hahah yeah, hdmi in is only really usable from android as today.

[MaximKurakin]

I don't give up with it. I'm just taking a pause to collect more information about HDMI protocol and rk388 chip memory access features. But after 2.5 weeks reading tons of web-sites and manuals ... I need vacation

[neofeo]

I don't give up with it. I'm just taking a pause to collect more information about HDMI protocol and rk388 chip memory access features. But after 2.5 weeks reading tons of web-sites and manuals ... I need vacation

Please contact me at armbian discord. Microlinux(salva)

[dfloer]

I'm definitely interested in collaborating on this, as I've been working with Rockchip SoCs for about a year and a half now for mobile livestreaming.

I've generally found the HDMI input to work well enough, except when it doesn't. And it can be frustrating when it doesn't.

Couple thoughts:

Early on, I was seeing pretty choppy performance, dropped frames, etc, but it seemed like at least some of that wasn't the HDMI input itself's fault, but a downstream element not being fast enough. In my case, the HDMI input was using a pixel format the encoder I was using didn't support, so I need to use a videoconvert element in between the capture and encoder. By default, this seems to be running on the CPU, rather than the RGA hardware (which does support some pixel format conversions). When running on the CPU, it wasn't fast enough to always keep up, especially with 2160p sources, so it'd drop frames as buffers filled up. I was able to get good performance by setting the GST_VIDEO_CONVERT_USE_RGA environment variable to 1, which uses the RGA for hardware acceleration.

The drivers seem very fragile in their current state. Slight disturbances to the kernel, etc, seem to break things. I've had audio just stop working multiple times, and at least at one point, the colours went all weird.

When a briefly looked at OBS support, I was seeing issues due to multi-planar formats. I haven't looked closely, but this patchset may help solve some of those (or other) issues with the RGA and multi-planar formats: https://patchwork.kernel.org/project/linux-media/cover/20230914-rockchip-rga-multiplanar-v1-0-abfd77260ae3@pengutronix.de/

[neofeo]

Nice, I used that env var on my pipelines too and yes, obs seems to not support multiplannar formats. https://github.com/neofeo/BOX86-BOX64-WINEx86-TUTORIAL/blob/main/gst/hdmi_in.sh

[MaximKurakin]

Hmm . This GST_VIDEO_CONVERT_USE_RGA didn't help me. Same frame drops and 30 fps max -((((

More insteresting is OBS crach logs:

sendmsg(11, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\4\0\0\0\0\0\f\08\0\0\0\4\0\0\0\0\0\f\0;\0\0\0\4\0\0\0\1\0\f\0"..., iov_len=708}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 708
futex(0xaaaae62622fc, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xaaaae62622a0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0xaaaae6526c58, FUTEX_WAKE_PRIVATE, 1) = 1
ppoll([{fd=12, events=POLLIN}, {fd=14, events=POLLIN}, {fd=21, events=POLLIN}, {fd=29, events=POLLIN}, {fd=33, events=POLLIN}], 5, {tv_sec=0, tv_nsec=0}, NULL, 0) = 1 ([{fd=14, revents=POLLIN}], left {tv_sec=0, tv_nsec=0})
read(14, "1\0\0\0\0\0\0\0", 8)          = 8
write(14, "\1\0\0\0\0\0\0\0", 8)        = 8
**openat(AT_FDCWD, "/dev/v4l/by-id/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)**
writev(2, [{iov_base="free(): invalid pointer", iov_len=23}, {iov_base="\n", iov_len=1}], 2free(): invalid pointer
) = 24
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffffb60b1000
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
gettid()                                = 5117
getpid()                                = 5117
tgkill(5117, 5117, SIGABRT)             = 0
--- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=5117, si_uid=1000} ---
+++ killed by SIGABRT (core dumped) +++
Aborted (core dumped)

and same time hdmirx shows only this one line:

[  826.694132] fdee0000.hdmirx-controller: hb
[  827.704136] fdee0000.hdmirx-controller: hb
**[  827.897675] fdee0000.hdmirx-controller: tx_5v_power_present: 1**
[  828.720816] fdee0000.hdmirx-controller: hb
[  829.734162] fdee0000.hdmirx-controller: hb

I hope it will give somebody a hint where is the issue

[MaximKurakin]

I haven't looked closely, but this patchset may help solve some of those (or other) issues with the RGA and multi-planar formats: https://patchwork.kernel.org/project/linux-media/cover/20230914-rockchip-rga-multiplanar-v1-0-abfd77260ae3@pengutronix.de/

I'm on it. Will test it in a couple hours...

[MaximKurakin]

I haven't looked closely, but this patchset may help solve some of those (or other) issues with the RGA and multi-planar formats: https://patchwork.kernel.org/project/linux-media/cover/20230914-rockchip-rga-multiplanar-v1-0-abfd77260ae3@pengutronix.de/

I'm on it. Will test it in a couple hours...

Nope... nothing changed. The most surprizing part is that i'm grabbing 1080p60 and getting 30 fps. Then I set 30 hz to the HDMI 2 display out (i'm using orange pi 5 plus as a source and as an input just made a cable loop hdmi2 to hdmi in) and grab it - i'm getting 20 fps... oddd

[MaximKurakin]

More research comming up:

I stopped totruring hdmi in and decided to make a clean perf test of my SBC caps. Here is the result:

1. test with 4K30 gave me more or less stable 30 fps with low drops. GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,framerate=30/1 ! fpsdisplaysink
2. test with 4K60 killed my board and gave me 0 fps ! all frames were dropped! I wonder why? GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,framerate=60/1 ! fpsdisplaysink
3. test with 1080p120 was surprisingly floppy. FPS were jumping betwen 110 and 60 fps. with massive drops. WTF? GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 videotestsrc ! video/x-raw,width=1920,height=1080,framerate=120/1 ! fpsdisplaysink

I did the test wrong way? Or it's my orange pi 5 plus actual limits ?

[dfloer]

@MaximKurakin You might have better luck running videotestsrc is-live=true. I know it sometimes matters, but might not here.

I have previously been able for capture from a 4k60 camera and encode and stream, but it seems highly dependent on kernel (+patches), MPP, RGA, etc libraries.

[MaximKurakin]

Nope. Same behaviour. And I'm testing at clean build 1016 from you. No patches nothing. =((( Myabe orange PI 5 is a piece of sh**t? and I have better to look at radxa boards?

[dfloer]

I don't think it's a hardware issue, because I've definitely had things working well, but then I'll update and things will break again. I'm presently in a state where the 6.1 kernel has no audio but captures+encodes video great, and the 5.10 kernel segfaults whenever I use the RGA to do format conversions, but does audio fine.

Though the HDMI input on the OPi5+ does have issues with HDMI backpowering, so that could point to other implementation issues existing. Who knows...

[MaximKurakin]

Yes encoding and decoding works like a charm with hardware acceleration. I managed to get only 3% CPU usage for my music youtube stream via ffmpeg. That's awesome!!!! But video capturing is such a pain!

[Joshua-Riek]

I would make sure to take note of https://github.com/Joshua-Riek/ubuntu-rockchip/issues/606#issuecomment-2068066423. The Orange Pi 5 Plus does have a known issue with HDMIRX.

[MaximKurakin]

The Orange Pi 5 Plus does have a known issue with HDMIRX.

Except:

• 4K60 capturing doesn't work. Only 30 fps even in 1080p mode!
• 9 errors in v4l2-compliance
• impossible to simply call this device as normal v4l2src. f.e. vlc v4l2://dev/video0 shouting with errors...
• I can continue the list....

I found so many places in the rockchip driver code which are completely oposide to synopsis driver code ... Which makes me think neighter Dingxian Wen shawn.wen@rock-chips.com (author of rockchip hdmirx driver) nor Shreeya Patel shreeya.patel@collabora.com (synopsis driver) doesn't follow IOCTL specs from the manual https://www.kernel.org/doc/html/v6.1/index.html.

Just look: rockchip: hdmirx_dev->cur_vic = val & VIC_VAL_MASK; synopsis: hdmirx_dev->cur_vic = val | VIC_VAL_MASK;

rockchip: ret = hdmirx_get_detected_timings(hdmirx_dev, timings, false); synopsis: ret = hdmirx_get_detected_timings(hdmirx_dev, timings, true);

...

[nyanmisaka]

Just tested it with ffmpeg-rockchip and there is no problem encoding 4k60(nv12) captured from HDMI RX to 1080p60 HEVC video. As for Gstreamer and OBS that's another story.

./ffmpeg -init_hw_device rkmpp -f v4l2 -i /dev/video0 \
-vf "format=nv12|nv16|bgr24,hwupload,scale_rkrga=w=1920:h=1080:format=nv12:afbc=1" \
-c:v hevc_rkmpp -qp_init 18 -g:v 120 -v verbose -f nut - | mpv -

[video4linux2,v4l2 @ 0xaaab18c6cfe0] fd:3 capabilities:84201000
[video4linux2,v4l2 @ 0xaaab18c6cfe0] Querying the device for the current frame size
[video4linux2,v4l2 @ 0xaaab18c6cfe0] Setting frame size to 3840x2160
[video4linux2,v4l2 @ 0xaaab18c6cfe0] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device, estimated framerate 60/1 from dv timings.
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 1301.440382, bitrate: 5971968 kb/s
  Stream #0:0: Video: rawvideo, 1 reference frame (NV12 / 0x3231564E), nv12, 3840x2160, 5971968 kb/s, 60 fps, 60 tbr, 1000k tbn
[out#0/null @ 0xaaab18c6ec80] No explicit maps, mapping streams automatically...
[vost#0:0/hevc_rkmpp @ 0xaaab18c79640] Created video stream from input stream 0:0
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> hevc (hevc_rkmpp))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0xaaab18c7d200] w:3840 h:2160 pixfmt:nv12 tb:1/1000000 fr:60/1 sar:0/1
rga_api version 1.10.0_[8]
[Parsed_scale_rkrga_2 @ 0xaaab18c7cfe0] w:3840 h:2160 fmt:nv12 -> w:1920 h:1080 fmt:nv12
[hevc_rkmpp @ 0xaaab18c79980] Using input frames context (format drm_prime) with hevc_rkmpp encoder.
[hevc_rkmpp @ 0xaaab18c79980] Rate Control mode is set to CQP
[hevc_rkmpp @ 0xaaab18c79980] QP Init/Max/Min/Max_I/Min_I is set to 18/18/18/18/18
[hevc_rkmpp @ 0xaaab18c79980] Profile is set to MAIN
[hevc_rkmpp @ 0xaaab18c79980] Level is set to 0
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: hevc (Main), 1 reference frame, drm_prime(progressive), 1920x1080 (0x0), q=2-31, 2000 kb/s, 60 fps, 60 tbn
    Metadata:
      encoder         : Lavc60.31.102 hevc_rkmpp
[hevc_rkmpp @ 0xaaab18c79980] Configured with size: 1920x1080 | pix_fmt: drm_prime | sw_pix_fmt: nv12
frame= 5809 fps= 60 q=-0.0 size=N/A time=00:01:36.81 bitrate=N/A speed=   1x

[MaximKurakin]

OMG. I've got the same result! FPS is smooth with insignificant frame drops! And no shouting in drivers debug about dropped frames! But input delay is crazy!

So the point is that by doing this (using raw signal is too much for my board?) GST_VIDEO_CONVERT_USE_RGA=1 gst-launch-1.0 videotestsrc ! video/x-raw,width=3840,height=2160,framerate=60/1 ! fpsdisplaysink i'm loosing ability to enable hardware encoding? So the main idea is to do not use RAW signal but encode in first with some codec like h264, hevc with hardware acceleration?

So, how is about to achieve the same result in gstreamer? How to teach this bastard to use hardware encoding? I'm going to use obs-gstreamer plugin to grab directly from gstreamer into OBS studio.

And one more issue is sound goes way before the video. How fix this delay? I can try to fix it in OBS maybe...

[MaximKurakin]

Another issue came from actual streaming test from my windows PC to orange pi. WHen i was testing with a loop cable connection from HDMI 2 to HDMI IN at the same orange PI it could recongnize resolution and colospace normally from HDMI 2. F.e. I set 1080p60 NV24 to HDMI2 and I see tha HDMI IN recognizes it propertly.

BUT. when i do the same from windows PC HDMI IN doesnt' want to change resolution and colospace. It always stays at default 4K60 with N12 colors which is bad because of blury colors and excesive resolution which needs to be encoded. How to teach HDMI IN to get resolution propertly from windows PC? Or, maybe I need to tweak monitor settings at Windows side? I tried to play with color management but with no success.

Technical details:

• WIndows uses "Generic PnP Monitor" driver for this HDMI IN
• Desktop resolution: 1920 x 1080, 60 Hz
• Active signal resolution: 3840 x 2160, 60Hz
• 8-bit signal
• Color forma: YCbCr420
• Colorspace: SDR

On Ubuntu I could set 1080p60 with N24 color format with much smoother color grades. I really don't need 4K in N12 shity colors. I'm ok to drop resolution to 2K but with RGB3 or N24 colors if possible.

[MaximKurakin]

BUT. when i do the same from windows PC HDMI IN doesnt' want to change resolution and colospace. It always stays at default 4K60 with N12 colors which is bad because of blury colors and excesive resolution which needs to be encoded. How to teach HDMI IN to get resolution propertly from windows PC? Or, maybe I need to tweak monitor settings at Windows side? I tried to play with color management but with no success.

I'll answer to myself. Thank you redit! Windows 11 is crazy! To fix Active ACTIVE SIGNAL RESOLUTION you need to jump to 30 hz on this monitor, apply settings. You monitor will jump to RGB mode with correct active signal resolution. Then you just set it back to 60 Hz. It works only for 1080 resolution.

BUT. Another issue: HDMI IN EDID is very limited it has only 1080 and 4k resolutions supported. And nothing in between!!!! Looks like i'll have to create "cusom resolution" for 2k? Or modify hdmirx EDID? Any andvice here?

[MaximKurakin]

And just to pin it here... I found an actual manual for rockchip rk3588! https://gitlab.com/rk3588_linux/linux/bsp/docs/-/blob/linux-5.10-gen-rkr3.5/Linux/Multimedia/Rockchip_User_Guide_Linux_Gstreamer_EN.pdf

taken from FAQ: Shortly: . There is no lagging when playing 4K 30FPS, but there is lagging when playing 4K 60FPS. Due to system load, DDR bandwidth and other issues, 4K 60FPS may not be achieved. You can try to enable AFBC, refer to the AFBC chapter. In addition, the synchronization function of subtitles and sink can be turned off, such as gst-play-1.0 test.mp4 --flags=3 --videosink="waylandsink sync=false" , when the frame rate cannot reach 60FPS, turning on sync will cause the video frame timestamps do not align with clocks resulting in obvious frame drop.

AFBC stands for ARM Frame Buffer Compression, which is a compression format used to save bandwidth. Currently, the encoding formats of AFBC supported by the mppvideodec plugin are: H264, H265, VP9, and the supported color formats are NV12, NV12 10bit, NV16.

# Enable global AFBC，applicable to situations where mppvideodec cannot be
directly operated using some command like gst-play-1.0
export GST_MPP_VIDEODEC_DEFAULT_ARM_AFBC=1
# Enable afbc for current pipeline
gst-launch-1.0 filesrc location=/test.mp4 ! parsebin ! mppvideodec arm-afbc=true
! waylandsink

None of this helped me anyway... I'm stuck at 30 fps -( with text-overlay ON and 45 fps with text-overlay off

This cmd gave me ~45 FPS 4K. GST_MPP_VIDEODEC_DEFAULT_ARM_AFBC have no effect on my chip.

GST_DEBUG=2,fpsdisplaysink:4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1 ! fpsdisplaysink video-sink=xvimagesink signal-fps-measurements=true text-overlay=false sync=false -v

[MaximKurakin]

So, how is about to achieve the same result in gstreamer?

I'll answer to myself ... again.

I stopped trying to get Active Resolution equal to Desktop resolution. And set it to 4x x 2K, 2560 x 1600 instead of native 4K60 at my Windows Desktop. After that HDMI IN recognized me as IMPORTANT 3840/2160 NV12 source. Well f***ck it. At least I could get stable smooth 60 fps in 2K injected to 4k... But the color gradients are killing me...

I've only managed to get stable 2k60 with this gstreamer cmd almost without input delay (unlike ffmpeg where delay was very very very noticeable): GST_DEBUG=2,fpsdisplaysink:4 gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1,pixel-aspect-ratio=1/1,interlace-mode=progressive ! fpsdisplaysink video-sink=waylandsink signal-fps-measurements=true text-overlay=false sync=false -v

IMPORTANT: sync=false gives performance boost very significant! IMPORTANT: using waylandsink give 2 to 3 % CPU load only! Tremendously great performance! xvimagesink a little bit worse but also doable!

Trying to do the same but in native 4k set to windows display led me to 4k60 but only in fpsdisplaysink logs. But with my eyes i saw that it's about 30 and v-sync was off so much that i could see screen tearing. Same was for both waylandsink and xvimagesink.

Next step to try it directly into obs-gstreamer plugin pipeline! gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1,pixel-aspect-ratio=1/1,interlace-mode=progressive ! waylandsink sync=false

Any other tests with custom 2K60 NV16, NV24, RGB resolutions gave me only pain and 12 to 20 fps because I always have to use videoconvert to pipeline the signal to the output. Otherwise i'm getting not negotiated issue in gstreamer. I'm pretty sure it's something about color spaces other then NV12. Any thoughts on that?

[MaximKurakin]

Ok. some final words about orange PI 5 Plus. I've done hundreds of test about streaming 2k in 60 fps... grabbing camera and so on. The results are quite frustrating. Experimentally I found out that board can handle 2K60 streaming to youtube in 14000 Kb\s with gstreamer Encoder H.264 encode type Rockchip MPP H264 only if CPU load is less then 30%. If you attach droidcam OBS as I did you get crytical situation where board start throttling you stream FPS down to ~50+... and after couple minuetes youtube starts shouting than frames are coming inconsistently and blah blah blah...

So, this board is good to only singe task usage! Like:

1. grabbing HDMI IN with hardare acceletated FFMPEG in 4K60 to the file or stream.
2. grabbing camera in HD 60 with high bit rate.
3. streaming to youtube without camera in 2K or with camera but in 1080p

but if you try to combine tastk like I did (camera HD60 + gstreamer xvimagesink 2K60 + stream to youtube in 2K60) board will be dropping FPS. As I see it now it's a DMA issue. Driver sends frames without drops to OBS via gstream (finally !!!!!) but something happends after it. OBS need to combine signal from camera and xvimagesink window and encode it with h26 rkmpp gstreamer codec and send to youtube. This task appears to much for OPI 5 PLUS =(((

SO, I don't go even deeper into this madness of transcodding and gst negotiation essues between userspace and driverspace... I'm pretty happy with the result anyway. But streaming in 2k60 i'll left for my Windows PC with NVIDIA hardware transcoding and restreaming to 3 platforms. This boy does this with no efforts...

Next, I'm taking rest. 3 weeks of 20 hours working a day killed me... After I will post my hdmirx patch to the public and maybe somebody will get fresh ideas from my work. The driver code is such an ass ! Believe me there are so many stubbs in the code which breakes normal IOCTL behaviour. IF not completely, but partially i'm going to reducre amount of v4l2-compliance issues from 9 to idealy 0.

Any help in this complicated task will be much appreciated.

P.S. I contacted boogie couple days ago ... still no answer. But maybe i'll find someboby else. Let's see. The issues like this: rockchip: hdmirx_dev->cur_vic = val & VIC_VAL_MASK; synopsis: hdmirx_dev->cur_vic = val | VIC_VAL_MASK;

rockchip: ret = hdmirx_get_detected_timings(hdmirx_dev, timings, false); // take dv timings from CTL synopsis: ret = hdmirx_get_detected_timings(hdmirx_dev, timings, true); // take dv timings from DMA

must be addressed and fix evetually. How come that bitmasking is done in an oposite way and driver still works...

Thank you everybody for you kind support and help!

[MaximKurakin]

Next step to try it directly into obs-gstreamer plugin pipeline! gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=NV12,height=2160,width=3840,framerate=60/1,pixel-aspect-ratio=1/1,interlace-mode=progressive ! waylandsink sync=false

Shortly. No OBS-gstreamer video. sink cannot handle 2K60. OBS looks choppy and it's even before you start streamin. Whe you do start streamin FPS dropps even more and video looks like 30 FPS. Impossible to stream games like this.

[dfloer]

I gave it another try with the same camera that I've used before on my OPi5+ and a fresh 24.04 image, and I wasn't able to get a reasonable framerate with a 2160p30 signal, plus I also seem to have developed an issue where NV16 video from my camera produces an encode with lots of little lines when there's motion, even at 1080p30.

[BUP-BIP-BOP]

I was able to add HDMI RX to OBS

[MaximKurakin]

Great !! I'll try tomorrow on my orange pi. But i see 28 fps and massive CPU load. Did you try to actully stream it? What was the fps then? MY result ~53 fps. With gstreamer I have 3% idle CPU load captured to OBS via Windows capture. So fat it's the best what I could get from it.

[benhoff]

Would you mind posting instructions for building your module? I've cloned your repo, but haven't messed with this stuff much, so I'm at step -1 for staring out.

I bought an orange pi 5 plus, and sort of expected that the hdmi input would just work.

[MaximKurakin]

Are you sure you want to go into it? If you want to help with development then start from here but if you need just use just install joshua build:

install ubuntu kernel build dependencies

sudo apt install build-essential
sudo apt-get install libncurses-dev flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf

Then create build.sh script with this text:

# Setup env
export $(dpkg-architecture -aarm64)
export CROSS_COMPILE=aarch64-linux-gnu-
export CC=aarch64-linux-gnu-gcc
export LANG=C

### Compile the kernel into a deb package
fakeroot debian/rules clean binary-headers binary-rockchip do_mainline_build=true

then:

chmod +x build.sh

then execute it and wait for 50 minuts! FInd some cooling for orange PI because it will go up to 70+ during compilation!

./build.sh

then deploy all 5 dpkg packages to you system

sudo dpkg -i linux-buildinfo-6.1.0-1016-rockchip_6.1.0-1016.16_arm64.deb
sudo dpkg -i linux-image-6.1.0-1016-rockchip_6.1.0-1016.16_arm64.deb
sudo dpkg -i linux-modules-6.1.0-1016-rockchip_6.1.0-1016.16_arm64.deb
sudo dpkg -i linux-headers-6.1.0-1016-rockchip_6.1.0-1016.16_arm64.deb
sudo dpkg -i linux-rockchip-headers-6.1.0-1016_6.1.0-1016.16_arm64.deb

Short instruction to activate HDMI IN (read more about ubuntu device-tree overlays) :

sudo nano /etc/default/u-boot
U_BOOT_FDT_OVERLAYS="device-tree/rockchip/overlay/rk3588-hdmirx.dtbo"
sudo u-boot-update

Bu default nothing is activated . 40 pin connector PWMs also disabled. You have to enable exactly what you need using Overlays.

If you wondering how to burn system image to eMMS, SSD, TF card then read this manual it explains every type of deployment https://drive.google.com/file/d/1TfUG9zrp8q2ZceLrxXYWGj3JfnR4l0Ji/view

HDMI IN "Just works" only in 2 original builds: Android 12 and Orangepi5plus_1.0.8_ubuntu_jammy_desktop_xfce_linux5.10.160.7z.

Android 12 - useless because HDMIRX app can only show you the output of HDMI IN to the screen. WOrks as a charm but useless because there is no software capable to grab HDMI IN as i found. Only streaming embedded cameras and USB cameras.

Orangepi5plus_1.0.8_ubuntu_jammy_desktop_xfce_linux5.10.160.7z - has HDMIRX Enabled out of the box. Has 11 degrees more board temperature in idle ! WTF? IN my case 55 degrees ! Has build in tests for hdmi in using gstreamer. Can grab 4k60 using gstreamer to xvimagesing. OBS can detect V4l2 device but cannot get video out of it (erros about cannot open device and so on) BUT! useless because: has no wayland and OBS cannot find glx-something graphics and uses x11 - so OBS in idle generated 20 % CPU load - and it's even before you grab you hdmi rx into it...

I've tested about 15 distributives within this month (armbian, joshua build, debian, android12...) There is only one which works the best out of the box - Joshua build!

By the way just yesterday they posted this giant banner at the product page. So looks like finally orange PI 5 plus slow memory issue bottle neck will be fixed? I really hope that it's true.

[MaximKurakin]

About OBS... There was a good advice to use obs-gstreamer plugin!. But it creates like 15% CPU load anyways, so I don't use it. BUT by compiling and installing it you will get FREE benefit - OBS will enlist gstreamer encoder with rochchip support in the interface! We all know that hardware encoding is much better then standard h264 encoder which uses only CPU.

[MaximKurakin]

I've managed to compile obs NDI for orange PI 5 plus even if it Runtime doesn't exist for it yet! So the results are frustrating 4 FPS only with preset: Highest! crap...

But with preset lowest I have stable 60 fps 14% cpu load (but 60 only in the OBS tray, but with my eyes I saw something chopy and more like 30+ fps). but picture is unusable -(((( Where is other presets I wonder? Like medium, high, low? hmmmm.

Also I compiled OBS-beam also bad results here -(((((((((((((((((((((((((((((((((( Network connection always resets even if I set the lowest jpeg qty preset! FFFFFFFFFF***k. No way you can use it in real life. Black screen blinking and constant re-connections and 20% cpu load... dead end also.

As for now, no way we can utilize 2.5 Gbit lan port.

[ramiropolla]

I'm working on upstreaming support for HDMI input for the Orange Pi 5 Plus to FFmpeg.

I didn't try the driver from Collabora, since it looks like it's still a work-in-progress. But we definitely need the VIDIOC_G_PARM ioctl. I opened a PR here: https://github.com/Joshua-Riek/linux-rockchip/pull/18

I didn't manage to change the pixel format for HDMI input, so I ended up having to rebuild the kernel whenever I wanted to test different pixel formats. There is still a long way to go before the driver is fully operational.

[jaffenaegel]

Hello all, sorry to be a complete noob at this but just like benhoff I kind of bought an OPi5+ and expected it to just work lol. I'm not 100% sure that I'm correct to post this here so if I'm wrong let me know and if you can redirect me to a place where I can handle my issue that would be great, anyway here I go.

My goal is to have a headless OPi5+ that can read HDMI input (video only, 30fps @ 1920x1080, not really the most demanding) in C++ using opencv. I read somewhere online it would just show up as a video device but that's not how it went down.

This is my code:

#include <opencv2/core.hpp>
#include <opencv2/videoio.hpp>
#include <iostream>

using namespace cv;

int main()
{
    VideoCapture cap("/dev/video0");
    if (!cap.isOpened())
    {
        std::cout << "Doesn't work :(" << std::endl;
    }
}

I get something like:

[ WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (2075) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module source reported: could not read from resource.
[WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (1053) open OpenCV | GStreamer warning: unable to start pipeline
[WARN:0] global ./modules/videoio/src/cap_gstreamer.cpp (616) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Doesn't work :(

GStreamer says that /dev/video0 is not a device though...

using:

v4l2-ctl -d /dev/video0 --get-fmt-video

I can get the configuration of the hdmi input (laptop connected as a test)

Then I do this command:

ffmpeg -f v4l2 -input_format BGR3 -video_size 1920x1080 -i /dev/video0 -vcodec libx264 output.mp4

However the output of this command ends with:

[video4linux2 @ 0x5596cd4bc0] Not a video capture device.
/dev/video0: No such device

So I'm at a total loss on how to do this tbh. I have enabled hdmirx and I can see /dev/video0...

Again, I'm not the best developer out there, still I hope to simply get all the pixel data in c++. I hope I'm not wasting your time, I was just wondering if this is the reason why it isn't working.