Kernel 6 and gstreamer OpenGL is unusable slow?

[chrisiberg]

with Kernel 5.10 and Kernel 6 there is the same error.

DISPLAY=:0 glxinfo | grep "OpenGL renderer"
libGL error: MESA-LOADER: failed to open rockchip: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/dri, suffix _dri)
libGL error: failed to load driver: rockchip
libGL error: MESA-LOADER: failed to open rockchip: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/dri, suffix _dri)
libGL error: failed to load driver: rockchip
OpenGL renderer string: softpipe

when i use this gstreamer pipeline

gst-launch-1.0 v4l2src device=/dev/video0 ! glupload ! glcolorconvert ! glcolorscale ! video/x-raw(memory:GLMemory), format=RGBA, width=1920, height=1080 ! gldownload ! glimagesink

it's super fast with nearly no CPU usage with Kernel 5.10 but with Kernel 6 it is unusable slow?

Do you have an idea Jeffy?

[chrisiberg]

downgrading to gstreamer 1.20.7 did not help.

[JeffyCN]

if let me guess, there might be a memcpy() in v4l2src -> glupload, the v4l2src might provide non-dmabuf to glupload, which would cause a memcpy() when uploading userspace buf to texture in mali DDK.

you can try to "export GST_DEBUG=v4l2bufferpool:5" to check the logs.

and you can try to workaround it by hard-coding can_share_own_pool to 1 in gst1-plugins-good/sys/v4l2/gstv4l2object.c

[chrisiberg]

Thnx I will try this.

the v4l2src is the HDMI-IN of rk3588 "hdmirx"

[chrisiberg]

DISPLAY=:0 gst-launch-1.0 v4l2src device=/dev/video0 ! glupload ! glcolorconvert ! glcolorscale ! 'video/x-raw(memory:GLMemory)', format=RGBA, width=3840, height=2160 ! gldownload ! glimagesink > /tmp/gst
MESA-LOADER: failed to open rockchip: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/dri, suffix _dri)
failed to load driver: rockchip
MESA-LOADER: failed to open rockchip: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/dri, suffix _dri)
failed to load driver: rockchip
0:00:00.237009918 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:545:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> config GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\,\ format\=\(string\)BGR\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)120/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:1:7:1", size=(uint)6220800, min-buffers=(uint)0, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)NULL;
0:00:00.237044917 32549   0x7f5c000b70 INFO          v4l2bufferpool gstv4l2bufferpool.c:593:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> increasing minimum buffers to 2
0:00:00.237056291 32549   0x7f5c000b70 INFO          v4l2bufferpool gstv4l2bufferpool.c:606:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> reducing maximum buffers to 64
0:00:00.237101789 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:545:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> config GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\,\ format\=\(string\)BGR\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)120/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:1:7:1", size=(uint)6220800, min-buffers=(uint)3, max-buffers=(uint)0, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)NULL, options=(string)< GstBufferPoolOptionVideoMeta >;
0:00:00.237122205 32549   0x7f5c000b70 INFO          v4l2bufferpool gstv4l2bufferpool.c:606:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> reducing maximum buffers to 64
0:00:00.237149621 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:545:gst_v4l2_buffer_pool_set_config:<v4l2src0:pool0:src> config GstBufferPoolConfig, caps=(GstCaps)"video/x-raw\,\ format\=\(string\)BGR\,\ width\=\(int\)1920\,\ height\=\(int\)1080\,\ framerate\=\(fraction\)120/1\,\ interlace-mode\=\(string\)progressive\,\ colorimetry\=\(string\)2:1:7:1", size=(uint)6220800, min-buffers=(uint)3, max-buffers=(uint)64, allocator=(GstAllocator)"NULL", params=(GstAllocationParams)NULL, options=(string)< GstBufferPoolOptionVideoMeta >;
0:00:00.237199202 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:791:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> activating pool
0:00:00.237210285 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:838:gst_v4l2_buffer_pool_start:<v4l2src0:pool0:src> requesting 3 MMAP buffers
0:00:00.240749225 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1610:gst_v4l2_buffer_pool_complete_release_buffer:<v4l2src0:pool0:src> complete release buffer 0x7f540091b0 (queued = no)
0:00:00.240784224 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1610:gst_v4l2_buffer_pool_complete_release_buffer:<v4l2src0:pool0:src> complete release buffer 0x7f54009500 (queued = no)
0:00:00.240804056 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1610:gst_v4l2_buffer_pool_complete_release_buffer:<v4l2src0:pool0:src> complete release buffer 0x7f54008620 (queued = no)
0:00:00.240848680 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:709:gst_v4l2_buffer_pool_streamon:<v4l2src0:pool0:src> Started streaming
0:00:00.240857721 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1103:gst_v4l2_buffer_pool_flush_stop:<v4l2src0:pool0:src> stop flushing
0:00:00.240878137 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1513:gst_v4l2_buffer_pool_acquire_buffer:<v4l2src0:pool0:src> acquire
0:00:00.359070317 32549   0x7f5c000b70 WARN          v4l2bufferpool gstv4l2bufferpool.c:1397:gst_v4l2_buffer_pool_dqbuf:<v4l2src0:pool0:src> Driver should never set v4l2_buffer.field to ANY
0:00:00.359116107 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1973:gst_v4l2_buffer_pool_process:<v4l2src0:pool0:src> process buffer 0x7f540091b0
0:00:01.026645149 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1610:gst_v4l2_buffer_pool_complete_release_buffer:<v4l2src0:pool0:src> complete release buffer 0x7f540091b0 (queued = no)
0:00:04.710216143 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1513:gst_v4l2_buffer_pool_acquire_buffer:<v4l2src0:pool0:src> acquire
0:00:04.710265141 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1973:gst_v4l2_buffer_pool_process:<v4l2src0:pool0:src> process buffer 0x7f54009500
0:00:09.336884524 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1610:gst_v4l2_buffer_pool_complete_release_buffer:<v4l2src0:pool0:src> complete release buffer 0x7f54009500 (queued = no)
0:00:10.971165072 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1513:gst_v4l2_buffer_pool_acquire_buffer:<v4l2src0:pool0:src> acquire
0:00:10.971284650 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1973:gst_v4l2_buffer_pool_process:<v4l2src0:pool0:src> process buffer 0x7f54008620
0:00:10.971375647 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1610:gst_v4l2_buffer_pool_complete_release_buffer:<v4l2src0:pool0:src> complete release buffer 0x7f54008620 (queued = no)
0:00:10.971431644 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1513:gst_v4l2_buffer_pool_acquire_buffer:<v4l2src0:pool0:src> acquire
0:00:10.971485892 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1973:gst_v4l2_buffer_pool_process:<v4l2src0:pool0:src> process buffer 0x7f540091b0
0:00:11.548104578 32549   0x7f5c000b70 DEBUG         v4l2bufferpool gstv4l2bufferpool.c:1610:gst_v4l2_buffer_pool_complete_release_buffer:<v4l2src0:pool0:src> complete release buffer 0x7f540091b0 (queued = no)

[JeffyCN]

there're some hack patches for glupload to import dmabuf directly without memcpy(): https://github.com/JeffyCN/meta-rockchip/tree/master/recipes-multimedia/gstreamer/gstreamer1.0-plugins-base_1.22.9

make sure that they are applied.

export GST_DEBUG=glupload:5 and check log: attempting upload with uploader DirectDmabufExternal

[chrisiberg]

is it possible, that the symbolic links from 1.22.9 to 1.22.11 don't work?

[JeffyCN]

normally it should work, since you said the 1.20.7 also failed.

maybe you can check the build/tmp/work//gstgood/*/temp/log.do_patch about which patches been applied

[chrisiberg]

I freshly cloned meta-rockchip, because I already did some mess.

I had to remove the patch recipes-kernel/linux/linux-rockchip_6.1/0004-media-rockchip-vicap-fixes-error-of-gcc-compile.patch because it seems this is already done here. https://github.com/friendlyarm/kernel-rockchip/blob/nanopi6-v6.1.y/drivers/media/platform/rockchip/cif/capture.c

gst patches seem all to be done. But the problem is not solved.

GST_DEBUG=glupload:5

Setting pipeline to PAUSED ...
MESA-LOADER: failed to open rockchip: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/dri, suffix _dri)
failed to load driver: rockchip
MESA-LOADER: failed to open rockchip: /usr/lib/dri/rockchip_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/dri, suffix _dri)
failed to load driver: rockchip
Pipeline is live and does not need PREROLL ...
Got context from element 'sink': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:00.322878966 15931   0x7f80000b70 DEBUG               glupload gstglupload.c:2551:_upload_find_method:<glupload1> attempting upload with uploader GLMemory
0:00:00.322915117 15931   0x7f80000b70 DEBUG               glupload gstglupload.c:2551:_upload_find_method:<glupload1> attempting upload with uploader DirectDmabufExternal
0:00:00.322928237 15931   0x7f80000b70 DEBUG               glupload gstglupload.c:2551:_upload_find_method:<glupload1> attempting upload with uploader DirectDmabuf
0:00:00.322938149 15931   0x7f80000b70 DEBUG               glupload gstglupload.c:2551:_upload_find_method:<glupload1> attempting upload with uploader Dmabuf
0:00:00.322947770 15931   0x7f80000b70 DEBUG               glupload gstglupload.c:2551:_upload_find_method:<glupload1> attempting upload with uploader UploadMeta
0:00:00.322961764 15931   0x7f80000b70 DEBUG               glupload gstglupload.c:2551:_upload_find_method:<glupload1> attempting upload with uploader Raw Data
0:00:02.516616933 15931   0x7f80000b70 DEBUG               glupload gstglupload.c:2551:_upload_find_method:<glupload0> attempting upload with uploader GLMemory
Redistribute latency...

[JeffyCN]

you can upload your rootfs.img, maybe i can try to check it.

[chrisiberg]

I sent a fast shot, I hope you can use it. https://drive.google.com/file/d/14DGc65808G1IiVn1UdAjgSbutAawavkE/view?usp=drive_link

[JeffyCN]

try: export GST_V4L2_MIN_BUFS=24

could be buffer not enough

[JeffyCN]

another try is to skip sync in kernel:

+++ b/drivers/media/common/videobuf2/videobuf2-cma-sg.c
@@ -242,7 +242,7 @@ static void vb2_cma_sg_prepare(void *buf_priv)
        struct vb2_cma_sg_buf *buf = buf_priv;
        struct sg_table *sgt = buf->dma_sgt;

-       if (buf->vb->skip_cache_sync_on_prepare)
+       if (1 || buf->vb->skip_cache_sync_on_prepare)
                return;

the proper way is to set V4L2_BUF_FLAG_NO_CACHE_CLEAN in the v4l2src.

[JeffyCN]

Also try to set FlipFB to none in: /usr/share/X11/xorg.conf.d/20-modesetting.conf

which would affect xserver render performance.

[chrisiberg]

Nothing helped. I missed to try "can_share_own_pool to 1 in gst1-plugins-good/sys/v4l2/gstv4l2object.c" will try now.

[JeffyCN]

so the first a few frames in glimagesink is slow(due to x11 window initialization), the gst will try to sync them and causing those delays.

you can try to add sync=false in your pipeline's glimagesink to avoid syncing (with system clock or audio)

[chrisiberg]

gst-launch-1.0 v4l2src device=/dev/video0 ! glupload ! glcolorconvert ! glcolorscale ! 'video/x-raw(memory:GLMemory)', format=RGBA, width=3840, height=2160 ! gldownload ! glimagesink sync=false

no change, the framrate is about 1 in 5sec

I s it possible there is a problem with hdmirx?

[JeffyCN]

could be, you can try to test it with video: gst-launch-1.0 uridecodebin uri=file:///

[chrisiberg]

ok, not hdmirx, same problem with a video file.

[JeffyCN]

it works well on my side with camera and video.

i have these environments: export DISPLAY=:0 export GST_GL_API="gles2" export GST_GL_PLATFORM="egl" export GST_V4L2_MIN_BUFS=16 unset GST_V4L2_USE_LIBV4L2

and with above hacks.

[chrisiberg]

Thnx a lot, this works! But I have to work out which on of these environments did it.

[chrisiberg]

I have to set both export GST_GL_API="gles2" export GST_GL_PLATFORM="egl"

[chrisiberg]

are they usual set in a script, did I mess up something?

[chrisiberg]

I will rebuild all without the patches, and try it again.

[JeffyCN]

the Mali DDK(GPU library) doesn't support OpenGL, it only support EGL and OpenGLESv2, so if the gst has OpenGL enabled, it might use mesa3d(software compositor in this case).

[JeffyCN]

You can set these in /etc/profile.d/gst.sh, or set it in the gst-plugin-base's configure arguments:

EXTRA_OEMESON += " \ -Ddoc=disabled \ ${@get_opengl_cmdline_list('gl_api', d.getVar('OPENGL_APIS'), d)} \ ${@get_opengl_cmdline_list('gl_platform', d.getVar('OPENGL_PLATFORMS'), d)} \ ${@get_opengl_cmdline_list('gl_winsys', d.getVar('OPENGL_WINSYS'), d)} \ "