libcamerasrc produces buffers with invalid size on Raspberry Pi 3 with official RPi Cam v2.1 attached

[RSATom]

and it leads to "green" picture.

pi@rpi3:~ $ LIBCAMERA_LOG_LEVELS=7 gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1280,height=720' ! queue ! autovideosink
Setting pipeline to PAUSED ...
[0:20:12.784582040] [1178]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0+120-eb00c13d
[0:20:12.833118956] [1181]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[0:20:12.837026456] [1181]  WARN RPI vc4.cpp:392 Mismatch between Unicam and CamHelper for embedded data usage!
[0:20:12.838046348] [1181]  INFO RPI vc4.cpp:446 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media3 and ISP device /dev/media0
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[0:20:12.847490082] [1185]  INFO Camera camera.cpp:1183 configuring streams: (0) 1280x720-NV21
[0:20:12.848495183] [1181]  INFO RPI vc4.cpp:621 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
Redistribute latency...
handling interrupt.:99.
Interrupt: Stopping pipeline ...
Execution ended after 0:01:52.063719828
Setting pipeline to NULL ...
Freeing pipeline ...

If I'm using gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink it works as expected. ~Also, there is no issue on Raspberry Pi 4 with the same camera module.~ Checked another time - I was wrong. It's broken on both Rpi3 and Rpi4 (I'm not sure why I've thought it's not broken on RPi4, maybe something changed since that try).

On both devices installed the latest Raspbian Bookworm x64.

[RSATom]

It looks like I found the point when it breaks. Right after install fresh Raspberry Pi Os and install all required packages (but without sudo apt update && sudo apt upgrade) it works. But it breaks after sudo apt update && sudo apt upgrade. So I have a feeling it's something related to broken kernel drivers. Going to do rpi-update...

[RSATom]

No, rpi-update didn't help...

[RSATom]

on RPi 4 after rpi-update even gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink stops working.... I see video with very distorted colors (mostly yellow). At the same time libcamera-hello works fine :thinking:

[naushir]

gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink

on a Pi 4 seems to work as expected for me. However, note that I am budling/running libcamera from the next branch in this repo. Can you share an image showing what goes wrong for you?

[RSATom]

@naushir Just to avoid possible side effects I've did everything from scratch :

1. flash Raspberry Pi OS Lite (64-bit) Released 2024-03-15
2. sudo apt update && sudo apt upgrade
3. sudo reboot
4. sudo apt install gstreamer1.0-plugins-{base,good,bad,ugly} gstreamer1.0-tools gstreamer1.0-libcamera -y
5. gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink
6. gst-launch-1.0 libcamerasrc ! queue ! autovideosink

[naushir]

Hmm, with

gst-launch-1.0 libcamerasrc ! queue ! autovideosink

I see the green corruption like you, but with

gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink

everything works fine. I suspect it may be something in the libcamerasrc gst component not setting up a default format and getting confused. I have to admit, I don't know anything about gstreamer, so it's only a guess.

And just to confirm, rpicam-hello gives you a reasonable looking image?

[RSATom]

And just to confirm, rpicam-hello gives you a reasonable looking image?

yes, it works fine.

gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink

It was working for me some time ago too, but now it doesn't. Something changed and I don't know what exactly...

[naushir]

Are you able to build and deploy libcamera yourself for testing purposes. If so, can you try the next branch on this repo, which is what I am using, and see if it works?

[RSATom]

yes I can. I'll try.

[RSATom]

@naushir tried libcamera from next branch (did fresh Raspbian install on SD card before) - result is the same as before for both gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink and gst-launch-1.0 libcamerasrc ! queue ! autovideosink...

[naushir]

And presumably this is with an up-to-date kernel through rpi-update? I can't really see what could be going wrong :(

[RSATom]

And presumably this is with an up-to-date kernel through rpi-update?

Unfortunately I didn't do it. I'll try.

[RSATom]

Tried after rpi-update - didn't help...

[RSATom]

Just for record.

Output from libcamerasrc. In that case image size looks correct, but colors are distorted (as "yellow" pic above):

$ gst-launch-1.0 libcamerasrc ! capsfilter caps=video/x-raw,width=1920,height=1080 ! queue ! autovideosink
Setting pipeline to PAUSED ...
[0:08:33.115217283] [1016]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0+120-eb00c13d
[0:08:33.147332683] [1026]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[0:08:33.149852809] [1026]  WARN RPI vc4.cpp:392 Mismatch between Unicam and CamHelper for embedded data usage!
[0:08:33.150565160] [1026]  INFO RPI vc4.cpp:446 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayGBM\)\ gldisplaygbm0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[0:08:33.160903645] [1029]  INFO Camera camera.cpp:1183 configuring streams: (0) 1920x1080-NV21
[0:08:33.161380070] [1026]  INFO RPI vc4.cpp:621 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
Redistribute latency...
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:48.352883661
Setting pipeline to NULL ...
Freeing pipeline ...

Now output from libcamera-hello for the same resolution (and correct output image):

$ libcamera-hello --width=1920 --height=1080
[0:13:38.056339909] [1070]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0+120-eb00c13d
[0:13:38.103568727] [1073]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[0:13:38.105623169] [1073]  WARN RPI vc4.cpp:392 Mismatch between Unicam and CamHelper for embedded data usage!
[0:13:38.106321292] [1073]  INFO RPI vc4.cpp:446 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
[0:13:38.106392847] [1073]  INFO RPI pipeline_base.cpp:1102 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Made DRM preview window
Mode selection for 1640:922:12:P
    SRGGB10_CSI2P,640x480/0 - Score: 5220.23
    SRGGB10_CSI2P,1640x1232/0 - Score: 2420.22
    SRGGB10_CSI2P,1920x1080/0 - Score: 1112.39
    SRGGB10_CSI2P,3280x2464/0 - Score: 3138.22
    SRGGB8,640x480/0 - Score: 6220.23
    SRGGB8,1640x1232/0 - Score: 3420.22
    SRGGB8,1920x1080/0 - Score: 2112.39
    SRGGB8,3280x2464/0 - Score: 4138.22
Stream configuration adjusted
[0:13:38.211593356] [1070]  INFO Camera camera.cpp:1183 configuring streams: (0) 1640x922-YUV420 (1) 1920x1080-SBGGR10_CSI2P
[0:13:38.212045352] [1073]  INFO RPI vc4.cpp:621 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA

Obvious difference between outputs is:

[0:13:38.106392847] [1073]  INFO RPI pipeline_base.cpp:1102 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
Made DRM preview window
Mode selection for 1640:922:12:P
    SRGGB10_CSI2P,640x480/0 - Score: 5220.23
    SRGGB10_CSI2P,1640x1232/0 - Score: 2420.22
    SRGGB10_CSI2P,1920x1080/0 - Score: 1112.39
    SRGGB10_CSI2P,3280x2464/0 - Score: 3138.22
    SRGGB8,640x480/0 - Score: 6220.23
    SRGGB8,1640x1232/0 - Score: 3420.22
    SRGGB8,1920x1080/0 - Score: 2112.39
    SRGGB8,3280x2464/0 - Score: 4138.22
Stream configuration adjusted

Maybe it's related...

[naushir]

I think this is the main difference: gstreamer: [0:08:33.160903645] [1029] INFO Camera camera.cpp:1183 configuring streams: (0) 1920x1080-NV21

rpicam-hello [0:13:38.211593356] [1070] INFO Camera camera.cpp:1183 configuring streams: (0) 1640x922-YUV420 (1) 1920x1080-SBGGR10_CSI2P

gstreamer is selecting an NV12 format, while rpicam-hello is selecting a planar YUV240 format. The hardware supports NV21 output (and it does seem to work on my system), so this difference should be inconsequential.

[RSATom]

If I remember correctly I've tried to use video/x-raw,width=1920,height=1080,format=I420 - it didn't help. I'll check one more time and will provide output.

[naushir]

Ok, I think I may have some clue:

pi@pi4:~ $ gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1920,height=1080' ! queue ! autovideosink
Setting pipeline to PAUSED ...
[26:13:11.827615845] [5830]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/home/pi/libcamera/build/src/ipa' to the IPA search path
[26:13:11.831605113] [5830]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[26:13:11.938209318] [5844]  INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/pi/libcamera/src/ipa/rpi/vc4/data'
[26:13:11.955848674] [5844]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[26:13:11.966447321] [5844]  WARN RPI vc4.cpp:392 Mismatch between Unicam and CamHelper for embedded data usage!
[26:13:11.968061735] [5844]  INFO RPI vc4.cpp:446 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[26:13:11.970135646] [5848]  INFO RPI pipeline_base.cpp:346  stream res 1920x1080-YUV420 bpl 1920
[26:13:13.248961696] [5848]  INFO RPI pipeline_base.cpp:346  stream res 1280x1080-NV21 bpl 1920
[26:13:13.249276823] [5848]  INFO RPI pipeline_base.cpp:346  stream res 1280x1080-NV21 bpl 1920
[26:13:13.249318749] [5848]  INFO Camera camera.cpp:1183 configuring streams: (0) 1280x1080-NV21
[26:13:13.250261612] [5844]  INFO RPI vc4.cpp:621 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR10_1X10 - Selected unicam format: 1640x1232-pBAA

This works fine for me - no green image.

pi@pi4:~ $ gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=1280,height=1080' ! queue ! autovideosink
Setting pipeline to PAUSED ...
[26:13:45.594504561] [5853]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/home/pi/libcamera/build/src/ipa' to the IPA search path
[26:13:45.598571217] [5853]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[26:13:45.705708025] [5867]  INFO IPAProxy ipa_proxy.cpp:130 libcamera is not installed. Loading IPA configuration from '/home/pi/libcamera/src/ipa/rpi/vc4/data'
[26:13:45.723194493] [5867]  WARN RPiSdn sdn.cpp:40 Using legacy SDN tuning - please consider moving SDN inside rpi.denoise
[26:13:45.733930027] [5867]  WARN RPI vc4.cpp:392 Mismatch between Unicam and CamHelper for embedded data usage!
[26:13:45.735623811] [5867]  INFO RPI vc4.cpp:446 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
[26:13:45.737730296] [5871]  INFO RPI pipeline_base.cpp:346  stream res 1920x1080-YUV420 bpl 1920
[26:13:45.744202750] [5871]  INFO RPI pipeline_base.cpp:346  stream res 1280x1080-NV21 bpl 1920
[26:13:45.744563192] [5871]  INFO RPI pipeline_base.cpp:346  stream res 1280x1080-NV21 bpl 1920
[26:13:45.744635710] [5871]  INFO Camera camera.cpp:1183 configuring streams: (0) 1280x1080-NV21
[26:13:45.745480871] [5867]  INFO RPI vc4.cpp:621 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR10_1X10 - Selected unicam format: 1640x1232-pBAA

This one gives me the green image.

Both output NV12 format output but with different resolutions. However, I think there is actually nothing wrong with the image in the second run, but gstreamer is confused by the stride value.

There has been a recent fix to our kernel driver to respect a stride value given by the userland process if supplied. Previously, we would ignore userland provided stride values which is not correct according to the v4l2 spec. It sounds like gstreamer is not setting the stride correctly, and getting confused as to what the actual stride is when displaying the buffer. A similar problem (and more details) had been reported in the cam app here: https://github.com/raspberrypi/libcamera/issues/138#issuecomment-2146920985

@RSATom, if you set the resolution to 1920x1080 as per my command above, do you get a correct looking image?

[RSATom]

@RSATom, if you set the resolution to 1920x1080 as per my command above, do you get a correct looking image?

Some time ago I did. I'm sure 100%. But now I don't. I don't know what changed. Maybe it was on RPi 3, I don't remember. Now I do testing on RPi 4 since it's required to build lib from sources and RPi 4 is more powerful for that...

[naushir]

INFO RPI pipeline_base.cpp:346  stream res 1280x1080-NV21 bpl 1920

This is the key line in the failure case. Assuming if you try 1920x1080 and it works, then we can narrow the problem down to the 1280x1080 framebuffer stride being incorrectly used by gstreamer in one of 2 ways:

• The stride is actually 1920, but gstreamer is treating it as if it was 1280
• The stride is 1280, but gstreamer is treating it as 1920

I think the correct "fix", like what was suggested in https://github.com/raspberrypi/libcamera/issues/138 is to ensure gstreamer clears the stride value in the StreamConfiguration structure when it configures libcamera. That way the stride reported back is guaranteed to be correct for downstream use.

Unfortunately I know next to nothing about the libcameragst component. Perhaps @kbingham might be able to help here?

[naushir]

I'm now considering not setting a stride in the pipeline handler generateConfiguration() call. The following patch:

diff --git a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp
index 0034e06d5ec2..ab123b22248e 100644
--- a/src/libcamera/pipeline/rpi/common/pipeline_base.cpp
+++ b/src/libcamera/pipeline/rpi/common/pipeline_base.cpp
@@ -490,14 +490,13 @@ PipelineHandlerBase::generateConfiguration(Camera *camera, Span<const StreamRole
                StreamFormats formats(deviceFormats);
                StreamConfiguration cfg(formats);
                cfg.size = size;
+               cfg.stride = 0;
                cfg.pixelFormat = pixelFormat;
                cfg.colorSpace = colorSpace;
                cfg.bufferCount = bufferCount;
                config->addConfiguration(cfg);
        }

-       config->validate();
-
        return config;
 }

fixes the 1280x1080 case for me.

[RSATom]

Unfortunately will not be able to try it today, but I promise I'll do it tomorrow. Thanks!

[kbingham]

I think the correct "fix", like what was suggested in #138 is to ensure gstreamer clears the stride value in the StreamConfiguration structure when it configures libcamera. That way the stride reported back is guaranteed to be correct for downstream use.

Unfortunately I know next to nothing about the libcameragst component. Perhaps @kbingham might be able to help here?

I haven't got all context here yet, but also looking up - I see a patch going to the generateConfiguration/validate() calls. That's where this should be fixed I think - as it's the pipeline handlers responsibility to make sure the configurations are valid.

Applications shouldn't have to know in advance to reset the stride - but they should be able to 'set' it explicitly if they are queueing buffers in from an external source I think ?

Sounds like review topics to handle when Naush posts his patch to the lists though.

[RSATom]

@naushir your patch helped partially. Now size of video is correct no matter what video size I requesting. But colors are still distorted.

I tried on both main and next. But it looks like you did your patch on some different branch (or with some private commits) - since lines are different.

[RSATom]

@naushir please ignore my last complain. Broken colors - it's something wrong with glimagesink without X11 running...

[RSATom]

@naushir now I can confirm your patch definitely fixes issue. I've tested it with app where I initially found issue - and it works great now. The only remaining issue is compatibility with RPi 5 - but it's totally different story. Thank you!

[naushir]

Patch has been posed, so I'll resolve this issue now.