Segmentation faults with IMX219

[IGNNE]

Hi,

I am using an IMX219 sensor connected to a RasPi Zero 2. The Pi is running Raspberry OS 12, and ROS humble via the official docker image.

The issue is, sometimes it works, sometimes it crashes with a Segfault. It only crashes when something actually subscribes to the pictures. It seems to be somewhat related to resolution and frame rate, with really low resolutions and rates it sometimes runs for >10 min.

There is nothing suspicious in the console output, even with libcamera debug logging. I have increased CMA to 128M, but I am not sure if that has improved things. Watching /proc/meminfo shows lots of free CMA.

Here is what gdb shows (I seem to be missing the debug symbols?):

Thread 14 "camera_node" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fda6b58e0 (LWP 811)]
0x0000007fdb743058 in libcamera::ipa::RPi::IpaBase::prepareIsp(libcamera::ipa::RPi::PrepareParams const&) () from /opt/ros/humble/lib/libcamera/ipa_rpi_vc4.so
(gdb) where
#0  0x0000007fdb743058 in libcamera::ipa::RPi::IpaBase::prepareIsp(libcamera::ipa::RPi::PrepareParams const&) () from /opt/ros/humble/lib/libcamera/ipa_rpi_vc4.so
#1  0x0000007ff5ceed70 in libcamera::Object::message(libcamera::Message*) () from /opt/ros/humble/lib/libcamera-base.so.0.2
#2  0x0000007ff5cf0ef8 in libcamera::Thread::dispatchMessages(libcamera::Message::Type) () from /opt/ros/humble/lib/libcamera-base.so.0.2
#3  0x0000007ff5cf6d04 in libcamera::EventDispatcherPoll::processEvents() () from /opt/ros/humble/lib/libcamera-base.so.0.2
#4  0x0000007ff5cf4c24 in libcamera::Thread::exec() () from /opt/ros/humble/lib/libcamera-base.so.0.2
#5  0x0000007ff7c031fc in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x0000007ff79cd5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
#7  0x0000007ff7a35edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

This is from the apt version, I have also cloned and built the current master, with the same behavior. Do you know what could be the cause? Is there a good way to install this that does not require me to compile ROS on the Pi (the camera does not work on ubuntu, for some reason)?

[christianrauch]

I am using Ubuntu for this. But I recently also observed crashes inside the RPi ipa. Are you able to build libcamera 0.1 from source to see if this is a regression from 0.1 to 0.2?

[IGNNE]

It looks like libcamera v0.1 works better, I haven't had any crashes so far. However, now the images come very slow (5+ s per image, regardless of parameters).

I want to double check that it is not caused by our setup (distributed ros with a dedicated "camera pi"). But before, framerates worked fine (if they worked at all).

[IGNNE]

Okay, looks like the low frame rate is somehow an issue of distributed ROS. With libcamera v0.1, I can get a stable 1920x1080@10Hz locally, for example. Thanks for the tip!

How would one report this? I guess it is a libcamera problem, right?

[christianrauch]

How would one report this? I guess it is a libcamera problem, right?

You can report libcamera issues via https://bugs.libcamera.org but you would need a minimal reproducer outside of ROS, for example the cam or qcam executables.

[christianrauch]

I built arm64 Debian packages for version 0.1 and 0.2 of libcamera that contain the cam executable: ros-humble-libcamera.zip. You can use them to check if the example executable reads data continuously without issues.

[nbaldy]

Can confirm same behavior with the IMX477. (Crashes on subscribe, works better at slower data rates or lower resolutions)

Trying these libcamera versions now... (Screenshot of GDB backtrace from original libcamera version)

rpicam-apps build: 6202c09b2bfa-intree 19-04-2024 (05:46:57)
libcamera build: v0.2.0

[christianrauch]

I manually compiled the node with libcamera 0.1 and can confirm that it does not crash for me with the older version.

[christianrauch]

I reverted the libcamera version in the ROS repos back to 0.1 and this fixes the issue for me. With the "testing repos" (https://docs.ros.org/en/humble/Installation/Testing.html) you should be able to install the older version again that does not show this crash.

@IGNNE @nbaldy Can you conform this works for you with libcamera 0.1 in the testing repos?

[IGNNE]

Sorry for my late answers, there are some deadlines coming up and I am just happy that I found a setup that "works for me" (manually compiled against libcamera 0.1). I'll try that version as soon as I can, thank you for looking into this!

[barulicm]

I'm hitting the same issue and was able to try the testing version tonight. I can confirm the crash does not occur in the v0.1 version.

Environment:

• RPi 4B
• Camera Module v2 (IMX219)
• Ubuntu 22.04

Node arguments: ros2 run camera_ros camera_node --ros-args -p width:=640 -p height:=480 -p format:=BGR888

In all cases, the subscriber is an instance of ros2 topic hz /camera/image_raw.

Behavior with libcamera v0.2:

• subscribing locally shows messages coming in at 15 Hz
• subscribing from another machine causes the node on the Pi to segfault

Behavior with libcamera v0.1:

• subscribing locally shows messages coming in at 15 Hz
• subscribing from another machine shows messages coming in at 6 Hz

[christianrauch]

I can reproduce the issue without ROS and have reported this as https://bugs.libcamera.org/show_bug.cgi?id=220. However, since the libcamera examples (cam, qcam) run fine on the Raspberry Pi, the issue is still somehow triggered by the way the code is interacting with libcamera.

[christianrauch]

Some race condition probably caused this issue in libcamera 0.2 and it has been fixed since then. Versions 0.1 (released to humble) and 0.3 (released to jazzy) do not have this issue.