[BUG] depthai_ros_driver camera.launch.py crashes "double free detected in tcache 2"

[FabienDanieau]

Describe the bug When I ran ros2 launch depthai_ros_driver camera.launch.py, it crashes at a random time with the error

[component_container-2] free(): double free detected in tcache 2
[ERROR] [component_container-2]: process has died [pid 55444, exit code -6, cmd '/opt/ros/humble/install/lib/rclcpp_components/component_container --ros-args --log-level debug --ros-args -r __node:=oak_container -r __ns:=/'].

Minimal Reproducible Example

Run ros2 launch depthai_ros_driver camera.launch.py with default parameters.

(stereo_inertial_node in depthai_examples works well.)

Attach system log Logs are here: log.txt log_system_information.json.txt

I am using a Jetson Orin Nano.

[Serafadam]

Hi, is your library installed from apt, or built from source?

[FabienDanieau]

It is built from source. I've just updated depthai-core to 2.22.0, and depthai-ros to 2.7.3 It also happens with ros2 launch depthai_ros_driver example_marker_publish.launch.py, maybe more quickly

[Hitomamacs]

Same problem here

[Serafadam]

Does it also happen with rgbd_pcl.launch.py?

[FabienDanieau]

Looks good on this side

[FabienDanieau]

Any news on this issue? I'd like to use the ROS driver with 2d object detection but it crashes too often

[tkelestemur-bdai]

Same problem here -- I've also installed from source from the latest humble branch.

[Serafadam]

Hi @FabienDanieau, sorry for the delay, I've been trying to reproduce that issue without success so far. Does this happen only on Jetson Orin, or are you able to reproduce it on another system? And is it also present when using libraries from apt/ in Docker container?

[tkelestemur-bdai]

@Serafadam I've encountered this problem on Ubuntu 22.04 with ROS2 Humble. I'm able to run the driver on v2.7.1 but then the point cloud rates is extremely slow. It gives a cloud at every minute or so. I'll give it a try with Docker. I can send the full log if it helps.

[Serafadam]

Sure, it would be nice if you provide information as in bug template. When it comes to the pointcloud rate, that could depend on the network/DDS setup

[BOOTCFG]

Hi, I was experiencing the same issue, both on a Ubuntu 22 laptop and a PI4. After attaching myself with a debugger, I narrowed the error to the IMU interpolation function (to be exact line 182 of ImuConverter.hpp).

The workaround I found is to set the sync method to COPY. Here is my pcl.yaml:

/oak:
  ros__parameters:
    camera:
      i_nn_type: none
    right:
      i_publish_topic: true
    stereo:
      i_align_depth: false
      i_subpixel: true
    imu:
      i_sync_method: COPY

[Serafadam]

Although I'm not sure if this is the cause as stereo_inertial_method uses same sync method, could you try it out @FabienDanieau @tkelestemur-bdai ?

[FabienDanieau]

Indeed I do not have the error with i_sync_method: COPY

[ksharma-bdai]

@Serafadam +1, this workaround resolves the issue for me as well (same lab as @tkelestemur-bdai)

@BOOTCFG nice detective work, thanks!

Also for some additional context if it is useful, this crash only occurs on our OAK-D Pro Wide USB cameras, the non-pro Wide USB version does not crash in the same launch setup.

[Serafadam]

Hi @FabienDanieau, @ksharma-bdai could you check if this branch works for you?

[ksharma-bdai]

Hi @FabienDanieau, @ksharma-bdai could you check if this branch works for you?

@Serafadam I'll try to give it a test today or tomorrow (otherwise I'll be away until next week). Assuming the branch is: https://github.com/luxonis/depthai-ros/tree/imu_sync_fix_humble ?

[Serafadam]

Yes, sorry, for some reason the link didn't display properly

[ksharma-bdai]

Yes, sorry, for some reason the link didn't display properly

Unfortunately it's still crashing when built from source (verified it's running the branch with a few printouts). Happy to run with any debug that could be useful. This was tested with the OAK-D Pro Wide USB.

Our devices will be under test and I'll be away starting Wednesday, but can resume testing next week.

[Serafadam]

I've pushed some additional changes

[ksharma-bdai]

I've pushed some additional changes

No crashes after ~10 minutes on the updated branch with the same OAK-D Pro Wide USB camera (crashes within ~1 minute on apt version). I'm currently only able to verify it does not crash and rgb/depth streams to rviz, I don't have a known good regression test suite - so it would be nice if anyone else can confirm,

Thanks for the updates and let me know if there's anything else I can help with (will be out of contact the rest of this week after today).

[asimonov]

I am also seeing the issue. Just to add - it is on Ethernet version of OAK-D Pro W.

[asimonov]

can you do a fix to noetic version?

[Serafadam]

@asimonov you can try the fix for noetic here

[asimonov]

@asimonov you can try the fix for noetic here

yes, this works without crashes for me. ubuntu 20 inside docker on jetson orin NX with jetpack 5.1.1. Oak-D Pro W Ethernet version.

i am only using mono left/right images and IMU in interpolated mode

[FabienDanieau]

Hi @FabienDanieau, @ksharma-bdai could you check if this branch works for you? Sorry I was away. I'll test it asap.

[Serafadam]

Closing due to inactivity, please reopen if the issues still appear.