[BUG] {Problem with 3D Reconstruction}

[lucianoperetti]

Describe the bug

I have a problem with rebuilding the pointcloud using the latest version of depthai_ros on the commit 0144d79. In the following video, you can see how the reconstruction changes place randomly when the camera is still:

PointcloudNewDriver.webm

I'm also attaching a second video recorded with our fork (made 8 months ago approx.) in which the reconstruction seems to be stable. You can see in the following video:

PontcloudOldDriver1.webm

Minimal Reproducible Example

We use the launch file:

We use the params file:

camera: i_enable_imu: true i_enable_ir: true i_pipeline_type: RGBD i_nn_type: none i_laser_dot_brightness: 1000 i_floodlight_brightness: 700 i_ip: '' i_mx_id: '' i_usb_port_id: '' i_usb_speed: SUPER_PLUS i_pipeline_dump: false i_calibration_dump: false i_external_calibration_path: '' imu: i_acc_mode: linear i_acc_freq: 50 i_acc_cov: 0.003 i_gyro_mode: calibrated i_gyro_freq: 50 i_gyro_cov: 0.003 i_mag_mode: calibrated i_mag_freq: 50 i_enable_rotation: true i_message_type: IMU i_rot_mode: default i_rot_freq: 50 i_rot_cov: 0.003 i_sync_method: LINEAR_INTERPOLATE_ACCEL i_rotation_vector_type: ROTATION_VECTOR rgb: i_board_socket_id: 0 i_fps: 10.0 i_width: 640 i_height: 360 i_resolution: 1080P r_focus: 0 r_set_man_focus: false i_isp_num: 1 left: i_board_socket_id: 1 i_fps: 10.0 i_width: 1280 i_height: 720 i_resolution: 720P i_publish_topic: true right: i_board_socket_id: 2 i_fps: 10.0 i_width: 1280 i_height: 720 i_resolution: 720P i_publish_topic: true stereo: i_align_depth: true i_board_socket_id: 0 i_publish_right_rect: true i_left_socket_id: 1 i_right_socket_id: 2 i_depth_filter_size: 7 i_enable_distortion_correction: true i_enable_speckle_filter: true i_depth_preset: HIGH_ACCURACY i_stereo_conf_threshold: 230 i_subpixel: true i_width: 640 i_height: 360 i_lr_check: true i_extended_disp: true i_disparity_width: DISPARITY_96

Expected behavior

I hope it works like the old driver, which can be seen in the second video.

You can save it in depthai_ros_driver, either by calling /save_pipeline ROS service, or by setting parameter camera.i_pipeline_dump in ROS 2 or camera_i_pipeline_dump in ROS. Pipeline dump is saved to /tmp/pipeline.json.

Attach system log

• Ubuntu 22.04
• ROS2
• Humble

Additional context We have more than 10 oak D pro cameras working well, but in the new version bought in February, we have problem to launch the system with our Fork because there is an inconpatibility between the boot system of the new cameras and the old version of depthai_ros. For these reason we need to update the driver to the new version but we have some problems with the new driver.

[Serafadam]

Hi, thanks for the report, would it be possible for you to share a rosbag with topics available? Also, what is the CPU/RAM usage when the driver is running?

[lucianoperetti]

Hi, thanks for the report, would it be possible for you to share a rosbag with topics available? Also, what is the CPU/RAM usage when the driver is running?

This rosbag contain the main topic from de camera. You can find the pointcloud in the topic /environment_points . If you want to see on rviz, you need to change the qos of /environment_points to best_effort.

rosbag link: https://drive.google.com/drive/folders/1WGnM5Xltjr3b60gvSPU53p0B2VB1XDTg?usp=drive_link

On this picture, you can see the CPU/RAM usage:

[MariuszSzczepanikSpyrosoft]

@Serafadam do you have some update if there is some way to workaround mentioned issue? Maybe we could disable some functionalities on the OAK camera to avoid issues? If you have an idea how to process I would appreciate your sharing it with us.

[Serafadam]

Hi @MariuszSzczepanikSpyrosoft @lucianoperetti, I investigated a bit and unfortunately I wasn't able to replicate the issue, although I might've found a clue - stereo/camera_info topic seems to change randomly and it shouldn't as it is set once at the startup and is being taken from CameraInfoManager, which theoretically could make it possible to change that message in runtime, although this shouldn't happen randomly. I would check if anything calls for updates to calibration or if there is some other publisher that hijacked this topic.

[MariuszSzczepanikSpyrosoft]

@Serafadam currently from our software side: we didn't publish on topic /stereo/camera_info, only reading using the subscription mechanism.

Could you tell me more about: "I would check if anything calls for updates to calibration" and how this could be checked? some dedicated topics messages?

Second question: after upgrading OAK drivers to the newest ones do we need to recalibrate the camera/IMU if old drivers work correctly on the same hardwares?

[Serafadam]

@MariuszSzczepanikSpyrosoft

Could you tell me more about: "I would check if anything calls for updates to calibration" and how this could be checked? some dedicated topics messages?

I would check with rqt_graph how the node graph looks and whether there is another node publishing on the camera_info topic. CameraInfoManager has a service for changing the calibration data, I don't think that on successful calibration it will print logs so you could try build it from source and add some debug prints. Additionally you could build driver from source and print the info message before publishing.

Second question: after upgrading OAK drivers to the newest ones do we need to recalibrate the camera/IMU if old drivers work correctly on the same hardwares?

The calibration values are the same, with one of the future updates we will add IMU extrinsics but that shouldn't affect the driver much.

[MariuszSzczepanikSpyrosoft]

Hi @Serafadam Thank you for your recommendations regarding the rqt_graph and source modifications. We've checked our side and haven't found any publications to the camera_info topic, it appears these might be internal to Luxonis. Could you check if there's been any improvement in the new drivers that would allow us to utilize the latest cameras effectively?

Appreciate your support in resolving this.

[Serafadam]

Hi, could you checkout this branch and see what is the output? Additionally it would be good to compare it with what appears on topic itself

[MateoGiova]

Hi @Serafadam. I tested your branch, but the function where you added the logger never executes. I added a similar logger in the camera configuration function to discard a logger error, but there I do see the message. Basically I added the message below the confirmation that the camera is ready:

RCLCPP_INFO(this->get_logger(), “Camera ready!”); RCLCPP_INFO(rclcpp::get_logger(“oak”), “info width 1: ”);

However, I never see the message coming from the file
depthai_ros_driver/src/dai_nodes/sensors/sensor_helpers.cpp, so I can assume that function is not executed, since I don't see the message, even adding a log outside the if.

[Serafadam]

Hi @MateoGiova, you might need to subscribe to the stereo topic (should be enough to display it using rviz or rqt) for those logs to appear

[MateoGiova]

Hi @Serafadam I subscribe to the stereo topic (and the pointcloud topic) by ros2 topic echo ... and with rviz but the logs never appear.

[Serafadam]

Hi @MateoGiova I have updated the branch to publish the information while using IPC and changed the launch files a bit to enable IPC in them, although I haven't been able to reproduce the issue where two publishers are spawned for /stereo/camera_info, I would suggest starting from minimal launch file and seeing if that behavior can be reproduced with it

[MateoGiova]

Hi @Serafadam i have trouble using your branch right now. It seems there's some issue with the i_decimation_filter_decimation_factor parameter, since it's set to 2, but i'm having the following error:

[component_container_mt-4] terminate called after throwing an instance of 'std::out_of_range' [component_container_mt-4] what(): vector::_M_range_check: __n (which is 57600) >= this->size() (which is 57600) [ERROR] [component_container_mt-4]: process has died [pid 41006, exit code -6, cmd '/opt/ros/humble/lib/rclcpp_components/component_container_mt --ros-args -r node:=pipeline -r ns:=/oak'].

I tried changing it to 3, this error doesn't appear anymore but there are other issues. So I tried to change it to 4 and it gave the same error with a smaller vector, so where it is 57600 it should be 14400.

The only way it works is with 1, but I cannot see anything on recorded rosbags, perhaps this is because the performance since the image isn't reescaled.

These are the parameters of the camera:

camera: i_enable_imu: true i_enable_ir: true i_pipeline_type: rgbd i_nn_type: none i_laser_dot_brightness: 1000 i_floodlight_brightness: 700 i_ip: '' i_mx_id: '' i_usb_port_id: '' i_usb_speed: SUPER_PLUS i_pipeline_dump: false i_calibration_dump: false i_external_calibration_path: '' imu: i_acc_mode: linear i_acc_freq: 50 i_acc_cov: 0.003 i_gyro_mode: calibrated i_gyro_freq: 50 i_gyro_cov: 0.000001 i_mag_mode: calibrated i_mag_freq: 50 i_enable_rotation: true i_rot_mode: default i_rot_freq: 50 i_rot_cov: 0.00005 i_sync_method: LINEAR_INTERPOLATE_ACCEL rgb: i_fps: 10.0 i_width: 640 i_height: 360 i_resolution: '1080' r_focus: 0 r_set_man_focus: false i_isp_num: 1 left: i_fps: 10.0 i_width: 1280 i_height: 720 i_resolution: '720' i_publish_topic: false right: i_fps: 10.0 i_width: 1280 i_height: 720 i_resolution: '720' i_publish_topic: false stereo: i_align_depth: true i_bilateral_sigma: 2 i_depth_filter: 7 i_depth_filter_size: 7 i_enable_brightness_filter: false i_enable_distortion_correction: true i_enable_disparity_shift: true i_enable_spatial_filter: true i_spatial_filter_iterations: 1 i_enable_speckle_filter: true i_enable_temporal_filter: true i_enable_threshold_filter: true i_threshold_filter_min_range: 400 i_threshold_filter_max_range: 8000 i_brightness_filter_max_brightness: 200 i_depth_preset: HIGH_ACCURACY i_stereo_conf_threshold: 230 i_subpixel: true i_width: 640 i_height: 360 i_lr_check: true i_enable_decimation_filter: true i_decimation_filter_decimation_factor: 2 i_extended_disp: true

Hope you could give us a hand. Thank you in advance.

[Serafadam]

Hi, I am unable to reproduce this error, although this config seems to produce some warnings for stereo node. could you try it on the newest humble branch? And additionally please post logs.

[MateoGiova]

Hi, I'm trying to compile the latest version of the humble branch, but it seems that 2 files are missing:

Starting >>> depthai_bridge --- stderr: depthai_bridge /home/apelie/workspace/src/AviSense-Apelie/depthai-ros/depthai_bridge/src/ImageConverter.cpp:6:10: fatal error: depthai-shared/datatype/RawEncodedFrame.hpp: No such file or directory 6	#include "depthai-shared/datatype/RawEncodedFrame.hpp"	^~~~~~~~~~~~~ compilation terminated. gmake[2]: [CMakeFiles/depthai_bridge.dir/build.make:90: CMakeFiles/depthai_bridge.dir/src/ImageConverter.cpp.o] Error 1 gmake[2]: Waiting for unfinished jobs.... gmake[1]: [CMakeFiles/Makefile2:137: CMakeFiles/depthai_bridge.dir/all] Error 2 gmake: [Makefile:146: all] Error 2 Failed <<< depthai_bridge [7.02s, exited with code 2] Summary: 12 packages finished [22.8s] 1 package failed: depthai_bridge 1 package had stderr output: depthai_bridge 14 packages not processed

And this one:

Starting >>> depthai_bridge --- stderr: depthai_bridge
In file included from /home/apelie/workspace/src/AviSense-Apelie/depthai-ros/depthai_bridge/src/ImageConverter.cpp:2: /home/apelie/workspace/src/AviSense-Apelie/depthai-ros/depthai_bridge/include/depthai_bridge/ImageConverter.hpp:13:10: fatal error: depthai/pipeline/datatype/EncodedFrame.hpp: No such file or directory 13 | #include "depthai/pipeline/datatype/EncodedFrame.hpp" | ^~~~~~~~~~~~ compilation terminated. gmake[2]: [CMakeFiles/depthai_bridge.dir/build.make:90: CMakeFiles/depthai_bridge.dir/src/ImageConverter.cpp.o] Error 1 gmake[2]: Waiting for unfinished jobs.... gmake[1]: [CMakeFiles/Makefile2:137: CMakeFiles/depthai_bridge.dir/all] Error 2 gmake: [Makefile:146: all] Error 2 Failed <<< depthai_bridge [11.2s, exited with code 2] Summary: 12 packages finished [8min 50s] 1 package failed: depthai_bridge 2 packages had stderr output: avisense_websocket_client depthai_bridge 14 packages not processed

I tried to find them but I wasn't able to do it. Also another question, is there some way to know the production date of the cameras? We bought cameras on 3 times and we want to know which ones are the latest. Thank you.

[Serafadam]

Hi, which version of depthai-core do you have? Is it installed from APT? Or built from source? As for the production date, you should be able to get it from calibration data on the device

[MateoGiova]

Hi, it seems it's installed from APT. The version is 2.7.4. That's because we are using a fork form an old commit, since that was the stable version of the driver for us. The commit hash is f5e82a1, I don't know if you can see it.

[Serafadam]

What I meant is depthai-core which is available under depthai alias on apt (so ros-humble-depthai, not ros-humble-depthai-ros

[MateoGiova]

Ok, so the version 2.26.1-1jammy.20240612.131023

[Serafadam]

Please check if you don't have additional depthai-core installed on your machine (you can try running ldconfig -p | grep depthai) since EncodedFrames and RawEncodedFrames are available on 2.26. You can also try checking which version the driver builds against with ldd

[MateoGiova]

Adam, I apologize I've made a mistake. The version we are running on the robot is 2.22.0-1jammy.20230624.064826. I checked with ldconfig -p | grep depthai and it doesn't show anything.

[MateoGiova]

Hi Adam, could we schedule a meeting to talk about this topic? Unfortunately the other software developer quitted and I'm currently handling this, and I want to understand better what's going on. I think this would be the fastest and more productive way. Thank you.

[Serafadam]

Hi, I am currently out of office but I will be available from 16.09. Please send me an email with proposed time that would shoot you ( I can be available between 9am -6 pm CEST)