os: Ubuntu 22.04 but ros is in a docker: Ubuntu 20.04

/rosdistro: noetic
/rosversion: 1.16.0

I have a D455, I use it combined with rtabmap to generate a .bag file, and then reconstruct the point cloud .ply file, put the ply into Blender and use the /tf information to reconstruct the motion of the virtual camera. In theory, the rendered image generated from Blender should be consistent with the rgb image (for example, if there is a cup in an rgb image, then the cup in the corresponding rendered image should be in the same position). However, the fact is that the position of the object in the rendered image has a certain degree of offset compared to the real position.

181

For example, in this image, I merged the rendered image with the rgb image, and set the transparency of the rendered image to 50%, and found that the objects were not aligned.

Related Information:

.bag's tf tree Screenshot_20230707_084644

Camera's tf tree tf_tree

rtabmap rviz looks like: Screenshot_20230707_084546

Observing the offset direction and the tf tree, I guess that there may be two types of offset:

Because the position corresponding to the camera_optical_color_frame used is not the position of the rgb camera, but the position of the left sensor (looking forward from the back).
The saved rgb image is the image obtained by the left sensor, not the generated interpolation image (pure guesswork).

So I'd like to start with the first guess. If I can get other tf information, maybe the problem of offset can be solved. However, I am a ros newbie and I have made quite a few attempts in camera driven launch and rtabmap launch, however unsuccessful, so here is my current launch file, please help me, how should I modify them?

handheld_realsense_driver.launch

<?xml version="1.0"?>
<launch>

    <arg name="l515" value="false"/>
    <arg name="d435i" value="false"/>
    <arg name="d455" value="true"/>

    <!-- https://desk.zoho.com/portal/dotproduct/en/kb/articles/d455connection -->
    <!-- Make sure USB-C connector at camera is upright for SuperSpeed mode -->
    <group if="$(arg d455)">
        <include file="$(find realsense2_camera)/launch/rs_camera.launch">
            <arg name="clip_distance" value="4.0"/>
            <arg name="align_depth" value="true"/>
            <!-- <arg name="frame_id" value="camera_color_frame"/> -->
            <!-- <arg name="filters" value="decimation,temporal"/> -->

            <arg name="unite_imu_method" value="linear_interpolation"/>
            <arg name="enable_gyro" value="true"/>
            <arg name="enable_accel" value="true"/>
            <arg name="enable_sync" value="true"/>
            <arg name="color_width" value="640"/>
            <arg name="color_height" value="480"/>
            <arg name="color_fps" value="30"/>
            <arg name="depth_width" value="640"/>  <!--L515: 1024x768, 640x480, 320x240-->
            <arg name="depth_height" value="480"/>
            <arg name="depth_fps" value="30"/>
            <!-- <param name="camera_info_url" type="string" value="file:///$(find cronos_rtabmap)/launch/ost.yaml"/> -->
            <!-- <remap from="camera_info_url" to="/rec/info" /> -->
         </include>
    </group>

    <node name="imu_filter_node" pkg="imu_filter_madgwick" type="imu_filter_node" output="screen">
        <param name="use_mag" type="bool" value="false"/>
        <param name="publish_tf" type="bool" value="false"/>
        <!-- <param name="publish_tf" type="bool" value="true"/> -->
        <param name="world_frame" type="string" value="enu"/>
        <remap from="/imu/data_raw" to="/camera/imu"/>
        <remap from="/imu/data" to="/rtabmap/imu" />
    </node>

</launch>

handheld_mapping.launch

<?xml version="1.0"?>
<launch>
    <node pkg="tf" type="static_transform_publisher" name="tf_publisher" args="0 0 0 0 0 0 camera_link camera_color_frame 100" />

    <include file="$(find rtabmap_ros)/launch/rtabmap.launch">
        <arg name="rtabmap_args"        value= "--delete_db_on_start --Rtabmap/DetectionRate 2  RGBD/LinearUpdate 0.01  RGBD/AngularUpdate 0.01" />
        <arg name="use_sim_time"        value="false"/>
        <arg name="rtabmapviz"          value="true" />
        <arg name="rviz"                value="false" />

        <arg name="depth_topic"         value="/camera/aligned_depth_to_color/image_raw" />
        <arg name="rgb_topic"           value="/camera/color/image_raw" />
        <arg name="camera_info_topic"   value="/camera/color/camera_info" />
        <arg name="depth_camera_info_topic" value="/camera/aligned_depth_to_color/camera_info" />

        <arg name="approx_sync"         value="false" />
        <arg name="wait_imu_to_init"    value="true"  />
        <arg name="imu_topic"           value="/rtabmap/imu" />
        <!-- <arg name="imu_topic"           value="/camera/imu" /> -->

        <arg name="frame_id"            value="camera_link"/>
        <arg name="map_frame_id"        value="map"/>
    </include>
</launch>

Hi @ydzat The Intel RealSense team member who created the ROS1 wrapper describes the difference between camera_color_frame and camera_optical_color_frame at https://github.com/IntelRealSense/realsense-ros/issues/743

After the RealSense launch (rs_camera.launch) has completed, you should be able to retrieve the camera_color_frame TF values by inputting the echo instruction below.

rosrun tf tf_echo /camera_link /camera_color_frame

When importing depth information into Blender via a ply, the measurement scale of Blender (or any other 3D modelling tool that a RealSense ply is imported into) should be set to the same depth unit scale that RealSense uses, otherwise the ply may be mis-scaled after import. The default depth unit scale of the D455 camera is 0.001.

Hi @ydzat The Intel RealSense team member who created the ROS1 wrapper describes the difference between camera_color_frame and camera_optical_color_frame at #743

After the RealSense launch (rs_camera.launch) has completed, you should be able to retrieve the camera_color_frame TF values by inputting the echo instruction below.

rosrun tf tf_echo /camera_link /camera_color_frame

When importing depth information into Blender via a ply, the measurement scale of Blender (or any other 3D modelling tool that a RealSense ply is imported into) should be set to the same depth unit scale that RealSense uses, otherwise the ply may be mis-scaled after import. The default depth unit scale of the D455 camera is 0.001.

Thank you. tf_echo can indeed return this information. But ...... D455

Here's a image depicting a few different coordinate systems in D455 (I don't know the source of this image)

As you can see, the camera_link is on the far left, and as the image in my previous problem description shows, "the rendered image is slightly offset relative to the rgb image", so I'm guessing that perhaps the camera_color_frame in this image is the one I should be getting.

So how do I subscribe to that coordinate information in the launch file?

camera_link is at the center-line of the left infrared sensor, as this is the 0,0,0 origin coordinate of depth. This is on the right side of the camera when viewing it from the front like in the image above.

The RGB sensor is offset horizontally from the left infrared sensor's position. On the D455 model it is on the opposite side of the camera beside the right infrared sensor, second to the left when viewing the camera from the front).

base_link corresponds to the tripod screw-thread hole on the bottom of the camera.

My knowledge of TF is admittedly limited. So in regard to your question "how do I subscribe to that coordinate information in the launch file", I unfortunately do not know.

In the guide at the link below for connecting a D455 to ROS though, the author seems to be having a similar situation to yours and states the following:

I was looking for TF from camera_color_frame to base_link, and found that the point group coordinate system is camera_depth_optical_frame, and the xyz direction is different after calculation (I didn't notice the difference, I was confused) so x = t_z, y = - t_x, z = - t_y

https://www.hackster.io/Elephant-Robotics-Official/operate-mycobot-with-the-spatial-recognition-of-d455-7f298e#toc-6----connecting-mycobot-to-realsense-d455-using-ros-12

The author's solution was apparently to rewrite values in a file called tf_broadcaster.cpp, though this may not be applicable to your particular project.

# Put the obtained values in t_x,y,z and theta_1,_2,_3 respectively 
    transformStamped.transform.translation.x = t_z; 
    transformStamped.transform.translation.y = -t_x; 
    transformStamped.transform.translation.z = -t_y; 

    tf2::Quaternion q; 
    q.setEuler(theta_1, theta_2, theta_3);

This filename is referenced in the transformation tutorial below.

https://husarion.com/tutorials/ros-tutorials/6-transformation-in-ROS/#broadcasting-transformation

Various tutorials online describe that tf_broadcaster.cpp is a file that ROS users can create themselves in their catkin workspace's /src folder.

I apologize that I could not be of more assistance on this particular question.

camera_link is at the center-line of the left infrared sensor, as this is the 0,0,0 origin coordinate of depth. This is on the right side of the camera when viewing it from the front like in the image above.

The RGB sensor is offset horizontally from the left infrared sensor's position. On the D455 model it is on the opposite side of the camera beside the right infrared sensor, second to the left when viewing the camera from the front).

base_link corresponds to the tripod screw-thread hole on the bottom of the camera.

My knowledge of TF is admittedly limited. So in regard to your question "how do I subscribe to that coordinate information in the launch file", I unfortunately do not know.

In the guide at the link below for connecting a D455 to ROS though, the author seems to be having a similar situation to yours and states the following:

I was looking for TF from camera_color_frame to base_link, and found that the point group coordinate system is camera_depth_optical_frame, and the xyz direction is different after calculation (I didn't notice the difference, I was confused) so x = t_z, y = - t_x, z = - t_y

https://www.hackster.io/Elephant-Robotics-Official/operate-mycobot-with-the-spatial-recognition-of-d455-7f298e#toc-6----connecting-mycobot-to-realsense-d455-using-ros-12

The author's solution was apparently to rewrite values in a file called tf_broadcaster.cpp, though this may not be applicable to your particular project.
# Put the obtained values in t_x,y,z and theta_1,_2,_3 respectively 
    transformStamped.transform.translation.x = t_z; 
    transformStamped.transform.translation.y = -t_x; 
    transformStamped.transform.translation.z = -t_y; 

    tf2::Quaternion q; 
    q.setEuler(theta_1, theta_2, theta_3); 
This filename is referenced in the transformation tutorial below.

https://husarion.com/tutorials/ros-tutorials/6-transformation-in-ROS/#broadcasting-transformation

Various tutorials online describe that tf_broadcaster.cpp is a file that ROS users can create themselves in their catkin workspace's /src folder.

I apologize that I could not be of more assistance on this particular question.

I've noticed issues with different xyz orientations, but apparently that's not what's causing the image to appear offset... Screenshot_20230708_123435

I use rviz to view the tf after the subscriber's launch starts, as shown in the image. The one on the left is camera_link, the one on the right is camera_color_optical_frame. (camera_color_frame actually coincides with camera_color_optical_frame origin, the direction is different, as you said). There are several other frames in rviz's tf, but the positions are between camera_color_optical_frame and camera_link. This shows that the subscribed camera_color_optical_frame should be exactly the frame I need in theory, but there is a problem when writing the .bag, which causes the written tf information to be incorrect. In order to verify this guess, I recorded a small .bag file.

Start the camera driver
Start the launch of rtabmap
rostopic echo /tf
```
frame_id: map
child_frame_id: odom
```

frame_id: odom child_frame_id: camera_link


4. python script, used to save as .bag file
5. `rosrun rqt_tf_tree rqt_tf_tree` & `rosbag play rgbdp_data.bag`
![Screenshot_20230708_125554](https://github.com/IntelRealSense/realsense-ros/assets/50231148/1701f1bb-9412-4e2c-a931-ad272f667673)

6. `rostopic echo /tf` & `rosbag play rgbdp_data.bag`

frame_id: "map" child_frame_id: "camera_color_optical_frame"


You can see that there is no camera_color_optical_frame in the 3 step, so I guess there is a problem in this step?

The launch file of the second step is:

<?xml version="1.0"?>

Rather than use a Python script to record the TF to a bag file, a more common approach may be to use the instruction rosbag record /tf to record the TF to a bag with ROS' own rosbag record tool.

Rather than use a Python script to record the TF to a bag file, a more common approach may be to use the instruction rosbag record /tf to record the TF to a bag with ROS' own rosbag record tool.

right....but for reasons of my project, I only have the option to use python for writing. When I used realsense + rtabmap to record before, I noticed that the terminal corresponding to the launch of the driver output several warnings, so I wondered if this was the problem, so I completely reinstalled it according to the realsense tutorial--finally, Now there are no errors or warnings when running the camera, however the result remains the same.

Here are the steps I took to install the driver:

For hardware compatibility reasons, my laptop can only install Ubuntu 22.04, and for realsense kernel patching requirements, I downgrade the kernel to 5.15.0-75-generic;
But I need ROS Noetic as a development environment, so Docker is the final choice;
Docker installs librealsense/librealsense:latest, and the container is named ros04
Install ros noetic in ros04
Install realsense-ros, because the driver version in librealsense/librealsense:latest is lower, so realsense-ros-2.2.24 must be downloaded;
According to the installation steps, git checkoutgit tag | sort -V | grep -P "^2.\d+.\d+" | tail -1`` is skipped;
root@ros03host:/# roslaunch cronos_rtabmap handheld_realsense_driver.launch

... logging to /root/.ros/log/a9690b02-1e54-11ee-bc29-0242acac0165/roslaunch-ros03host-220.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ros03host:33843/

SUMMARY
========

PARAMETERS
 * /camera/realsense2_camera/accel_fps: -1
 * /camera/realsense2_camera/accel_frame_id: camera_accel_frame
 * /camera/realsense2_camera/accel_optical_frame_id: camera_accel_opti...
 * /camera/realsense2_camera/align_depth: True
 * /camera/realsense2_camera/aligned_depth_to_color_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /camera/realsense2_camera/allow_no_texture_points: False
 * /camera/realsense2_camera/base_frame_id: camera_link
 * /camera/realsense2_camera/calib_odom_file: 
 * /camera/realsense2_camera/clip_distance: 4.0
 * /camera/realsense2_camera/color_fps: 30
 * /camera/realsense2_camera/color_frame_id: camera_color_frame
 * /camera/realsense2_camera/color_height: 480
 * /camera/realsense2_camera/color_optical_frame_id: camera_color_opti...
 * /camera/realsense2_camera/color_width: 640
 * /camera/realsense2_camera/confidence_fps: -1
 * /camera/realsense2_camera/confidence_height: -1
 * /camera/realsense2_camera/confidence_width: -1
 * /camera/realsense2_camera/depth_fps: 30
 * /camera/realsense2_camera/depth_frame_id: camera_depth_frame
 * /camera/realsense2_camera/depth_height: 480
 * /camera/realsense2_camera/depth_optical_frame_id: camera_depth_opti...
 * /camera/realsense2_camera/depth_width: 640
 * /camera/realsense2_camera/device_type: 
 * /camera/realsense2_camera/enable_accel: True
 * /camera/realsense2_camera/enable_color: True
 * /camera/realsense2_camera/enable_confidence: True
 * /camera/realsense2_camera/enable_depth: True
 * /camera/realsense2_camera/enable_fisheye1: False
 * /camera/realsense2_camera/enable_fisheye2: False
 * /camera/realsense2_camera/enable_fisheye: False
 * /camera/realsense2_camera/enable_gyro: True
 * /camera/realsense2_camera/enable_infra1: False
 * /camera/realsense2_camera/enable_infra2: False
 * /camera/realsense2_camera/enable_infra: False
 * /camera/realsense2_camera/enable_pointcloud: False
 * /camera/realsense2_camera/enable_pose: False
 * /camera/realsense2_camera/enable_sync: True
 * /camera/realsense2_camera/filters: 
 * /camera/realsense2_camera/fisheye1_frame_id: camera_fisheye1_f...
 * /camera/realsense2_camera/fisheye1_optical_frame_id: camera_fisheye1_o...
 * /camera/realsense2_camera/fisheye2_frame_id: camera_fisheye2_f...
 * /camera/realsense2_camera/fisheye2_optical_frame_id: camera_fisheye2_o...
 * /camera/realsense2_camera/fisheye_fps: -1
 * /camera/realsense2_camera/fisheye_frame_id: camera_fisheye_frame
 * /camera/realsense2_camera/fisheye_height: -1
 * /camera/realsense2_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /camera/realsense2_camera/fisheye_width: -1
 * /camera/realsense2_camera/gyro_fps: -1
 * /camera/realsense2_camera/gyro_frame_id: camera_gyro_frame
 * /camera/realsense2_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /camera/realsense2_camera/imu_optical_frame_id: camera_imu_optica...
 * /camera/realsense2_camera/infra1_frame_id: camera_infra1_frame
 * /camera/realsense2_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /camera/realsense2_camera/infra2_frame_id: camera_infra2_frame
 * /camera/realsense2_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /camera/realsense2_camera/infra_fps: 30
 * /camera/realsense2_camera/infra_height: 480
 * /camera/realsense2_camera/infra_rgb: False
 * /camera/realsense2_camera/infra_width: 848
 * /camera/realsense2_camera/initial_reset: False
 * /camera/realsense2_camera/json_file_path: 
 * /camera/realsense2_camera/linear_accel_cov: 0.01
 * /camera/realsense2_camera/odom_frame_id: camera_odom_frame
 * /camera/realsense2_camera/ordered_pc: False
 * /camera/realsense2_camera/pointcloud_texture_index: 0
 * /camera/realsense2_camera/pointcloud_texture_stream: RS2_STREAM_COLOR
 * /camera/realsense2_camera/pose_frame_id: camera_pose_frame
 * /camera/realsense2_camera/pose_optical_frame_id: camera_pose_optic...
 * /camera/realsense2_camera/publish_odom_tf: True
 * /camera/realsense2_camera/publish_tf: True
 * /camera/realsense2_camera/rosbag_filename: 
 * /camera/realsense2_camera/serial_no: 
 * /camera/realsense2_camera/stereo_module/exposure/1: 7500
 * /camera/realsense2_camera/stereo_module/exposure/2: 1
 * /camera/realsense2_camera/stereo_module/gain/1: 16
 * /camera/realsense2_camera/stereo_module/gain/2: 16
 * /camera/realsense2_camera/tf_publish_rate: 0.0
 * /camera/realsense2_camera/topic_odom_in: odom_in
 * /camera/realsense2_camera/unite_imu_method: linear_interpolation
 * /camera/realsense2_camera/usb_port_id: 
 * /imu_filter_node/publish_tf: False
 * /imu_filter_node/use_mag: False
 * /imu_filter_node/world_frame: enu
 * /rosdistro: noetic
 * /rosversion: 1.16.0

NODES
  /
    imu_filter_node (imu_filter_madgwick/imu_filter_node)
  /camera/
    realsense2_camera (nodelet/nodelet)
    realsense2_camera_manager (nodelet/nodelet)

ROS_MASTER_URI=http://localhost:11311

process[camera/realsense2_camera_manager-1]: started with pid [234]
process[camera/realsense2_camera-2]: started with pid [235]
process[imu_filter_node-3]: started with pid [236]
[ INFO] [1688905968.492073618]: Starting ImuFilter
[ INFO] [1688905968.493418820]: Using dt computed from message headers
[ INFO] [1688905968.493428938]: The gravity vector is kept in the IMU message.
[ INFO] [1688905968.495364960]: Imu filter gain set to 0.100000
[ INFO] [1688905968.495370737]: Gyro drift bias set to 0.000000
[ INFO] [1688905968.495376404]: Magnetometer bias values: 0.000000 0.000000 0.000000
[ INFO] [1688905968.498205322]: Initializing nodelet with 16 worker threads.
[ INFO] [1688905968.787848861]: RealSense ROS v2.2.24
[ INFO] [1688905968.787912475]: Built with LibRealSense v2.44.0
[ INFO] [1688905968.787939509]: Running with LibRealSense v2.44.0
[ INFO] [1688905968.814329098]:  
[ INFO] [1688905968.828236452]: Device with serial number 043422250498 was found.

[ INFO] [1688905968.828261027]: Device with physical ID /sys/devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3:1.0/video4linux/video2 was found.
[ INFO] [1688905968.828266627]: Device with name Intel RealSense D455 was found.
[ INFO] [1688905968.828432084]: Device with port number 2-3 was found.
[ INFO] [1688905968.828442090]: Device USB type: 3.2
[ INFO] [1688905968.830201514]: getParameters...
[ INFO] [1688905968.843207668]: setupDevice...
[ INFO] [1688905968.843217919]: JSON file is not provided
[ INFO] [1688905968.843222428]: ROS Node Namespace: camera
[ INFO] [1688905968.843232198]: Device Name: Intel RealSense D455
[ INFO] [1688905968.843238538]: Device Serial No: 043422250498
[ INFO] [1688905968.843250451]: Device physical port: /sys/devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3:1.0/video4linux/video2
[ INFO] [1688905968.843255169]: Device FW version: 05.15.00.02
[ INFO] [1688905968.843260586]: Device Product ID: 0x0B5C
[ INFO] [1688905968.843266902]: Enable PointCloud: Off
[ INFO] [1688905968.843272165]: Align Depth: On
[ INFO] [1688905968.843276218]: Sync Mode: On
[ INFO] [1688905968.843326765]: Device Sensors: 
[ INFO] [1688905968.860179722]: Stereo Module was found.
[ INFO] [1688905968.874968646]: RGB Camera was found.
[ INFO] [1688905968.875550906]: Motion Module was found.
[ INFO] [1688905968.875575176]: (Confidence, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1688905968.876339756]: num_filters: 1
[ INFO] [1688905968.876350843]: Setting Dynamic reconfig parameters.
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
[ INFO] [1688905968.898301688]: Done Setting Dynamic reconfig parameters.
[ INFO] [1688905968.898687637]: depth stream is enabled - width: 640, height: 480, fps: 30, Format: Z16
[ INFO] [1688905968.898969614]: color stream is enabled - width: 640, height: 480, fps: 30, Format: RGB8
[ INFO] [1688905968.899607466]: gyro stream is enabled - fps: 200
[ INFO] [1688905968.899619827]: accel stream is enabled - fps: 63
[ INFO] [1688905968.899628668]: setupPublishers...
[ INFO] [1688905968.900773210]: Expected frequency for depth = 30.00000
[ INFO] [1688905968.918959376]: Expected frequency for color = 30.00000
[ INFO] [1688905968.925459792]: Expected frequency for aligned_depth_to_color = 30.00000
[ INFO] [1688905968.931706408]: Start publisher IMU
[ INFO] [1688905968.932101600]: setupStreams...
[ INFO] [1688905968.932989755]: insert Depth to Stereo Module
[ INFO] [1688905968.933017631]: insert Color to RGB Camera
[ INFO] [1688905968.933029455]: insert Gyro to Motion Module
[ INFO] [1688905968.933040479]: insert Accel to Motion Module
 09/07 12:32:48,935 WARNING [139997156103936] (ds5-motion.cpp:473) IMU Calibration is not available, default intrinsic and extrinsic will be used.
[ INFO] [1688905968.969844779]: SELECTED BASE:Depth, 0
[ INFO] [1688905968.990320563]: RealSense Node Is Up!
[ WARN] [1688905969.072537359]: 
[ INFO] [1688905969.180867654]: First IMU message received.

root@ros03host:/# roslaunch cronos_rtabmap handheld_mapping.launch


... logging to /root/.ros/log/a9690b02-1e54-11ee-bc29-0242acac0165/roslaunch-ros03host-329.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ros03host:36011/

SUMMARY

CLEAR PARAMETERS

/rtabmap/rgbd_odometry/
/rtabmap/rtabmap/
/rtabmap/rtabmap_viz/

PARAMETERS

/rosdistro: noetic
/rosversion: 1.16.0
/rtabmap/rgbd_odometry/approx_sync: True
/rtabmap/rgbd_odometry/approx_sync_max_interval: 0.0
/rtabmap/rgbd_odometry/config_path:
/rtabmap/rgbd_odometry/expected_update_rate: 0.0
/rtabmap/rgbd_odometry/frame_id: camera_link
/rtabmap/rgbd_odometry/ground_truth_base_frame_id:
/rtabmap/rgbd_odometry/ground_truth_frame_id:
/rtabmap/rgbd_odometry/guess_frame_id:
/rtabmap/rgbd_odometry/guess_min_rotation: 0.0
/rtabmap/rgbd_odometry/guess_min_translation: 0.0
/rtabmap/rgbd_odometry/keep_color: False
/rtabmap/rgbd_odometry/max_update_rate: 0.0
/rtabmap/rgbd_odometry/odom_frame_id: odom
/rtabmap/rgbd_odometry/publish_tf: True
/rtabmap/rgbd_odometry/queue_size: 10
/rtabmap/rgbd_odometry/subscribe_rgbd: False
/rtabmap/rgbd_odometry/wait_for_transform_duration: 0.2
/rtabmap/rgbd_odometry/wait_imu_to_init: True
/rtabmap/rtabmap/Mem/IncrementalMemory: true
/rtabmap/rtabmap/Mem/InitWMWithAllNodes: false
/rtabmap/rtabmap/approx_sync: True
/rtabmap/rtabmap/config_path:
/rtabmap/rtabmap/database_path: ~/.ros/rtabmap.db
/rtabmap/rtabmap/frame_id: camera_link
/rtabmap/rtabmap/gen_depth: False
/rtabmap/rtabmap/gen_depth_decimation: 1
/rtabmap/rtabmap/gen_depth_fill_holes_error: 0.1
/rtabmap/rtabmap/gen_depth_fill_holes_size: 0
/rtabmap/rtabmap/gen_depth_fill_iterations: 1
/rtabmap/rtabmap/gen_scan: False
/rtabmap/rtabmap/ground_truth_base_frame_id:
/rtabmap/rtabmap/ground_truth_frame_id:
/rtabmap/rtabmap/initial_pose:
/rtabmap/rtabmap/landmark_angular_variance: 9999.0
/rtabmap/rtabmap/landmark_linear_variance: 0.0001
/rtabmap/rtabmap/map_frame_id: map
/rtabmap/rtabmap/odom_frame_id:
/rtabmap/rtabmap/odom_frame_id_init:
/rtabmap/rtabmap/odom_sensor_sync: False
/rtabmap/rtabmap/odom_tf_angular_variance: 0.001
/rtabmap/rtabmap/odom_tf_linear_variance: 0.001
/rtabmap/rtabmap/publish_tf: True
/rtabmap/rtabmap/queue_size: 10
/rtabmap/rtabmap/scan_cloud_max_points: 0
/rtabmap/rtabmap/subscribe_depth: True
/rtabmap/rtabmap/subscribe_odom_info: True
/rtabmap/rtabmap/subscribe_rgb: True
/rtabmap/rtabmap/subscribe_rgbd: False
/rtabmap/rtabmap/subscribe_scan: False
/rtabmap/rtabmap/subscribe_scan_cloud: False
/rtabmap/rtabmap/subscribe_scan_descriptor: False
/rtabmap/rtabmap/subscribe_stereo: False
/rtabmap/rtabmap/subscribe_user_data: False
/rtabmap/rtabmap/wait_for_transform_duration: 0.2
/rtabmap/rtabmap_viz/approx_sync: True
/rtabmap/rtabmap_viz/frame_id: camera_link
/rtabmap/rtabmap_viz/odom_frame_id:
/rtabmap/rtabmap_viz/queue_size: 10
/rtabmap/rtabmap_viz/subscribe_depth: True
/rtabmap/rtabmap_viz/subscribe_odom_info: True
/rtabmap/rtabmap_viz/subscribe_rgb: True
/rtabmap/rtabmap_viz/subscribe_rgbd: False
/rtabmap/rtabmap_viz/subscribe_scan: False
/rtabmap/rtabmap_viz/subscribe_scan_cloud: False
/rtabmap/rtabmap_viz/subscribe_scan_descriptor: False
/rtabmap/rtabmap_viz/subscribe_stereo: False
/rtabmap/rtabmap_viz/wait_for_transform_duration: 0.2

NODES /rtabmap/ rgbd_odometry (rtabmap_odom/rgbd_odometry) rtabmap (rtabmap_slam/rtabmap) rtabmap_viz (rtabmap_viz/rtabmap_viz)

ROS_MASTER_URI=http://localhost:11311

process[rtabmap/rgbd_odometry-1]: started with pid [344] process[rtabmap/rtabmap-2]: started with pid [345] process[rtabmap/rtabmap_viz-3]: started with pid [346] [ INFO] [1688906079.663113040]: Starting node... [ INFO] [1688906079.688218839]: Initializing nodelet with 16 worker threads. [ INFO] [1688906079.714620498]: Initializing nodelet with 16 worker threads. [ INFO] [1688906079.759870527]: Starting node... QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' [ INFO] [1688906079.783605965]: /rtabmap/rtabmap(maps): map_filter_radius = 0.000000 [ INFO] [1688906079.783632336]: /rtabmap/rtabmap(maps): map_filter_angle = 30.000000 [ INFO] [1688906079.783639725]: /rtabmap/rtabmap(maps): map_cleanup = true [ INFO] [1688906079.783647210]: /rtabmap/rtabmap(maps): map_always_update = false [ INFO] [1688906079.783653898]: /rtabmap/rtabmap(maps): map_empty_ray_tracing = true [ INFO] [1688906079.783660183]: /rtabmap/rtabmap(maps): cloud_output_voxelized = true [ INFO] [1688906079.783667206]: /rtabmap/rtabmap(maps): cloud_subtract_filtering = false [ INFO] [1688906079.783675151]: /rtabmap/rtabmap(maps): cloud_subtract_filtering_min_neighbors = 2 [ INFO] [1688906079.784228435]: /rtabmap/rtabmap(maps): octomap_tree_depth = 16 [ INFO] [1688906079.788771083]: rtabmap_viz: Using configuration from "/root/.ros/rtabmap_gui.ini" [ INFO] [1688906079.788887544]: Odometry: frame_id = camera_link [ INFO] [1688906079.788912817]: Odometry: odom_frame_id = odom [ INFO] [1688906079.788924246]: Odometry: publish_tf = true [ INFO] [1688906079.788935194]: Odometry: wait_for_transform = true [ INFO] [1688906079.788955162]: Odometry: wait_for_transform_duration = 0.200000 [ INFO] [1688906079.788965075]: Odometry: log_to_rosout_level = 4 [ INFO] [1688906079.789256207]: Odometry: initial_pose = xyz=0.000000,0.000000,0.000000 rpy=0.000000,-0.000000,0.000000 [ INFO] [1688906079.789277207]: Odometry: ground_truth_frame_id = [ INFO] [1688906079.789284092]: Odometry: ground_truth_base_frame_id = [ INFO] [1688906079.789290077]: Odometry: config_path = [ INFO] [1688906079.789296321]: Odometry: publish_null_when_lost = true [ INFO] [1688906079.789306636]: Odometry: guess_frame_id = [ INFO] [1688906079.789315179]: Odometry: guess_min_translation = 0.000000 [ INFO] [1688906079.789321363]: Odometry: guess_min_rotation = 0.000000 [ INFO] [1688906079.789326311]: Odometry: guess_min_time = 0.000000 [ INFO] [1688906079.789332464]: Odometry: expected_update_rate = 0.000000 Hz [ INFO] [1688906079.789342219]: Odometry: max_update_rate = 0.000000 Hz [ INFO] [1688906079.789351565]: Odometry: min_update_rate = 0.000000 Hz [ INFO] [1688906079.789359752]: Odometry: wait_imu_toinit = true [ INFO] [1688906079.789386177]: Odometry: stereoParams=0 visParams=1 icpParams=0 [ INFO] [1688906079.803666134]: rtabmap: frame_id = camera_link [ INFO] [1688906079.803687169]: rtabmap: map_frame_id = map [ INFO] [1688906079.803694129]: rtabmap: log_to_rosout_level = 4 [ INFO] [1688906079.803704826]: rtabmap: initial_pose = [ INFO] [1688906079.803715382]: rtabmap: use_action_for_goal = false [ INFO] [1688906079.803728034]: rtabmap: tf_delay = 0.050000 [ INFO] [1688906079.803735531]: rtabmap: tf_tolerance = 0.100000 [ INFO] [1688906079.803742482]: rtabmap: odom_sensor_sync = false [ INFO] [1688906079.804087239]: rtabmap: gen_scan = false [ INFO] [1688906079.804092908]: rtabmap: gen_depth = false [ INFO] [1688906079.915121250]: Setting RTAB-Map parameter "Mem/IncrementalMemory"="true" [ INFO] [1688906079.915282125]: Setting RTAB-Map parameter "Mem/InitWMWithAllNodes"="false" [ INFO] [1688906079.938924891]: Odometry: Ignored parameter "RGBD/AngularUpdate"="0.01" from arguments [ INFO] [1688906079.938940266]: Odometry: Ignored parameter "RGBD/LinearUpdate"="0.01" from arguments [ INFO] [1688906079.938944657]: Odometry: Ignored parameter "Rtabmap/DetectionRate"="2" from arguments [ INFO] [1688906080.032710490]: Update RTAB-Map parameter "RGBD/AngularUpdate"="0.01" from arguments [ INFO] [1688906080.032736371]: Update RTAB-Map parameter "RGBD/LinearUpdate"="0.01" from arguments [ INFO] [1688906080.032744535]: Update RTAB-Map parameter "Rtabmap/DetectionRate"="2" from arguments [ INFO] [1688906080.040090044]: odometry: Subscribing to IMU topic /rtabmap/imu [ INFO] [1688906080.042127431]: RGBDOdometry: approx_sync = true [ INFO] [1688906080.042140605]: RGBDOdometry: approx_sync_max_interval = 0.000000 [ INFO] [1688906080.042147173]: RGBDOdometry: queue_size = 10 [ INFO] [1688906080.042160042]: RGBDOdometry: subscribe_rgbd = false [ INFO] [1688906080.042165611]: RGBDOdometry: rgbd_cameras = 1 [ INFO] [1688906080.042170215]: RGBDOdometry: keep_color = false [ INFO] [1688906080.049533637]: /rtabmap/rgbd_odometry subscribed to (approx sync): /camera/color/image_raw \ /camera/aligned_depth_to_color/image_raw \ /camera/color/camera_info [ INFO] [1688906080.086729632]: RTAB-Map detection rate = 2.000000 Hz [ INFO] [1688906080.087820322]: rtabmap: Deleted database "/root/.ros/rtabmap.db" (--delete_db_on_start or -d are set). [ INFO] [1688906080.087849914]: rtabmap: Using database from "/root/.ros/rtabmap.db" (0 MB). [ INFO] [1688906080.168580213]: rtabmap: Database version = "0.21.1". [ INFO] [1688906080.168609305]: rtabmap: SLAM mode (Mem/IncrementalMemory=true) [ INFO] [1688906080.182634256]: /rtabmap/rtabmap: subscribe_depth = true [ INFO] [1688906080.182662108]: /rtabmap/rtabmap: subscribe_rgb = true [ INFO] [1688906080.182673195]: /rtabmap/rtabmap: subscribe_stereo = false [ INFO] [1688906080.182691839]: /rtabmap/rtabmap: subscribe_rgbd = false (rgbd_cameras=1) [ INFO] [1688906080.182706896]: /rtabmap/rtabmap: subscribe_odom_info = true [ INFO] [1688906080.182719562]: /rtabmap/rtabmap: subscribe_user_data = false [ INFO] [1688906080.182730929]: /rtabmap/rtabmap: subscribe_scan = false [ INFO] [1688906080.182741541]: /rtabmap/rtabmap: subscribe_scan_cloud = false [ INFO] [1688906080.182749892]: /rtabmap/rtabmap: subscribe_scan_descriptor = false [ INFO] [1688906080.182760086]: /rtabmap/rtabmap: queue_size = 10 [ INFO] [1688906080.182771942]: /rtabmap/rtabmap: approx_sync = true [ INFO] [1688906080.183342576]: Setup depth callback [ INFO] [1688906080.191175182]: /rtabmap/rtabmap subscribed to (approx sync): /rtabmap/odom \ /camera/color/image_raw \ /camera/aligned_depth_to_color/image_raw \ /camera/color/camera_info \ /rtabmap/odom_info [ WARN] [1688906080.196829584]: odometry: waiting imu (/rtabmap/imu) to initialize orientation (wait_imu_to_init=true) [ WARN] (2023-07-09 12:34:40.229) Odometry.cpp:317::process() Updated initial pose from xyz=0.000000,0.000000,0.000000 rpy=0.000000,-0.000000,0.000000 to xyz=0.000000,0.000000,0.000000 rpy=-0.024283,-0.008371,-1.338854 with IMU orientation [ INFO] [1688906080.236562221]: Odom: quality=0, std dev=99.995000m|99.995000rad, update time=0.007077s [ INFO] [1688906080.240203446]: rtabmap 0.21.1 started... [ INFO] [1688906080.276716664]: Odom: quality=79, std dev=0.013467m|0.020556rad, update time=0.013786s [ INFO] [1688906080.315165521]: Odom: quality=78, std dev=0.014857m|0.020040rad, update time=0.017371s [ INFO] [1688906080.352644644]: Odom: quality=70, std dev=0.018554m|0.022044rad, update time=0.021271s [ INFO] [1688906080.389122982]: Odom: quality=78, std dev=0.020358m|0.025637rad, update time=0.023557s [ INFO] [1688906080.425407132]: Odom: quality=90, std dev=0.016257m|0.023166rad, update time=0.024597s [ WARN] (2023-07-09 12:34:40.428) Transform.cpp:551::getTransform() No transform found for stamp 1688906080.364948! Earliest is 1688906080.380594 [ WARN] [1688906080.428961633]: We are receiving imu data (buffer=9), but cannot interpolate imu transform at time 1688906080.364948. IMU won't be added to graph.

[ INFO] [1688906080.465883642]: Odom: quality=81, std dev=0.019193m|0.023937rad, update time=0.029616s libpng warning: iCCP: known incorrect sRGB profile libpng warning: iCCP: known incorrect sRGB profile libpng warning: iCCP: known incorrect sRGB profile [ INFO] [1688906080.505515969]: Odom: quality=62, std dev=0.013660m|0.022563rad, update time=0.035778s [ INFO] [1688906080.541540937]: Odom: quality=91, std dev=0.016786m|0.020556rad, update time=0.035403s [ INFO] [1688906080.554681388]: rtabmap_viz: Reading parameters from the ROS server... [ INFO] [1688906080.577228429]: Odom: quality=77, std dev=0.013677m|0.017689rad, update time=0.035031s [ INFO] [1688906080.614434806]: Odom: quality=79, std dev=0.023584m|0.021909rad, update time=0.036603s [ INFO] [1688906080.647777730]: rtabmap_viz: Parameters read = 360 [ INFO] [1688906080.647796866]: rtabmap_viz: Parameters successfully read. [ INFO] [1688906080.654230217]: Odom: quality=71, std dev=0.015216m|0.027269rad, update time=0.039184s [ INFO] [1688906080.690376809]: Odom: quality=79, std dev=0.021349m|0.024446rad, update time=0.035514s [ INFO] [1688906080.731512722]: Odom: quality=90, std dev=0.020009m|0.024041rad, update time=0.040481s [ INFO] [1688906080.767496266]: Odom: quality=96, std dev=0.017231m|0.020216rad, update time=0.034776s [ INFO] [1688906080.777318108]: /rtabmap/rtabmap_viz: subscribe_depth = true [ INFO] [1688906080.777346236]: /rtabmap/rtabmap_viz: subscribe_rgb = true [ INFO] [1688906080.777361313]: /rtabmap/rtabmap_viz: subscribe_stereo = false [ INFO] [1688906080.777375197]: /rtabmap/rtabmap_viz: subscribe_rgbd = false (rgbd_cameras=1) [ INFO] [1688906080.777387831]: /rtabmap/rtabmap_viz: subscribe_odom_info = true [ INFO] [1688906080.777404604]: /rtabmap/rtabmap_viz: subscribe_user_data = false [ INFO] [1688906080.777419285]: /rtabmap/rtabmap_viz: subscribe_scan = false [ INFO] [1688906080.777432089]: /rtabmap/rtabmap_viz: subscribe_scan_cloud = false [ INFO] [1688906080.777446398]: /rtabmap/rtabmap_viz: subscribe_scan_descriptor = false [ INFO] [1688906080.777461369]: /rtabmap/rtabmap_viz: queue_size = 10 [ INFO] [1688906080.777484569]: /rtabmap/rtabmap_viz: approx_sync = true [ INFO] [1688906080.777514685]: Setup depth callback [ INFO] [1688906080.803502587]: Odom: quality=106, std dev=0.013340m|0.025199rad, update time=0.035356s [ INFO] [1688906080.803607278]: /rtabmap/rtabmap_viz subscribed to (approx sync): /rtabmap/odom \ /camera/color/image_raw \ /camera/aligned_depth_to_color/image_raw \ /camera/color/camera_info \ /rtabmap/odom_info [ INFO] [1688906080.803682994]: rtabmap_viz started. [ INFO] [1688906080.837494636]: Odom: quality=101, std dev=0.014600m|0.024736rad, update time=0.033281s [ INFO] [1688906080.875947061]: Odom: quality=98, std dev=0.014936m|0.019282rad, update time=0.037716s [ INFO] [1688906080.911508472]: Odom: quality=90, std dev=0.029382m|0.024246rad, update time=0.034814s [ INFO] [1688906080.951097688]: Odom: quality=86, std dev=0.024835m|0.020880rad, update time=0.038771s [ INFO] [1688906080.986562778]: Odom: quality=58, std dev=0.016915m|0.029187rad, update time=0.034501s

[ INFO] [1688906081.022114351]: Odom: quality=93, std dev=0.018621m|0.024144rad, update time=0.034336s [ INFO] [1688906081.062777253]: Odom: quality=90, std dev=0.017009m|0.024041rad, update time=0.039989s [ INFO] [1688906081.097528497]: Odom: quality=83, std dev=0.016825m|0.024640rad, update time=0.034115s [ INFO] [1688906081.134820019]: Odom: quality=75, std dev=0.018262m|0.020880rad, update time=0.036654s [ INFO] [1688906081.168953847]: Odom: quality=80, std dev=0.017897m|0.023049rad, update time=0.033488s [ INFO] [1688906081.206107999]: Odom: quality=87, std dev=0.026377m|0.021630rad, update time=0.036493s [ INFO] [1688906081.242651338]: Odom: quality=83, std dev=0.020107m|0.019282rad, update time=0.035908s [ INFO] [1688906081.278485159]: Odom: quality=88, std dev=0.024180m|0.021909rad, update time=0.035069s [ INFO] [1688906081.311256849]: Odom: quality=97, std dev=0.016986m|0.022044rad, update time=0.032091s [ INFO] [1688906081.344587135]: Odom: quality=93, std dev=0.016652m|0.024041rad, update time=0.032689s [ INFO] [1688906081.379352647]: Odom: quality=99, std dev=0.017517m|0.019282rad, update time=0.034122s [ INFO] [1688906081.417273250]: Odom: quality=77, std dev=0.014180m|0.022308rad, update time=0.037299s [ INFO] [1688906081.451039870]: Odom: quality=92, std dev=0.015356m|0.020880rad, update time=0.033181s [ INFO] [1688906081.485669638]: Odom: quality=88, std dev=0.012739m|0.023506rad, update time=0.033967s [ INFO] [1688906081.517676797]: Odom: quality=85, std dev=0.018104m|0.022688rad, update time=0.031425s [ INFO] [1688906081.551524242]: Odom: quality=92, std dev=0.014163m|0.019282rad, update time=0.033197s [ WARN] (2023-07-09 12:34:41.557) Rtabmap.cpp:4334::process() Republishing data of requested node(s) 1 (Rtabmap/MaxRepublished=2)

[ INFO] [1688906081.588555064]: Odom: quality=75, std dev=0.017389m|0.023616rad, update time=0.036346s [ INFO] [1688906081.625096940]: Odom: quality=77, std dev=0.023799m|0.022178rad, update time=0.035729s [ INFO] [1688906081.657402206]: Odom: quality=82, std dev=0.021293m|0.021630rad, update time=0.031664s [ INFO] [1688906081.689800207]: Odom: quality=88, std dev=0.016959m|0.026054rad, update time=0.031780s [ INFO] [1688906081.722289341]: Odom: quality=94, std dev=0.022223m|0.030639rad, update time=0.031873s [ INFO] [1688906081.756606047]: Odom: quality=91, std dev=0.021259m|0.019282rad, update time=0.033681s ...



Can you help me to find out what I have done wrong(Put ply in Blender, use the virtual camera to reconstruct the path based on /tf, the rendered image is offset from the actual rgb image) from the above information?

I cannot see any problems in your RealSense ROS launch logs and your installation procedure seems fine. However, I note from the logs that you are using camera firmware driver version 5.15.0.2 with SDK 2.44.0. The 5.15.0.2 firmware should only be used with the current latest SDK 2.54.1. Using this firmware with an older SDK may result in errors.

For the 2.44.0 SDK, the old firmware version 5.12.12.0 from March 2021 is the recommended version to use with it.

I cannot see any problems in your RealSense ROS launch logs and your installation procedure seems fine. However, I note from the logs that you are using camera firmware driver version 5.15.0.2 with SDK 2.44.0. The 5.15.0.2 firmware should only be used with the current latest SDK 2.54.1. Using this firmware with an older SDK may result in errors.

For the 2.44.0 SDK, the old firmware version 5.12.12.0 from March 2021 is the recommended version to use with it.

Thank you very much for your help, at least now I can say that the problem is not with realsense. I should have asked over at rtabmap for example.

You are very welcome. :)

You are very welcome. :)

Sorry, I have problems with the realsense driver: As mentioned before, I am now using the Docker version of the realsense SDK, which is version 2.44.0, and its realsense-ros counterpart is version 2.2.24, which does not contain the urdf file for D455. So, is there any way I can upgrade the realsense SDK in Docker (to the latest version)?

There should not be a problem with changing the librealsense version installed in your Docker container. However, if you upgrade to a different librealsense version such as 2.50.0 or 2.51.1 then you will also have to rebuild the ROS wrapper afterwards and use a newer wrapper version (2.3.2). As you are using ROS1 Noetic you will not be able to upgrade to a newer SDK version such as 2.53.1 or 2.54.1 as the ROS1 wrapper has ceased development and was not updated for use with SDKs beyond 2.51.1.

If you build the wrapper from source code then an alternative to upgrading librealsense that you could explore would be to download the _d455_urdf.xacro URDF file from the realsense2_description/urdf folder of the most recent ROS1 wrapper (2.3.2) at the link below and copy it into the realsense2_description folder of wrapper 2.2.24.

https://github.com/IntelRealSense/realsense-ros/tree/ros1-legacy/realsense2_description/urdf

There should not be a problem with changing the librealsense version installed in your Docker container. However, if you upgrade to a different librealsense version such as 2.50.0 or 2.51.1 then you will also have to rebuild the ROS wrapper afterwards and use a newer wrapper version (2.3.2). As you are using ROS1 Noetic you will not be able to upgrade to a newer SDK version such as 2.53.1 or 2.54.1 as the ROS1 wrapper has ceased development and was not updated for use with SDKs beyond 2.51.1.

If you build the wrapper from source code then an alternative to upgrading librealsense that you could explore would be to download the _d455_urdf.xacro URDF file from the realsense2_description/urdf folder of the most recent ROS1 wrapper (2.3.2) at the link below and copy it into the realsense2_description folder of wrapper 2.2.24.

https://github.com/IntelRealSense/realsense-ros/tree/ros1-legacy/realsense2_description/urdf

According to the SDK build guidelines, Ubuntu's kernel needs to be patched before building the SDK, i.e. the following code needs to be executed:

Run Intel Realsense permissions script from librealsense root directory: ./scripts/setup_udev_rules.sh

Notice: One can always remove permissions by running: ./scripts/setup_udev_rules.sh --uninstall

Build and apply patched kernel modules for:

Ubuntu 20/22 (focal/jammy) with LTS kernel 5.13, 5.15 ./scripts/patch-realsense-ubuntu-lts-hwe.sh

Do I still need to perform this step if I choose to upgrade the current version of the SDK?

If you have not changed your kernel version from the 5.15.0-75-generic kernel that you installed and patched earlier then you should not need to patch the kernel again when installing a new librealsense version.

If your preference would be to use a more modern kernel than 5.15 then you can do so if librealsense is built from source code with CMake with the flag -DFORCE_RSUSB_BACKEND=TRUE included in the CMake build instruction. An RSUSB build of librealsense bypasses the kernel, so that it is not dependent on Linux versions or kernel versions and does not require a kernel patch script to be applied.

If you have not changed your kernel version from the 5.15.0-75-generic kernel that you installed and patched earlier then you should not need to patch the kernel again when installing a new librealsense version.

If your preference would be to use a more modern kernel than 5.15 then you can do so if librealsense is built from source code with CMake with the flag -DFORCE_RSUSB_BACKEND=TRUE included in the CMake build instruction. An RSUSB build of librealsense bypasses the kernel, so that it is not dependent on Linux versions or kernel versions and does not require a kernel patch script to be applied.

Thank you very much!

@MartyG-RealSense Sorry, I had to reopen this, I still have some questions.

Firstly, I always start the Stereo Module first, then the RGB Camera. Here are two screenshots:

Screenshot_20230710_130715 In the right rgb image, the eyedrop bottle in the centre can be seen to overlap with the side of the thin wall behind it. However, on the left, the silhouette of the eyedrop bottle in the depth image is clearly to the right of the thin wall.

Screenshot_20230710_130956 In this image, even though you can see the eyedropper bottle in the middle being correctly aligned to the depth image, look closely, there is still a small eyedropper bottle in between the cup and the eyedropper bottle in the middle, and I think this is where the error lies.

Screenshot_20230710_133323 Here's a screenshot of the rtabmap, you can see that the eyedropper bottle fails to line up with the thin wall on the back side, what causes this?

It's no trouble at all.

Looking at the image, it appears to me that the depth image may be showing the wide section of the wall rather than its thin side, with this thin edge not showing up on the depth image.

It's no trouble at all.

Looking at the image, it appears to me that the depth image may be showing the wide section of the wall rather than its thin side, with this thin edge not showing up on the depth image.

you are right. I have written <arg name="align_depth" value="true"/> in the launch file, does this correspond to the second image just now? And as shown in that image, showing the large eyedrops is supposed to be the correct result?

When align_depth is True, the depth image will be aligned to the color image, resulting in a image resembling the one below that you provided.

When you ask if the 'large eyedrops' are correct, do you mean the large black shadow surrounding the eyedropper bottle, please?

When align_depth is True, the depth image will be aligned to the color image, resulting in a image resembling the one below that you provided.

What I expect, eyedrop bottle should be in the green box. The red box is in the wrong position. Screenshot_20230710_142821

Screenshot_20230710_143205

I.e., the eyedrop in the red box in the 3D Map on the right side of this image is located in the wrong location, it should be located in the green box

Okay, I see what you mean about the alignment of the red-boxed dropper to the edge of the wall compared to the green-boxed ones. The same seems to be true of the cup beside the dropper compared to the positions of the objects on the shelves and door behind. It looks as though the green-box dropper images are RGB images and the red-box dropper is an aligned depth and RGB image. It is as though the foreground of the image is horizontally offset from the background.

Please try using the instructions at https://github.com/IntelRealSense/librealsense/issues/10182#issuecomment-1019854487 to reset the camera to its factory-new calibration settings in the RealSense Viewer tool to see whether it improves the aligned image.

Okay, I see what you mean about the alignment of the red-boxed dropper to the edge of the wall compared to the green-boxed ones. The same seems to be true of the cup beside the dropper compared to the positions of the objects on the shelves and door behind. It looks as though the green-box dropper images are RGB images and the red-box dropper is an aligned depth and RGB image. It is as though the foreground of the image is horizontally offset from the background.

Please try using the instructions at IntelRealSense/librealsense#10182 (comment) to reset the camera to its factory-new calibration settings in the RealSense Viewer tool to see whether it improves the aligned image.

I ticked "I know what I'm doing" and clicked reset. But the write button is greyed out and cannot be clicked.

After upgrading the SDK to the latest version and installing the corresponding realsense-ros, I ran the programme again, and after careful observation, I found that the range of the "points" for generating the 3D Map is smaller than that of the rgb images, as shown in the figure, the part of the two rgb images on the left side, which I framed with a red line, is used to build the 3D map. This does not match the size of the rgb.

Screenshot_20230710_173544

My launch file:

<?xml version="1.0"?>
<launch>
    <arg name="l515" value="false"/>
    <arg name="d435i" value="false"/>
    <arg name="d455" value="true"/>
    <group if="$(arg d455)">
        <include file="$(find realsense2_camera)/launch/rs_camera.launch">
            <arg name="clip_distance" value="4.0"/>
            <arg name="align_depth" value="true"/>
            <arg name="unite_imu_method" value="linear_interpolation"/>
            <arg name="enable_gyro" value="true"/>
            <arg name="enable_accel" value="true"/>
            <arg name="enable_sync" value="true"/>

            <arg name="infra_width"         default="640"/>
            <arg name="infra_height"        default="480"/>
            <arg name="enable_infra"        default="true"/>

            <arg name="color_width" value="640"/>
            <arg name="color_height" value="480"/>
            <arg name="color_fps" value="30"/>
            <arg name="depth_width" value="640"/>
            <arg name="depth_height" value="480"/>
            <arg name="depth_fps" value="30"/>
         </include>
    </group>

    <node name="imu_filter_node" pkg="imu_filter_madgwick" type="imu_filter_node" output="screen">
        <param name="use_mag" type="bool" value="false"/>
        <param name="publish_tf" type="bool" value="false"/>
        <param name="world_frame" type="string" value="enu"/>
        <remap from="/imu/data_raw" to="/camera/imu"/>
        <remap from="/imu/data" to="/rtabmap/imu" />
    </node>
</launch>

You must enable the depth stream first before opening the camera data window in order for the Write Table button to be unlocked.

infra_rgb could likely be set to False instead of True as it is not related to depth to color alignment. Instead, it generates an RGB stream from the left infrared sensor, which is unnecessary if you already have the color stream enabled.

You must enable the depth stream first before opening the camera data window in order for the Write Table button to be unlocked.

infra_rgb could likely be set to False instead of True as it is not related to depth to color alignment. Instead, it generates an RGB stream from the left infrared sensor, which is unnecessary if you already have the color stream enabled.

Ok, I did this and successfully reset the D455. I deleted the lines in the startup file. Then I ran the programme, but the results remained as before.

As you are using rtabmap. how does it look if you launch with the opensource_tracking.launch launch file instead of rs_camera.launch. The opensource_tracking.launch file is suited for use with rtabmap and imports the settings from rs_camera.launch

As you are using rtabmap. how does it look if you launch with the opensource_tracking.launch launch file instead of rs_camera.launch. The opensource_tracking.launch file is suited for use with rtabmap and imports the settings from rs_camera.launch

Hi, I tried using opensource_tracking.launch as a unified launch file, but there is no change in the results, the angles of the 3D map are still different from the rgb, e.g. the position of the cups in the small red box in this screenshot in relation to the wall on the back side (which is different). Screenshot_20230711_095052

Here is my launch file:

<launch>
    <arg name="offline"          default="false"/>
    <include unless="$(arg offline)" 
        file="$(find realsense2_camera)/launch/rs_camera.launch">
        <arg name="align_depth" value="true"/>
        <arg name="linear_accel_cov" value="1.0"/>
        <arg name="unite_imu_method" value="linear_interpolation"/>
        <arg name="clip_distance" value="4.0"/>

        <arg name="enable_gyro" value="true"/>
        <arg name="enable_accel" value="true"/>
        <arg name="enable_sync" value="true"/>

        <arg name="depth_width" value="640"/> 
        <arg name="depth_height" value="480"/>
        <arg name="depth_fps" value="30"/>

        <arg name="color_width" value="640"/>
        <arg name="color_height" value="480"/>
        <arg name="color_fps" value="30"/> 
    </include>

    <node pkg="imu_filter_madgwick" type="imu_filter_node" name="ImuFilter">
        <param name="use_mag" type="bool" value="false" />
        <param name="_publish_tf" type="bool" value="false" />
        <param name="_world_frame" type="string" value="enu" />
        <remap from="/imu/data_raw" to="/camera/imu"/>
        <!-- <remap from="/imu/data" to="/rtabmap/imu" />     is this needed?--> 
    </node>

    <include file="$(find rtabmap_ros)/launch/rtabmap.launch">

        <arg name="args"        value= "--delete_db_on_start --Rtabmap/DetectionRate 2  RGBD/LinearUpdate 0.01  RGBD/AngularUpdate 0.01" />

        <arg name="rgb_topic" value="/camera/color/image_raw"/>
        <arg name="depth_topic" value="/camera/aligned_depth_to_color/image_raw"/>
        <arg name="camera_info_topic" value="/camera/color/camera_info"/>
        <arg name="depth_camera_info_topic" value="/camera/depth/camera_info"/>
        <arg name="rtabmapviz" value="true"/>
        <arg name="rviz" value="false"/>

        <arg name="use_sim_time"        value="false"/>

    </include>

    <include file="$(find robot_localization)/launch/ukf_template.launch"/>
    <param name="/ukf_se/frequency" value="300"/>
    <param name="/ukf_se/base_link_frame" value="camera_link"/>
    <param name="/ukf_se/odom0" value="rtabmap/odom"/>
    <rosparam param="/ukf_se/odom0_config">[true,true,true,
                                            true,true,true,
                                            true,true,true,
                                            true,true,true,
                                            true,true,true]
    </rosparam>
    <param name="/ukf_se/odom0_relative" value="true"/>
    <param name="/ukf_se/odom0_pose_rejection_threshold" value="10000000"/>
    <param name="/ukf_se/odom0_twist_rejection_threshold" value="10000000"/>

    <param name="/ukf_se/imu0" value="/imu/data"/>
    <rosparam param="/ukf_se/imu0_config">[false, false, false,
                                           true,  true,  true,
                                           true,  true,  true,
                                           true,  true,  true,
                                           true,  true,  true]
    </rosparam>
    <param name="/ukf_se/imu0_differential" value="true"/>
    <param name="/ukf_se/imu0_relative" value="false"/>
    <param name="/ukf_se/use_control" value="false"/>
    <!-- <param name="/ukf_se/odom0_config" value="{true,true,true,}"/> -->
</launch>

You could try generating a simple depth-color aligned pointcloud in the ROS wrapper with:

roslaunch realsense2_camera rs_camera.launch filters:=pointcloud

https://github.com/IntelRealSense/realsense-ros/tree/ros1-legacy#point-cloud

If the pointcloud's depth-color alignment appears correct then the problem is likely to be either with the wrapper's align_depth function or with rtabmap.

You could try generating a simple depth-color aligned pointcloud in the ROS wrapper with:

roslaunch realsense2_camera rs_camera.launch filters:=pointcloud

https://github.com/IntelRealSense/realsense-ros/tree/ros1-legacy#point-cloud

If the pointcloud's depth-color alignment appears correct then the problem is likely to be either with the wrapper's align_depth function or with rtabmap.

the pointcloud's depth-color alignment appears correct. I guess the problem really isn't with the camera itself. One last question, what is the best distance to use the D455? Since I'm trying to record indoor objects and the objects are small, like apples bananas something else, what should I set the rgb/depth/fov to to get the best results? As well, how far should I hold the camera from these objects?

A D455 will have a minimum depth sensing distance of around 0.4 meters / 40 cm, with the depth image progressively breaking up when moving the camera closer to the object than that minimum.

A method for reducing the minimum distance in the ROS wrapper is to define a disparity shift value in a json camera configuration file and then configure the roslaunch instruction or edit the launch file to load that json during launch to apply the new value. More information about this can be found at https://github.com/IntelRealSense/realsense-ros/issues/2012#issuecomment-890390205

With the RGB image, having the camera close to an object usually will not matter and it will not require a special setting.

A D455 will have a minimum depth sensing distance of around 0.4 meters / 40 cm, with the depth image progressively breaking up when moving the camera closer to the object than that minimum.

A method for reducing the minimum distance in the ROS wrapper is to define a disparity shift value in a json camera configuration file and then configure the roslaunch instruction or edit the launch file to load that json during launch to apply the new value. More information about this can be found at #2012 (comment)

With the RGB image, having the camera close to an object usually will not matter and it will not require a special setting.

Thank you very much for your continued help!

IntelRealSense / realsense-ros

How to get camera_color_frame instead of camera_optical_color_frame? #2808

SUMMARY