[BUG] {Left and right images synchronization issue}

[jnskkmhr]

Hi, I am currently working on OAKD-pro to publish left and right images synced (mainly for VIO). I think depthai_ros provides ros parameter for publishing left and right images with the same timestamp.

Below is from README (https://github.com/luxonis/depthai-ros#depth-alignment)

You can also set i_publish_synced_rect_pair: true to get both images with the same timestamps.

However, setting this parameter to true does not give me time-synced left and right image topics. Is there any ways I can solve this issue? Looking forward to the reply!

---

OS: Ubuntu 20.04 ROS: foxy installation: docker

Here is the config file I used for the ros2 foxy package. Stereo images at 15Hz, IMU (unite imu method=LINEAR_INTERPOLATE_ACCEL)

camera:
  i_enable_imu: true
  i_enable_ir: false
  i_floodlight_brightness: 0
  i_ip: ''
  i_laser_dot_brightness: 800
  i_mx_id: ''
  i_nn_type: none
  i_pipeline_type: Stereo
  i_usb_port_id: ''
  i_usb_speed: SUPER_PLUS
  i_pipeline_dump: false
  i_calibration_dump: false
  i_external_calibration_path: ''
left:
  i_board_socket_id: 1
  i_calibration_file: ''
  i_get_base_device_timestamp: true
  i_fps: 15.0
  i_height: 400
  i_low_bandwidth: false
  i_low_bandwidth_quality: 50
  i_max_q_size: 30
  i_publish_topic: true
  i_simulate_from_topic: false
  i_disable_node: false
  i_resolution: '400'
  i_width: 640
  r_exposure: 1000
  r_iso: 800
  r_set_man_exposure: false
nn:
  i_nn_config_path: depthai_ros_driver/mobilenet
  i_disable_resize: false
  i_enable_passthrough: false
  i_enable_passthrough_depth: false
rgb:
  i_board_socket_id: 0
  i_simulate_from_topic: false
  i_get_base_device_timestamp: true
  i_disable_node: false
  i_calibration_file: ''
  i_enable_preview: false
  i_fps: 15.0
  i_height: 720
  i_interleaved: false
  i_keep_preview_aspect_ratio: true
  i_low_bandwidth: false
  i_low_bandwidth_quality: 50
  i_max_q_size: 30
  i_preview_size: 300
  i_publish_topic: false
  i_resolution: '1080'
  i_set_isp_scale: true
  i_isp_num: 2
  i_isp_den: 3
  i_output_isp: true
  i_width: 1280
  r_exposure: 20000
  r_focus: 1
  r_iso: 800
  r_set_man_exposure: false
  r_set_man_focus: false
  r_set_man_whitebalance: false
  r_whitebalance: 3300
right:
  i_board_socket_id: 2
  i_calibration_file: ''
  i_get_base_device_timestamp: true
  i_fps: 15.0
  i_height: 400
  i_low_bandwidth: false
  i_low_bandwidth_quality: 50
  i_max_q_size: 30
  i_publish_topic: true
  i_simulate_from_topic: false
  i_disable_node: false
  i_resolution: '400'
  i_width: 640
  r_exposure: 1000
  r_iso: 800
  r_set_man_exposure: false
stereo:
  i_align_depth: true
  i_get_base_device_timestamp: true
  i_output_disparity: false
  i_bilateral_sigma: 0
  i_board_socket_id: 0
  i_depth_filter_size: 5
  i_depth_preset: HIGH_ACCURACY
  i_disparity_width: DISPARITY_96
  i_enable_companding: false
  i_enable_decimation_filter: false
  i_enable_distortion_correction: true
  i_enable_spatial_filter: false
  i_enable_speckle_filter: false
  i_enable_temporal_filter: false
  i_enable_threshold_filter: false
  i_extended_disp: false
  i_height: 720
  i_low_bandwidth: false
  i_low_bandwidth_quality: 50
  i_lr_check: true
  i_lrc_threshold: 10
  i_max_q_size: 30
  i_rectify_edge_fill_color: 0
  i_stereo_conf_threshold: 255
  i_set_input_size: false
  i_input_width: 1280
  i_input_height: 720
  i_width: 1280
  i_publish_right_rect: true
  i_publish_left_rect: true
  i_publish_synced_rect_pair: true
  i_reverse_stereo_socket_order: true
use_sim_time: false

I am also attaching left and right image topic inside foxglove.

I also tried with noetic (depthai-ros source install), but still left/right images are not synchronized.

[drdas]

This is a big issue on any software that relies on stereo images from the sensor such as VIO based SLAM. One would think that the left and right depth camera frames are synchronized in hardware, otherwise there is no way to provide consistent stereo depth pointclouds output. Is this just a ROS header timestamp issue or are the images really captured 6 to 10 microseconds apart?

[Serafadam]

Hi, if you want to use synced image pair, you should refer to rect topics as those are synced by the stereo node

[jnskkmhr]

Thanks! I will try it and let you know the result.

[jnskkmhr]

@Serafadam Hi, I tried the things you advised me. Unfortunately, I still have left and right pairs with different timestamps. Even worse, the seq number is different (the left image is 1 steps ahead).

Could you check if you also face the same issue? I am using noetic branch and the following launch file and parameter.

<?xml version="1.0"?>
<launch>

    <arg name="name" default="oak" />
    <arg name="params_file" default="$(find depthai_ros_driver)/config/stereo_inertial_rect.yaml"/>
    <arg name="camera_model" default="OAK-D" />
    <!-- 'zed' or 'zedm' -->
    <arg name="base_frame" default="oak-d_frame" />
    <arg name="parent_frame" default="oak-d-base-frame" />

    <arg name="cam_pos_x" default="0.0" />
    <!-- Position respect to base frame (i.e. "base_link) -->
    <arg name="cam_pos_y" default="0.0" />
    <!-- Position respect to base frame (i.e. "base_link) -->
    <arg name="cam_pos_z" default="0.0" />
    <!-- Position respect to base frame (i.e. "base_link) -->
    <arg name="cam_roll" default="0.0" />
    <!-- Orientation respect to base frame (i.e. "base_link) -->
    <arg name="cam_pitch" default="0.0" />
    <!-- Orientation respect to base frame (i.e. "base_link) -->
    <arg name="cam_yaw" default="0.0" />
    <!-- Orientation respect to base frame (i.e. "base_link) -->

    <rosparam file="$(arg params_file)" />
    <node pkg="rosservice" if="$(optenv DEPTHAI_DEBUG 0)" type="rosservice" name="set_log_level" args="call --wait /oak_nodelet_manager/set_logger_level 'ros.depthai_ros_driver' 'debug'" />
    <include file="$(find depthai_descriptions)/launch/urdf.launch">
        <arg name="base_frame" value="$(arg  name)" />
        <arg name="parent_frame" value="$(arg  parent_frame)"/>
        <arg name="camera_model" value="$(arg  camera_model)"/>
        <arg name="tf_prefix" value="$(arg  name)" />
        <arg name="cam_pos_x" value="$(arg  cam_pos_x)" />
        <arg name="cam_pos_y" value="$(arg  cam_pos_y)" />
        <arg name="cam_pos_z" value="$(arg  cam_pos_z)" />
        <arg name="cam_roll" value="$(arg  cam_roll)" />
        <arg name="cam_pitch" value="$(arg  cam_pitch)" />
        <arg name="cam_yaw" value="$(arg  cam_yaw)" />
    </include>

    <node pkg="nodelet" type="nodelet" name="$(arg  name)_nodelet_manager" args="manager" output="screen">
        <remap from="/nodelet_manager/load_nodelet" to="$(arg name)/nodelet_manager/load_nodelet"/>
        <remap from="/nodelet_manager/unload_nodelet" to="$(arg name)/nodelet_manager/unload_nodelet"/>
        <remap from="/nodelet_manager/list" to="$(arg name)/nodelet_manager/list"/>
    </node>

    <node name="$(arg  name)" pkg="nodelet" type="nodelet" output="screen" required="true" args="load depthai_ros_driver/Camera $(arg  name)_nodelet_manager">
    </node>
    <node pkg="rviz" type="rviz" name="pkg" args="-d $(find depthai_ros_driver)/rviz/stereo_inertial_rect.rviz"/>

</launch>

/oak:
  camera_i_enable_imu: true
  imu_i_acc_freq: 200
  imu_i_gyro_freq: 200
  imu_i_sync_method: LINEAR_INTERPOLATE_ACCEL
  imu_i_message_type: IMU
  camera_i_enable_ir: false
  camera_i_floodlight_brightness: 0.0
  # camera_i_floodlight_brightness: 1000.0 #night vision
  camera_i_ip: ''
  camera_i_laser_dot_brightness: 800.0
  camera_i_mx_id: ''
  camera_i_nn_type: none
  camera_i_pipeline_type: Depth
  camera_i_pipeline_dump: true
  camera_i_usb_port_id: ''
  camera_i_usb_speed: SUPER_PLUS
  imu_i_max_q_size: 2

  # rgb image
  rgb:
    image_raw:
      compressed:
        format: jpeg
        jpeg_optimize: false
        jpeg_progressive: false
        jpeg_quality: 80
        jpeg_restart_interval: 0
        png_level: 9
      compressedDepth:
        depth_max: 10.0
        depth_quantization: 100.0
        format: png
        png_level: 9
      theora:
        keyframe_frequency: 64
        optimize_for: 1
        quality: 31
        target_bitrate: 800000
  rgb_i_board_socket_id: 0
  rgb_i_calibration_file: ''
  rgb_i_enable_preview: false
  rgb_i_fps: 30.0
  rgb_i_height: 400
  rgb_i_interleaved: true
  rgb_i_keep_preview_aspect_ratio: true
  rgb_i_low_bandwidth: false
  rgb_i_low_bandwidth_quality: 50
  rgb_i_max_q_size: 30
  rgb_i_preview_size: 416
  rgb_i_publish_topic: false
  rgb_i_resolution: '1080'
  rgb_i_set_isp_scale: true
  rgb_i_width: 640
  rgb_r_exposure: 1000
  rgb_r_focus: 1
  rgb_r_iso: 800
  rgb_r_keep_preview_aspect_ratio: true
  rgb_r_set_man_exposure: false
  rgb_r_set_man_focus: false
  rgb_r_set_man_whitebalance: false
  rgb_r_whitebalance: 3300

  # left image
  left_i_board_socket_id: 1
  left_i_calibration_file: ''
  left_i_fps: 15.0
  left_i_height: 400
  left_i_low_bandwidth: true
  left_i_low_bandwidth_quality: 50
  left_i_max_q_size: 10
  left_i_preview_size: 416
  left_i_publish_topic: false
  left_i_resolution: '400'
  left_i_set_isp_scale: true
  left_i_width: 640
  left_r_exposure: 8000
  left_r_focus: 1
  left_r_iso: 100
  left_r_keep_preview_aspect_ratio: true
  left_r_set_man_exposure: false
  left_r_set_man_focus: false
  left_r_set_man_whitebalance: false
  left_r_whitebalance: 1000
  nn_i_nn_config_path: depthai_ros_driver/yolo

  # right image
  right_i_board_socket_id: 2
  right_i_calibration_file: ''
  right_i_fps: 15.0
  right_i_height: 400
  right_i_low_bandwidth: true
  right_i_low_bandwidth_quality: 50
  right_i_max_q_size: 10
  right_i_preview_size: 416
  right_i_publish_topic: false
  right_i_resolution: '400'
  right_i_set_isp_scale: true
  right_i_width: 640
  right_r_exposure: 8000
  right_r_focus: 1
  right_r_iso: 100
  right_r_keep_preview_aspect_ratio: true
  right_r_set_man_exposure: false
  right_r_set_man_focus: false
  right_r_set_man_whitebalance: false
  right_r_whitebalance: 1000

  # stereo
  stereo:
    image_raw:
      compressed:
        format: jpeg
        jpeg_optimize: false
        jpeg_progressive: false
        jpeg_quality: 80
        jpeg_restart_interval: 0
        png_level: 9
      compressedDepth:
        depth_max: 10.0
        depth_quantization: 100.0
        format: png
        png_level: 9
      theora:
        keyframe_frequency: 64
        optimize_for: 1
        quality: 31
        target_bitrate: 800000
  stereo_i_align_depth: false
  stereo_i_bilateral_sigma: 0
  stereo_i_board_socket_id: 0
  stereo_i_calibration_file: ''
  stereo_i_decimation_filter_decimation_factor: 1
  stereo_i_decimation_filter_decimation_mode: PIXEL_SKIPPING
  stereo_i_depth_filter_size: 5
  stereo_i_depth_preset: HIGH_ACCURACY
  stereo_i_enable_decimation_filter: false
  stereo_i_enable_distoartion_correction: false
  stereo_i_enable_spatial_filter: false
  stereo_i_enable_temporal_filter: false
  stereo_i_enable_threshold_filter: false
  stereo_i_extended_disp: false
  stereo_i_height: 400
  stereo_i_low_bandwidth: true
  stereo_i_low_bandwidth_quality: 50
  stereo_i_lr_check: true
  stereo_i_lrc_threshold: 10
  stereo_i_max_q_size: 30
  stereo_i_rectify_edge_fill_color: 0
  stereo_i_spatial_filter_alpha: 0.5
  stereo_i_spatial_filter_delta: 20
  stereo_i_spatial_filter_hole_filling_radius: 2
  stereo_i_spatial_filter_iterations: 1
  stereo_i_stereo_conf_threshold: 255
  stereo_i_temporal_filter_alpha: 0.4
  stereo_i_temporal_filter_delta: 20
  stereo_i_temporal_filter_persistency: VALID_2_IN_LAST_4
  stereo_i_threshold_filter_max_range: 15000
  stereo_i_threshold_filter_min_range: 400
  stereo_i_width: 640

  # stereo depth publish
  stereo_i_publish_topic: false
  stereo_i_low_bandwidth: true
  stereo_i_low_bandwidth_quality: 50

  # left rectified image
  stereo_i_publish_left_rect: true
  stereo_i_left_rect_low_bandwidth: true
  sterei_i_left_rect_low_bandwidth_quality: 50

  # right rectified image
  stereo_i_publish_right_rect: true
  stereo_i_right_rect_low_bandwidth: true
  sterei_i_left_rect_low_bandwidth_quality: 50

  # other stereo setting
  stereo_i_reverse_stereo_socket_order: true
  stereo_i_publish_synced_rect_pair: true

[Serafadam]

Hi @jnskkmhr , could you check if that branch works for you? Minimal config version should be:

/oak:
  camera_i_enable_imu: true
  camera_i_enable_ir: true
  camera_i_floodlight_brightness: 0.0
  camera_i_ip: ''
  camera_i_laser_dot_brightness: 800.0
  camera_i_mx_id: ''
  camera_i_nn_type: spatial
  camera_i_pipeline_type: rgbd
  camera_i_usb_port_id: ''
  camera_i_usb_speed: SUPER_PLUS
  nn_i_nn_config_path: depthai_ros_driver/mobilenet
  stereo_i_publish_synced_rect_pair: true

[jnskkmhr]

@Serafadam It now works! I updated mine to the latest commit (I pulled origin quite recently, so I thought that was not the reason) Thanks so much!