search

MIT-SPARK / Kimera-Semantics

Real-Time 3D Semantic Reconstruction from 2D data
BSD 2-Clause "Simplified" License
640 stars 136 forks source link

Kimera semantics does not work with rosbag recorded from REALSENSE D435i #75

Open lucamorando95 opened 2 years ago

lucamorando95 commented 2 years ago

Description: Hello and congratulation for this massive framework. It's about two days that i'm trying to use the kimera semantic framework with previously recorded rosbag data. Kimera semantics works fine if used with euroc rosbag instead. I am using kimera semantics with kimera Vio ROS, which works fine also on previosuly recorded rosbag. As specified in the readme, i recorded the following topics, which are given as input to kimera VIO ROS:

Command:

/camera/infra1/image_rect_raw
/camera/infra2/image_rect_raw
/camera/imu

Here i described the launching commands pipeline: I run kimera VIO ROS with the following command:

roslaunch kimera_vio_ros kimera_vio_ros_realsense_IR.launch

I run kimera semantics:

roslaunch kimera_semantics_ros kimera_metric_realsense.launch

I run kimera Rviz Vio pluging

I run kimera semantics Rviz Plugin

I run my rosbag file

while Kimera VIO ROS generates the per frame textured mesh, Kimera semantics does not create the voxblox mesh from the point cloud, differently from what it happens with the euroc dataset, where it is generated correctly.

The topic 

/kimera_semantics_node/mesh

outputs mesh_block of size zero, so inside is actually empty.

The launch file kimera_vio_ros_realsense_IR is reported below:

<launch>
  <arg name="dataset_name" value="RealSenseIR"/>
  <arg name="online" default="true" />

  <arg name="log_output" default="false"/>
  <arg name="log_output_path"
       default="$(find kimera_vio_ros)/output_logs/"
       if="$(arg log_output)"/>
  <arg name="log_gt_data" default="false"/>
  <arg name="gt_topic" default=""/>
  <!-- Set use_sim_time to true if you use rosbag with clock argument -->
  <arg name="should_use_sim_time" default="false"/>

  <!-- Only used when parsing a rosbag -->
  <arg name="rosbag_path" default="/home/arpl/kimera_ws/src/rosbag/realsense_lab7.bag"
       unless="$(arg online)"/>"

  <!-- Frame IDs. These DO NOT match frame id's on the video streams, as the 
    RealSense and Kimera publish conflicting Tf's -->
  <arg name="world_frame_id"     value="world"/>
  <arg name="base_link_frame_id" value="base_link"/>
  <arg name="map_frame_id"       value="map"/>
  <arg name="left_cam_frame_id"  value="left_cam"/>
  <arg name="right_cam_frame_id" value="right_cam"/>

  <!-- Subscribed Topics -->
  <arg name="left_cam_topic"  value="/camera/infra1/image_rect_raw"/>
  <arg name="right_cam_topic" value="/camera/infra2/image_rect_raw"/>
  <arg name="imu_topic"       value="/camera/imu"/>

  <!-- Launch actual pipeline -->
  <include file="$(find kimera_vio_ros)/launch/kimera_vio_ros.launch"
           pass_all_args="true"/>

 <!-- Launch static TF node from base_link to velo_link -->
 <node pkg="tf" type="static_transform_publisher" name="velo_link_broadcaster"
   args="0 0 0 0 0 0 1 base_link velo_link 100"/>

</launch>

The launch file kimera_metric_realsense is visible below:

<?xml version="1.0" encoding="ISO-8859-15"?>
<launch>
  <!-- If you aren't running a bag file, set to false-->
  <arg name="should_use_sim_time" default="true"/>
  <arg name="max_ray_length_m"    default="2.5"/>

  <!-- If you just want to run 3D reconstruction without semantics, set this flag to false-->
  <arg name="metric_semantic_reconstruction"      value="false"/>
  <arg name="semantic_label_2_color_csv_filepath" default="$(find kimera_semantics_ros)/cfg/maskrcnn_mapping.csv"/>

  <!-- Change sensor frame to:
   - 1. VIO's estimated base_link: `left_cam_base_link`
   - 2. Or, if you want to use simulator's ground-truth: `left_cam`
  -->
  <arg name="sensor_frame" value="base_link"/>

  <!-- Input -->
  <arg name="left_cam_info_topic"         default="/depth_camera/aligned_depth_to_color/camera_info"/>
  <arg name="left_cam_topic"              default="/depth_camera/color/image_raw"/> <!--/depth_camera/color/image_raw -->
  <arg name="left_cam_segmentation_topic" default="/depth_camera/color/semantic_image"/>
  <arg name="left_cam_depth_topic"        default="/depth_camera/aligned_depth_to_color/image_raw"/>

  <!-- Tracking IMU to Tracking Camera tf -->
  <node pkg="tf" type="static_transform_publisher" name="imu_to_left_cam"
    args="-0.0107491 0.00339225 0.02843741 0.0052734748 0.0001508149  0.0022978430 0.9999834436 base_link_dvio left_cam 100"/>
  <!-- Tracking to Mapping Camera tf -->
  <node pkg="tf" type="static_transform_publisher" name="inter_cam_tf"
    args="0.075 0 -0.065 0 0.7071 0 0.7071  left_cam depth_cam 100"/>
  <node pkg="tf" type="static_transform_publisher" name="inter_cam_tf_2"
    args="0.075 0 -0.065 0 0.7071 0 0.7071 left_cam depth_camera_color_optical_frame 100"/>

  <!-- Launch actual pipeline -->
  <include file="$(find kimera_semantics_ros)/launch/kimera_semantics.launch" pass_all_args="true"/>
</launch>

Additional files: Please attach all the files needed to reproduce the error.

Please give also the following information:

CCCC-user commented 2 years ago

hi,have you solved this problem. I hope you can give me some advice

aswinpajayan commented 1 year ago

From a presentation from the authors, kimera semantics will run only on EuroC (https://youtu.be/Zjevg5wQTdI?t=1001). Do you think they have added this feature now?. Refer to Issue #51

madhu-korada commented 1 year ago

I had to resolve all the issues related TF and it worked for me.

prmehta24 commented 1 year ago

@madhu-korada What issues were you facing related to TF?

image

Did you face this problem? If so, how did you solve it? (unknown frame world)