Get Started with Kimera-Semantics

[nbansal90]

Hello Guys,

Thank you for a great repo for qualitative analysis of the whole slam pipeline! I was able to set up the repo and run the simulation to see the 3D semantic construction for the office data.

I was explicitly looking to use Kimera semantics for now. I was just confused about how exactly shall I get started with it. I have the following data with me (ScanNet):

• RGB frames
• Corresponding Depth frames
• Corresponding Label frames
• Corresponding Extrinsics for each RGB

According to my understanding I need to make changes to Kimera_sematics.launch file for setting the variables to the right path. I see the following paths set in the launch file.

arg name="semantic_pointcloud"         default="/semantic_pointcloud"/>  
arg name="left_cam_info_topic"         default="/tesse/left_cam/camera_info"/&gt

I also see semantic_label_2_color_csv_filepath csv file which has color mapping and Id information, which I would need to create for my case. I presume that if I run it for a new dataset, I would need to change number of classes and csv file too. Do I have to rebuild the repo after making these changes?

Also Since I already have depth info with me, Do I convert them to point cloud using the extrinsic which I have, or it done internally if provide depth and extrinsic?

Would be great to know if there is a resource which actually points towards running Kimera-semantics from scratch on given set of data in a step by step fashion.

Regards, Nitin

[KaninchenM]

I have the same question. Generally speaking how to use Kimera-Semantics for my own data?

[nbansal90]

@KaninchenM I have resolved this issue. I got it working for my dataset which was Scannet. Steps are Following:

1. Create a rosbag for your dataset (rgb, depth, labels, pose, intrinsics) by publishing the data to different respective topics.(Run Command rosbag record -a in a parallel window while you are publishing your afore-mentioned data).
2. Create a new launch and rviz file for your case which would be pretty much similar to kimera_semantics.launch file, where remap the topics according to topics which you have created in step 1.
3. Run the launch and rviz file for visualization

[LoneTraveler-99]

@KaninchenM I have resolved this issue. I got it working for my dataset which was Scannet. Steps are Following:

1. Create a rosbag for your dataset (rgb, depth, labels, pose, intrinsics) by publishing the data to different respective topics.(Run Command rosbag record -a in a parallel window while you are publishing your afore-mentioned data).
2. Create a new launch and rviz file for your case which would be pretty much similar to kimera_semantics.launch file, where remap the topics according to topics which you have created in step 1.
3. Run the launch and rviz file for visualization

Can you share the code of step 1, thank you

[KaninchenM]

您的邮件已收到，我会尽快回复您！

[dongho-Han]

@KaninchenM @nbansal90 Hello! I have a hard time making a dataset into rosbag type.

@KaninchenM I have resolved this issue. I got it working for my dataset which was Scannet. Steps are Following:

1. Create a rosbag for your dataset (rgb, depth, labels, pose, intrinsics) by publishing the data to different respective topics.(Run Command rosbag record -a in a parallel window while you are publishing your afore-mentioned data).
2. Create a new launch and rviz file for your case which would be pretty much similar to kimera_semantics.launch file, where remap the topics according to topics which you have created in step 1.
3. Run the launch and rviz file for visualization

Can you share how you did in step 1 and 2? Code or more explanation will be very very helpful! I want to test the Kimera-Semantics in HyperSim(https://github.com/apple/ml-hypersim) dataset, but they only give the hdf5 format. Willing for the reply. Thanks.

[nbansal90]

@dongho-Han I no longer have access to my project. Nonetheless, I can give you a brief know-how about rough steps followed by me to get it working for ScanNet dataset. (These are the notes which I had made during my project)

1. Follow the Installation steps as mentioned in the Installation section of the readme page. (Prerequisite & Kimera Semantics Installation.)
2. ScanNet data preprocessing: ScanNet Dataset has the following ground-truth information present: ➢ RGB Images : shape (968, 1296, 3) ➢ Depth Images : shape (480, 640, 1) ➢ Label Images : shape (480, 640, 1) ➢ Pose : 4x4 matrix in form txt file. ➢ Intrinsics : 4x4 matrix

You have to see in your case, how can you get these information for your dataset.

KIMERA SEMANTICS CHANGES For a new Dataset at hand you need to create few new files to handle the 3D Semantic Reconstruction process.

1. Create a rosbag file for your dataset using ROS by publishing the images(RGB, Depth, Labels), Pose and Intrinsics on different topics using C++ or Python. and then recording via : rosbag record -a. Save the created rosbag file at catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/rosbagscatki n_ws/src/Kimera-Semantics/kimera_semantics_ros/rosbags
2. Create a launch file (at catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/launch) for your dataset, which takes input a rosbag file, which has been explicitly created for your dataset, and subscribes to topics which are present in the rosbag file.
3. Create rviz file (at catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/rviz) for your dataset.
4. Create csv file for semantic color mapping at catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/cfg

RUN COMMAND

1. Open 4 different terminals for running ros init, launching 3D Reconstruction ros node, rviz node and rqt grpah node, and run following command on each terminal.
2. roscore
3. roslaunch kimera_semantics_ros play_bag:=true
4. rviz -d
5. rosrun rqt_graph rqt_graph

[sandilyasg]

@nbansal90 @dongho-Han @LoneTraveler-99 for step 2 "Create a launch file (at catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/launch) for the dataset, which takes input a rosbag file, which has been explicitly created for your dataset, and subscribes to topics which are present in the rosbag file.", what does the launch file look like because we are directly using the RGB and depth streams, and not left/right stereo images to get depth? What would your "left_cam_info_topic", "left_cam_topic", "right_cam_info_topic", "right_cam_topic", "left_cam_segmentation_topic", "left_cam_depth_topic" be?

[KaninchenM]

您的邮件已收到，我会尽快回复您！