search

MIT-SPARK / Kimera-Semantics

Real-Time 3D Semantic Reconstruction from 2D data
BSD 2-Clause "Simplified" License
633 stars 134 forks source link

how to create the kinera_semantics_demo.bag myself? #8

Closed huangju91 closed 4 years ago

huangju91 commented 4 years ago

Hi,Thanks for sharing this awesome work, I wonder how you create the kinera_semantics_demo.bag, is it made from another module like Kimera-VIO ?

ToniRV commented 4 years ago

Hi @huangju91, the bag was generated with a simulator developed with our colleagues at LincolnLabs. We are planning to release it for a workshop competition at ICRA, I’ll update this issue with the official release. Thanks!

XanxusCrypto commented 4 years ago

Hi @huangju91, the bag was generated with a simulator developed with our colleagues at LincolnLabs. We are planning to release it for a workshop competition at ICRA, I’ll update this issue with the official release. Thanks!

I found the corresponding description in the Kimera paper that you utilized a simulator to generate the ground truth and 2-D semantic images so as to evaluate the Kimera-Semantics performance. Meanwhile you recommended to use deep learning neural networks to generate 2d semantic labels. While I am confused that when the stereo-dense-reconstruction-node was turned on which you said to utilize dense stereo to obtain 3d point clouds in the paper, I could not found where the depth information was calculated from the raw stereo image inputs. I tried to search in the Kimera-VIO module to find the dense stereo depth calculation processes but I got some trouble in the installation. So would you plz help me figure out this issue？ Thanks much for the aswsome work you shared.

ToniRV commented 4 years ago

@XanxusCrypto checkout the fag run_stereo_densein here: https://github.com/MIT-SPARK/Kimera-Semantics/blob/6637d8b54c1fc17903f507ff0f536397eb107d3d/kimera_semantics_ros/launch/kimera_semantics.launch#L38

If you turn on this flag in the launch file (and you have a pair of stereo images, Kimera-Semantics will make use of depth coming from the stereo reconstruction algorithm, but you could technically use any kind of point-cloud even a LIDAR based one like in here (where I used Semantic-Kitti): semantic_3d_meshlab_201 semantic_3d_mesh_closeup_van03 (1)

XanxusCrypto commented 4 years ago

@ToniRV Thanks much for answering my questions. So the stereo reconstruction algorithm is constructed in Volblox? I just read the Volblox paper which utilized stereo or RGB-D camera, but no more useful information was found further in their launch file of EuRoC dataset. While in their newer version of Volblox++, only RGB-D camera is utilized. Bcs I would like to run visual-inertial SLAM with stereo camrea to output poses and points, I would like to understand how and where was the dense stereo reconstruction algorithm constructed. Thank you again for sharing your code.

ToniRV commented 4 years ago

The stereo reconstruction is done using ROS image_proc package, in particular using stereo_image_proc which under the hood uses StereoBM algorithm implemented in OpenCV.

XanxusCrypto commented 4 years ago

The stereo reconstruction is done using ROS image_proc package, in particular using stereo_image_proc which under the hood uses StereoBM algorithm implemented in OpenCV.

Thanks very much for the elaborative answers. I have found the dense stereo reconstruction algorithm utilized. It helped me a lot!

ToniRV commented 4 years ago

Closing on this issue: the simulator is the same that we will be using in our ICRA2020 workshop this May: https://mit-spark.github.io/PAL-ICRA2020/

p-akanksha commented 3 years ago

Hi @ToniRV,

The simulator you mentioned in the last comment, was it the same one used for the goseek-challenge?

Also, is there a recording or any other material related to that ICRA2020 workshop available? It would be a great resource.

p-akanksha commented 3 years ago

I found the recording of the workshop. Posting the link here: Metric-Semantic SLAM with Kimera: A Hands On Tutorial

This work is amazing. Thanks for making it open-source!