How to export 3D reconstructed scene and semantic objects?

[CaoYuchen]

Hi, I'm recently using semantic Kimera to run my own dataset, I'm not so familiar with Kimera yet, can you illustrate a little bit about how can I export reconstructed scene and individual semantic objects through the pipeline? So far I don't see anything related in .launch file.

[ToniRV]

Hi @CaoYuchen, You can save the whole 3D mesh by calling the generate_mesh rosservice:

rosservice call /kimera_semantics_node/generate_mesh

This will save a .ply file in the mesh_results folder. The .ply stored depends on this argument in the launch file:

 <param name="mesh_filename"           value="$(find kimera_semantics_ros)/mesh_results/$(anon tesse).ply" />

Btw, Kimera-Semantics is using [Voxblox]() under the hood, so the documentation for Voxblox is applicable to Kimera-Semantics (mostly). I really recommend reading it here.

We don't have a script to extract the individual semantic objects, but with the resulting .ply file you could get all triangles corresponding to a particular object class, say chairs, by looking at their rgb (color) values. Checkout this file for a mapping between colors and semantic labels.

[CaoYuchen]

Hi @CaoYuchen, You can save the whole 3D mesh by calling the generate_mesh rosservice:

rosservice call /kimera_semantics_node/generate_mesh

This will save a .ply file in the mesh_results folder. The .ply stored depends on this argument in the launch file:

 <param name="mesh_filename"           value="$(find kimera_semantics_ros)/mesh_results/$(anon tesse).ply" />

Btw, Kimera-Semantics is using Voxblox under the hood, so the documentation for Voxblox is applicable to Kimera-Semantics (mostly). I really recommend reading it here.

We don't have a script to extract the individual semantic objects, but with the resulting .ply file you could get all triangles corresponding to a particular object class, say chairs, by looking at their rgb (color) values. Checkout this file for a mapping between colors and semantic labels.

Thanks for your answer @ToniRV . I made my own dataset exactly same with the given Kitti dataset like this:

And I also change the launch file and tesse_multiscene_office1_segmentation_mapping.csv accordingly to label my semantic objects. However, when I launch the file, it doesn't reconstruct mesh like running Kitti, the preview in rviz only shows the semantic+depth image, when in console, it gives me warning that

Detected jump back in time of 1.58489e+09s. Clearing TF buffer.

I wonder what did I miss or make it wrong? I tried to change every parameter in launch file but I think the issue is irrelevant to the launch file, also I tried to generate different dataset with different format and scenario, but it's also not the case. Please kindly give me some guide

[CaoYuchen]

Hi, I have solved the problem above, it's because the /clock message was wrongly set up by me. But it now still can't run my dataset, there is no warning complaining though. It just doesn't create any mesh. Can you tell me the requirements for the input dataset? Like do I need to take care of the fps, minimal time interval, or something else? I made my dataset almost same with Kitti, it's just my fps of image is 30 and fps of TF is 300. I don't know if this is a problem.

[violetteavi]

It may be an issue with your frame id's. You need to specify:

1. Your left and right camera image topics AND frame id's to Kimera
2. Your left and depth camera image topics to Kimera-Semantics

Please also note that if your left and depth cameras are not on the same frame, you will need to transform the depth onto the left frame using ros's depth_image_proc/register node. An example of this is in kimera_metric_realsense.launch in this PR

[bryce-wang-315]

Hi @CaoYuchen, You can save the whole 3D mesh by calling the generate_mesh rosservice:

rosservice call /kimera_semantics_node/generate_mesh

This will save a .ply file in the mesh_results folder. The .ply stored depends on this argument in the launch file:

 <param name="mesh_filename"           value="$(find kimera_semantics_ros)/mesh_results/$(anon tesse).ply" />

Btw, Kimera-Semantics is using Voxblox under the hood, so the documentation for Voxblox is applicable to Kimera-Semantics (mostly). I really recommend reading it here. We don't have a script to extract the individual semantic objects, but with the resulting .ply file you could get all triangles corresponding to a particular object class, say chairs, by looking at their rgb (color) values. Checkout this file for a mapping between colors and semantic labels.

Thanks for your answer @ToniRV . I made my own dataset exactly same with the given Kitti dataset like this:

And I also change the launch file and tesse_multiscene_office1_segmentation_mapping.csv accordingly to label my semantic objects. However, when I launch the file, it doesn't reconstruct mesh like running Kitti, the preview in rviz only shows the semantic+depth image, when in console, it gives me warning that

Detected jump back in time of 1.58489e+09s. Clearing TF buffer.

I wonder what did I miss or make it wrong? I tried to change every parameter in launch file but I think the issue is irrelevant to the launch file, also I tried to generate different dataset with different format and scenario, but it's also not the case. Please kindly give me some guide

Hi @CaoYuchen, You can save the whole 3D mesh by calling the generate_mesh rosservice:

rosservice call /kimera_semantics_node/generate_mesh

This will save a .ply file in the mesh_results folder. The .ply stored depends on this argument in the launch file:

 <param name="mesh_filename"           value="$(find kimera_semantics_ros)/mesh_results/$(anon tesse).ply" />

Btw, Kimera-Semantics is using Voxblox under the hood, so the documentation for Voxblox is applicable to Kimera-Semantics (mostly). I really recommend reading it here. We don't have a script to extract the individual semantic objects, but with the resulting .ply file you could get all triangles corresponding to a particular object class, say chairs, by looking at their rgb (color) values. Checkout this file for a mapping between colors and semantic labels.

Thanks for your answer @ToniRV . I made my own dataset exactly same with the given Kitti dataset like this:

And I also change the launch file and tesse_multiscene_office1_segmentation_mapping.csv accordingly to label my semantic objects. However, when I launch the file, it doesn't reconstruct mesh like running Kitti, the preview in rviz only shows the semantic+depth image, when in console, it gives me warning that

Detected jump back in time of 1.58489e+09s. Clearing TF buffer.

I wonder what did I miss or make it wrong? I tried to change every parameter in launch file but I think the issue is irrelevant to the launch file, also I tried to generate different dataset with different format and scenario, but it's also not the case. Please kindly give me some guide

When I record the demo bag to play, I also meet the same problem.I'm just re-recording the original demo. it gives me the same warning that

`Detected jump back in time of 1.58489e+09s. Clearing TF buffer.