search

MIT-SPARK / Kimera-Semantics

Real-Time 3D Semantic Reconstruction from 2D data
BSD 2-Clause "Simplified" License
633 stars 134 forks source link

Is it possible to make Kimera Semantics instance-aware? #32

Closed zhaozhongch closed 3 years ago

zhaozhongch commented 4 years ago

If I use something like maks RCNN to generate images having instance segmentation, i.e. same class have different colors, will the Kimera-Semantics be able to handle it? If not, any suggestion about how to do that? I noticed that in kimera semantics config file kimera_semantics_ros/cfg/tesse_multiscene_office1_segmentation_mapping you have something like

...
Painting_A,246,188,6,255,12
Painting_B,246,188,6,255,12
Painting_C,246,188,6,255,12
Painting_D,246,188,6,255,12
...

I am just wondering for the same class 12 if those paints can have a different color setting

ToniRV commented 4 years ago

Hi @zhaozhongch, that's right, you can use MaskRCNN, the semantic label corresponds to the last column (number 12 above).

I guess a quick approach to get this working with MaskRCNN would be to connect Kimera-Semantics with the semantic class output of MaskRCNN converted to a grayscale image where the pixel values correspond to the semantic class id. You may want to reserve as well a special id for non-segmented pixel. You can also change the covariance matrix here. I've just set it to be isotropic (constant diagonal matrix).

I had to add the decoding from colors to semantic labels because our segmentation (semantic mask) is given as RGBA colors instead of ids. The only good thing about this approach is that the semantic images and the 3d reconstruction have the same colors. For anything else, this is suboptimal because I am constantly mapping semantic ids to colors and vice-versa...