search

allenai / ai2thor

An open-source platform for Visual AI.
http://ai2thor.allenai.org
Apache License 2.0
1.09k stars 210 forks source link

Inaccurate instance and semantic segmentation #1089

Open vlongle opened 1 year ago

vlongle commented 1 year ago

Hi,

For FloorPlan401, it seems that the segmentation is not quite right.

Screen Shot 2022-10-01 at 1 53 28 PM

Colab notebook: https://colab.research.google.com/drive/1NOmfy62PjvPuxsa7pV-fB6ElZT55puDU?usp=sharing

For example, the bathtub instance segmentation (greenish) overlaps with what should have been classified as a floor. Is this an unexpected behavior? Are there anyways to fix it? Also, out of curiosity, how is the segmentation done?

Thanks!

winthos commented 1 year ago

Hello!

So this segmentation was actually a light ray volume that was erroneously tagged as a solid part of the environment, which caused it to be segmented and block out the things behind it. The large green volume you were seeing is actually this light ray mesh. This has been fixed in the referenced commit and will be live in the next major release of THOR.

The segmentation is basically done by gathering all meshes associated with a specific object and then masking it with a color. The Semantic Segmentation colors all objects of the same type as the same color (ie: if there are 3 chairs in a room, all will be the same color) while the Instance Segmentation separates all instances of individual objects such that they all have a unique color (the 3 chairs would each be a different color).