Cleaned Floor Labels

[yash-goel]

Hi,

Thank you for the excellent work!

I was using your scripts to recompute GT semantic maps by generating my own object point clouds since I am looking to use the datasets with different settings and more houses.

I soon realized that some houses contain objects with wrong level id. Is that the reason the cleaned floor labels for object point cloud were provided? I also saw some of the object point clouds were missing in the drive for which the objects are wrongly marked e.g. p5wJjkQkbXX_1. I would be really thankful if you could provide them.

Apart from this, a much larger question is - how did you generate these cleaned object point clouds? Was this done manually? If you could detail the process it will be really helpful since I am looking to use more houses than what is mentioned in the paper for which I will have to generate the object point clouds myself.

Thanks a lot in advance.

Yash

[vincentcartillier]

Hi @yash-goel ,

(1) That is exactly the reason why we provide the cleaned floor labels for objects. The floor labels in Matterport have been automatically annotated, and thus there is some mistakes here and there.

(2) The drive provides all houses used in the paper. If you want to run experiments with other/more houses you would have to recompute the object point cloud using the provided scripts. To set the floor level, you can either do it manually, or define your own set of rules to set it. Another solution would be to generate the point cloud from an agent tour. You would un-project the depth pixels at each time step (using the camera pose, camera intrinsics + depth) and accumulate a point cloud along the trajectory. If you restrict the agent to a given floor you will more or less end-up with an object point cloud of that given floor. This method is not perfect as the agent might be able to see what's on other floors (eg. mezzanine).

(3) The object point clouds are generated from the houses semantic meshes directly. We start by selecting the object surfaces of interest (the ones in our list of object categories). We then dense sample points on these surfaces (triangles).

[yash-goel]

Hi @vincentcartillier,

Thanks for the prompt and detailed reply!

So, by the method of this agent tour is how you generated the point cloud with cleaned object labels? Because using the scripts, the objects for level are taken based on their id (habitat_utils.py) - which are sometimes here and there as you mentioned.

I would also assume that while doing this agent tour - one also stores the information from ego-centric instance/ semantic segmentation against the point cloud as it is needed to generate the top-down semantic maps later.

Thank you!

[vincentcartillier]

The method used to generate the GT semantic maps in the paper is the one using the semantic meshes (with the noisy floor labels). See https://github.com/vincentcartillier/Semantic-MapNet/blob/main/data/build_point_cloud_from_mesh_h5.py . On L26 we restrict the list of objects to the ones of the given floor. This is done using the object floor id which is sometimes noisy (unfortunately..).

I have added a script to create the GT semantic maps from the agent tours. https://github.com/vincentcartillier/Semantic-MapNet/blob/main/compute_GT_topdown_semantic_maps/build_semmap_from_egoGT.py And you're right, to use this method you need to get the egocentric instance segmentation information.

[yash-goel]

Ah, I see. Thanks for the script too! Much appreciated.