Replica Test

[utopiac]

Hi,

Thanks for your brilliant work! I was facing an issue when testing the SMNet performance in Replica. I noticed the encapsulated function "HabitatUtils" was not appropriate for Replica when loading scenes. Would you let me know how you perform it or would you upload the related code?

Thanks!

[vincentcartillier]

I believe if you should be able to use the HabitatUtils class with replica. Could you try something like this:

hab = HabitatUtils(scene, 0, housetype='replica')

[utopiac]

Thanks for your quick reply! But it still doesn't work. The error is as follows:

21:19:54.200434 22240 simulator.py:170] Loaded navmesh /replica_v1/apartment_0/habitat/mesh_semantic.navmesh Traceback (most recent call last): File "data/build_point_cloud_from_mesh_h5.py", line 29, in objects = habitat.get_objects_in_level() File "/Semantic-MapNet/utils/habitat_utils.py", line 249, in get_objects_in_level assert self.level == int(self.semantic_annotations.levels[self.level].id) IndexError: list index out of range

[vincentcartillier]

I see. The issue is coming from the get_objects_in_level function. However Replica does not have floor level annotation. I have just pushed a patch that should handle that.

All but one scene in Replica have only 1 floor. For the apartment_0 scene you can create the object point cloud for the entire house and then manually separate the point cloud into 2 files using a threshold over the up-right axis. Specifically, we used a threshold of 1.2.

[utopiac]

Thanks a lot again! It still has another issue.

00:49:28.198795 28725 simulator.py:170] Loaded navmesh /replica_v1/apartment_0/habitat/mesh_semantic.navmesh Traceback (most recent call last): File "data/build_point_cloud_from_mesh_h5.py", line 30, in objects = habitat.keep_objects_in_whitelist(objects) File "/Semantic-MapNet/utils/habitat_utils.py", line 299, in keep_objects_in_whitelist name = o.category.name(mapping='mpcat40') AttributeError: 'NoneType' object has no attribute 'name'

[utopiac]

Hi @vincentcartillier ,

Would you upload the groudtruth to google drive directly, I am worried that the generated GT is different from yours, which affects the test performance of the replica. Thanks a lot!

[vincentcartillier]

That first issue is because the first element in the all_objects list is None - that's is Specific to Replica scenes. So if you skip the first element then you should be fine. Unfortunately I can't upload the GT semantic maps for Replica at the moment.

[utopiac]

We should always load '/replica_v1/{scene}/habitat/mesh_semantic.ply' whenever in 'HabitatUtils' or in 'PlyData'. Is that right?

[utopiac]

Hi @vincentcartillier,

I still have two questions. (1)You mentioned earlier splitting the point cloud of apartment_0 into two parts using the threshold. So I want to know how can I manually separate it using the threshold. (2)The houses_dim_replica.json file contains information about the dimensions of only two scenes, the other scenes are missing. Thanks a lot!

[vincentcartillier]

Yes you should load the mesh_semantic.ply when working with the Replica dataset.

(1) You can load the point cloud and then apply a threshold along the up-right axis. So something like this should do:

f = h5py.File('filename_full_apartment_0.h5','r')
vertices = np.array(f['vertices'])
f.close()

y_values = vertices[:,1]
bottom_mask = y_values < threshold
top_mask = ~bottom_mask

bottom_vertices = vertices[bottom_mask]
top_vertices = vertices[top_mask]

(2) I have just pushed the houses dimensions for all scenes in Replica.

[utopiac]

Thanks!

[utopiac]

Hi @vincentcartillier , I followed your instructions to generate the ground truth of the replica dataset. I have tested SMNet on replica dataset. , but the result is worst than you presented. The result is as follows: Mean IoU: 27.85 Overall Acc: 87.84 Mean Recall: 59.14 Mean Precision: 34.94

per class IoU: 88.92 void 0.00 shelving 0.00 chest_of_drawers 20.15 bed 29.63 cushion nan fireplace 65.70 sofa 59.05 table 52.56 chair 3.38 cabinet 0.00 plant 0.00 counter 14.80 sink

Precision: 96.28 void 0.00 shelving 0.00 chest_of_drawers 21.89 bed 36.10 cushion nan fireplace 77.85 sofa 73.76 table 76.82 chair 7.21 cabinet 0.00 plant 0.00 counter 29.33 sink

Recall: 92.08 void nan shelving nan chest_of_drawers 71.72 bed 62.30 cushion nan fireplace 80.80 sofa 74.75 table 62.47 chair 5.98 cabinet nan plant nan counter 23.00 sink

[utopiac]

But with the ground truth of mp3d that you provided the performance of SMNet is almost the same as you showed on the paper. I don't know if the ground truth of the replica I generated is the cause of the poor performance of SMNet on the replica.

[vincentcartillier]

It's a little hard to debug with the provided information. Have you tried visualizing the predicted and ground-truth maps?

[utopiac]

I have visualized the predicted map and ground truth of 'apartment_0_0'. The top is the predicted map, the bottom is the ground truth.

[vincentcartillier]

You seem to be missing a bunch of objects/classes in your GT maps. Things like plant/counter/shelving etc..

[utopiac]

I generated the ground truth of the replica as you instructed, but for some reason, there are some objects missing. I wonder if it is related to HabitatUtils. I would appreciate it if you could upload the GT maps. Thanks a lot!