How was the semantic mesh generated

[kaiwangntu]

Hi, According to the paper, the point cloud is obtained by sampling the reconstructed mesh and then annotated. So,

1. Is the rgb mesh provided the reconstructed one pls?
2. If yes, how was the semantic information of the semantic mesh obtained? 'cos only a subset of the vertices(i.e. the point cloud) are annotated, how did u annotate the semantic information of the other vertices?

[alexsax]

Hi, Let me respond inline:

1) Yes, all the meshes are reconstructed, including the RGB mesh. 2) Check out this fly-though video. The points are sampled quite densely, and there are many annotations for each face. We use these points to annotate each face (and we don't annotate individual vertices).

[kaiwangntu]

HI Alex,

1. By 'reconstructed', do you mean the RGB mesh is just the output of Matterport?
2. According to the illustration in the paper, the point cloud is sampled from the Matterport reconstructed mesh, and then annotated. I'm just wondering if the vertices of the semantic mesh correspond to the annotated point cloud, or those of the originally reconstructed one(i.e. the Matterport output)? Actually, both answers will make me puzzled. If the former guess is right, then the RGB mesh is not the direct output of Matterport ('cos the no. of vertices of the RGB mesh equals to that of the semantic mesh). In that case, how did you process the originally reconstructed mesh? If the latter is correct, how did you manage to get the semantic information of the other vertices of the semantic mesh (since the annotated points are just a subset of the semantic mesh vertices).

[alexsax]

1) Yes. Matterport provides a 3D mesh which is reconstructed from the various camera sweeps in each building. Matterport also provides the RGB textures for this mesh.

2) Your former guess is correct. The only difference between the semantic and RGB meshes is that the semantic textures contain the semantic label. The mesh geometry still comes directly from Matterport, and I believe that they reconstruct it from the sensor depthmaps (which are not included in the released dataset).

[kaiwangntu]

Did you mean the RGB/semantic mesh geometry in the dataset provided is just the one reconstructed directly from Matterport? If only parts of the vertices of the Matterport output mesh, i.e. the vertices correspond to the sampled point cloud, are annotated, how did you compute the semantic information of the remaining vertices pls (as you said 'the mesh geometry still comes directly from Matterport')?

[alexsax]

Yes, exactly. Regarding how we labeled the mesh: we labeled the faces by projecting the points from the pointcloud onto the mesh. The face label is the majority class of the projected points. Since the faces are the parts that are labeled, it the "class of a vertex" isn't well-defined because a vertex can be a part of multiple faces.

[kaiwangntu]

Got it! Thanks for your detailed explanation Alex!