Access semantic scene mesh directly

[rfalcon100]

❓ Questions and Help

Hi,

Is it possible to get access to the full scene mesh with semantic annotations, instead of just rendering images from the sensors? In particular, I am using the replica dataset, and I want to parse the scene into a list of surfaces (preferably triangles), where each triangle is assigned to a semantic class.

I have tried reading the mesh.ply and /habitat/mesh_semantic.ply files with a few mesh loading tools like https://github.com/nschloe/meshio or https://github.com/intel-isl/Open3D, but I can only get point clouds and not a list of surfaces. Is there a way to do this directly from Habitat-sim?

[aclegg3]

Unfortunately we don't provide an mesh-level access in the API at this time.

If you are comfortable with C++ you could try adding this support and writing a PR or forking and adapting our code to fit your needs. You may be most interested in this file where .ply parsing is done and semantic meshes are split by object id as you are hoping to do.

[Skylion007]

We have some bindings to access the NavMesh directly, feel free to look there for examples about how this could be done.

On Tue, Feb 9, 2021 at 11:31 AM Alexander Clegg notifications@github.com wrote:

Unfortunately we don't provide an mesh-level access in the API at this time.

If you are comfortable with C++ you could try adding this support and writing a PR or forking and adapting our code to fit your needs. You may be most interested in this file https://github.com/facebookresearch/habitat-sim/blob/master/src/esp/assets/GenericInstanceMeshData.cpp where .ply parsing is done and semantic meshes are split by object id as you are hoping to do.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/facebookresearch/habitat-sim/issues/1081#issuecomment-776067069, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPVMX3C26OMCOT7I4LYM6DS6FPPNANCNFSM4XLF6URQ .

[rfalcon100]

Thanks for the info, I'll see what I can do. But I am a bit confused, should parse the mesh_semantic.ply or the navmesh?

[Skylion007]

@rfalcon100 Referring to add new bindings to access the underling semantic mesh from the MeshData like so: https://github.com/facebookresearch/habitat-sim/blob/da03230101ca7b7b77ea5df90b8a74806d413942/src/esp/bindings/ShortestPathBindings.cpp#L100

[rfalcon100]

It seems, I am am able to get the vertices , surface, and semantic object data that I need. However, I would like to get the surface normals too. It seems like normals are not store in the GenericInstanceMeshData object: https://github.com/facebookresearch/habitat-sim/blob/87364fd0f7f36589f055b61fa41a7199d8ee4c97/src/esp/assets/GenericInstanceMeshData.cpp#L161

Would it make sense to store and expose the normals too?

[neepacmu]

@rfalcon100 can you share exactly how did you get the vertices , surface, and semantic object data? Which files did you parse?

[mahnoor-fatima-saad]

Hi @rfalcon100

Could you please elaborate on how you were able go access the mesh data directly? I'm also very interested in this problem

[aclegg3]

To follow up here, the mesh data is exposed in C++, but you'd need to write accessors and bindings to expose it as in #1149.

Note that internally, there are two meshes being loaded. One with semantics and the other with RGB textures.

In some datasets (e.g. MP3D and Gibson) the semantics are embedded per-vertex in the semantic mesh. These could be acquired per-triangle without losing resolution, but the mesh geometry is slightly different from the RGB mesh.

In other datasets (e.g. HM3D) the semantics are embedded in textures and there is no straightforward way to bake these into the geometry without losing information. For example, a semantic boundary can run through the middle of a triangle in UV space.