Encoder design in DMTet

[irichyoung]

Hi, Great work. Benefit a lot from kaolin!

I've learned the paper "DMTet" which describes PVCNN as the input encoder. However, I only find "MLP + positional encoding" in kaolin implementation.

So, how could this implementation ensure that the point embedding could capture the local/whole geometry information if there is only point-wise "positional encoding"? (PVCNN could do this, but i 'm not sure if Positional encoding from Nerf could do this)

[frankshen07]

Thank you for your interest in our work! The tutorial in Kaolin is just a simple example where we showcase the usage of our representation, DMTet, which is independent of the network used to parameterize it. To model more complex shapes or a batch of shapes you might need a network with stronger capacity to parameterize DMTet, like PVCNN. However, for overfitting a single shape we found MLP + proper positional encoding is typically capable enough to capture local details. This is also observed in other works like SIREN or instant-ngp.

[irichyoung]

Thank you for your interest in our work! The tutorial in Kaolin is just a simple example where we showcase the usage of our representation, DMTet, which is independent of the network used to parameterize it. To model more complex shapes or a batch of shapes you might need a network with stronger capacity to parameterize DMTet, like PVCNN. However, for overfitting a single shape we found MLP + proper positional encoding is typically capable enough to capture local details. This is also observed in other works like SIREN or instant-ngp.

Glad to get a timely reply from the author!

I've read your paper carefully. One of the brilliant parts of the representation in DMTet is the refinement for my taste. As you said, It trims the tet grid and scales it quadratically rather than cubically. However, it seems the part has not been implemented in kaolin as well.

Bty, is there any plan to release the complete code of DMTet in kaolin or standalone?

[frankshen07]

The volume subdivision is added to Kaolin recently: https://kaolin.readthedocs.io/en/latest/modules/kaolin.ops.mesh.html?highlight=subdivide#kaolin.ops.mesh.subdivide_tetmesh. We are planning to update the tutorial to showcase refinement with subdivision, which should be straightforward. Basically, we use this function to subdivide tet grid, running network inference ONLY on surface tets and adding predicted values as residuals to the initial SDF. About DMTet code release, we plan to add all modules (e.g. Marching tet, subdivisions etc) into Kaolin first and then release DMTet as a standalone repo.

[irichyoung]

The volume subdivision is added to Kaolin recently: https://kaolin.readthedocs.io/en/latest/modules/kaolin.ops.mesh.html?highlight=subdivide#kaolin.ops.mesh.subdivide_tetmesh. We are planning to update the tutorial to showcase refinement with subdivision, which should be straightforward. Basically, we use this function to subdivide tet grid, running network inference ONLY on surface tets and adding predicted values as residuals to the initial SDF. About DMTet code release, we plan to add all modules (e.g. Marching tet, subdivisions etc) into Kaolin first and then release DMTet as a standalone repo.

Good to hear that! Looking forward to the new version! I wanna get a more empirical understanding of this work. Is there anything I can take part in for now?

[ErXXoor]

May I ask if the mesh refinement part using gcn has been added?

[fzy139]

Looking forward to the release of full DMTet repo. Btw, does the current kaolin.ops.conversions.marching_tetrahedra implementation support extraction from the locally subdivided tet grid, as demonstrated in the paper?

Thanks