NN-based sdf may cause failure of surface extraction.

[Cenbylin]

Hi, I realy enjoy this excellent work.

I found the NN-based sdf may be unstable in the initial training stage.

Specifically, I replaced the geometric parameters (sdf,deform,cube_weight tensor) in nvdiffrec with a NN-based structure: query-->(pvcnn/triplane/voxel grid)-->MLP--> sdf,deform,cube_weight

If I randomly initiate the network, the mesh extraction will quickly fail because sdf predictions are all >0 or <0.

I have tried: 1) reduce learning rate, 2) pre-train using a known sphere sdf 3) change different network init method.

These methods did alleviate this problem, and successfully trained the network under certain random initialization. But it was very unstable (failure usually occurs during the first 100 steps).

As I am relatively new to this area, I would like to ask you for some advice about this. Do I need to reduce the weight of regularisation, or use point cloud supervision at the initial stage?

Thank you so much.

[frankshen07]

Sorry for the late reply.

An additional approach you might try is to fix the sign of the center and boundary vertices of the grid, ensuring the extracted surface won't be empty. This change could help the optimization recover from failure. For reference on how the indices are identified, you can look at GET3D's FlexiCubes geometry code. Besides this, I recommend checking out the setup of GET3D and other text-to-3D pipelines using DMTet, as they use isosurfacing combined with neural networks.

[Cenbylin]

Thanks for the advice!