search

nexuslrf / SPIDR

Codes for: "SPIDR: SDF-based Neural Point Fields for Illumination and Deformation"
https://nexuslrf.github.io/SPIDR_webpage
Other
76 stars 4 forks source link

question about SDF extraction #4

Closed Youngju-Na closed 1 year ago

Youngju-Na commented 1 year ago

Hi, thanks for sharing this work!

I was wondering about the SDF network. Because if we use Point-NeRF in estimating SDF as a baseline, the model would not learn the SDF values where the neural points don't exist.

Especially when we are converting SDF to mesh, we make a 3d grid and query every single voxel to get the SDF value. The SDF network would collapse when estimating the space where the model hasn't seen during training. How did you handle this?

And secondly, did you test the SDF network with the DTU dataset?

Thanks! I hope my question gets across to you well.

nexuslrf commented 1 year ago

Hi, This is indeed a problem for such hybrid-nerf structures. Since the model inputs now become local, discrete, and sparse points. The estimated SDF largely depends on where points locate. The SDF for the region without points will become undefined values.

That's why the point-growing process is important for this model (unlike other grid-based nerf models which start with dense grids).

The way I extract mesh from the estimated SDF is basically the same as the way you do, but I additionally cleaned/filtered the undefined SDF values (https://github.com/nexuslrf/SPIDR/blob/main/deform_tools/mesh_editing.py#L9).

As for the DTU dataset, I tried it before, but the DTU used in the original point-nerf is quite different from the ones used in other neural surface models. To avoid confusion, I didn't further test DTU.

Hope my info could help you.

Youngju-Na commented 1 year ago

Thanks for the answer! It really helped me to understand your approach.

What do you mean by "DTU used in the original point-nerf is quite different from the ones used in other neural surface models"?

I'd like to try your method to the DTU dataset and I want to clarify this.

Sincerely,

nexuslrf commented 1 year ago

In the original point-nerf, DTU is used to showcase their generalizable NeRF, which makes the DTU dataset in original point-nerf has a different training/testing routine. In addition, they also include a lot pre-processing/constraints for DTU (https://github.com/Xharlie/pointnerf/tree/master/data/dtu_configs). Given these reasons, I do want to evaluate models on DTU.

However, if you insist on running on DTU for a SDF-like reconstruction, I suggest you rewrite a dataloader to make training data follow the same form as other neural surface models.