ruiqixu37
commented
4 months ago Hello,
Thank you for posting this discussion. I first want to clarify that I am not the author of the paper and this repo is an unofficial implementation of Nuvo.
For the question regarding surface point detection in NeRF, I recommend going through the "Point Sampling" paragraph under "Experimental Details" in the supplementary material. I agree with you that vanilla NeRF doesn't provide information on whether a point is on the object's surface or not. It seems additional tricks based on Zip-NeRF are required for Nuvo to perform point sampling on NeRF instances. Unfortunately I am not familiar with the paper and can't provide explanation on how it works.
Hope this is helpful. If you figured out how to make Nuvo work with NeRF and would like to collaborate, feel free to make a PR!
seanzhuh
Hi, congratulations on the ECCV acceptance of this wonderful work. Since I'm new to computer graphics and 3D modeling, I'm quite confused how the symmetrical chamfer distance can approximate the surface or near the surface of the 3D object.
I noticed that the chamfer distance might come from this paper "A Point Set Generation Network for 3D Object Reconstruction from a Single Image", where given ground-truth point clouds of the object, authors learn to predict point clouds that minimize the similarity with ground-truth point clouds.
However, in NeRF, there is no annotation for which point lies on the surface of the object, yet still, Nuvo claims that it can establish a one-to-one correspondence between surface 3D points or nearby the surface and 2D points u in texture map. I'm really curious of how to identify which 3D points are surface points and connect it to 2D points u.
One explanation might be from NeuTex, it says voxels with high volume density occupies a large portion of the contribution to the final pixel color. Hence, it uses the volume density and transmittance to weight the cycle loss. With this, if 3D points that are not nearby the surface (for example, inside the object) got mapped to some point in texture map, it only incurs a small cycle loss I guess because of the small weight coefficient (density is low because it is inside the object). However if the points are on or nearby the object surface, its volume density is high, thus if the texture mapping and inverse mapping MLPs can not correctly connect them to some points in the texture map, then the loss is high, therefore NeuTex I think will establish the correspondence between 3D surface points and 2D texture map.
But Nuvo does not weigh the cycle loss, rather it adopts chamfer distance loss to encourage a similar effect from my understanding. But how this is achieved?
Really looking forward to author's explanations on this and it would be greatly appreciated. Thanks!