Question about neus with nerfacc

[Jiakui]

Hi Ruilong,

It looks for me that, although much faster, the neus with hash encoding is much worse than the vallina MLP version neus (I tried both instant-nsr and instant-nsr-pl , they are both not as good as neus, so I guess hash encoding should be responsible for the degradation of the reconstruction quality ) . Do you have any idea why ?

Have you tried the vallina MLP version neus with your nerfacc, without the hash encoding part ? I think it should be faster than neus , and bettter quality than instant-nsr / instant-nsr-pl.

Thanks!

[liruilong940607]

Hi the neus is a 3rd party impl. I didn't play with neus myself so have no intuition on this. I think it should not be too hard to play with the neus repo and replace hash grid with mlp. Though you would need to wait for some time to see the results.

[Jiakui]

OK, if you try Neus with nerfacc, please try it with the original vallina MLP version. I will also have a try and see the results.

[liruilong940607]

SG! Would be curious to see the difference.

[volcverse]

Hello, I'm also working on NeuS with nerfacc. But the density in NeuS needs view_dir to get. You can't get density like traditional NeRF with sigma=nerf.query_density(pts). Would you please tell me how you solve this problem?

[liruilong940607]

You can take reference from this impl of neus with nerfacc: https://github.com/bennyguo/instant-nsr-pl/blob/215274e2f99d2481ba00f3345fe3f3dd1b9c3d74/models/neus.py#L49-L60

Or, you can take reference from our example here: https://github.com/KAIR-BAIR/nerfacc/blob/488bca66fc05b5bff5982170f204052e75ae2a67/examples/train_ngp_nerf.py#L193-L208, where we calculate view-dependent step size on the fly with randomly sampled training cameras. In the case of neus I think you can do similar things -- randomly sample cameras and create view_dir at location x, then use network to get the alpha value.

The rule of thumb is that the occupancy grid should be used to cache the maximum possible opacity at location x. So if the opacity is dependent on some other factors, you need to randomly sample over other factors during training, so that the occupancy grid can capture the maximum opacity at location x.

[yyeboah]

@Jiakui AFAIK, this drop in surface quality when using hybrid representations (grid structure + non-linearity) is happening across the board. I stand to be corrected, but my initial intuition is that the introduction of a grid structure is doing something akin to the discretisation of a surface, leading to noise in the final extraction, something that does not happen in the vanilla SDF methods.