Without masks, unable to extract mesh and low quality NeRF with DTU dataset (scan122 was tested)

[kelvincai522]

I am testing NeuS with the DTU dataset (scan122 particularly). I am only using its RGB images without masks and computed new camera poses using COLMAP. However, it is not able to extract mesh and the NeRF is in low quality. It converged fine with masks on though. All other configs are defaults. What could I change to improve quality and extract the NeuS mesh without masks as input? Thanks

[kelvincai522]

Actually, I had to change to VanillaMLP instead of FullyFusedMLP in the colmap config, as tiny-cuda-nn errored at my V100.

[bennyguo]

Hi! The choice of the MLP network does not make much difference. The key here should be to use a background model. You could wait for my implementation, which is expected to come out in a few days.

[kelvincai522]

I see. Great! Looking forward to it. Thanks for the reply.

[bennyguo]

@kelvincai522 I've just pushed a new branch that supports NeuS training with a background model. I'm still testing this feature on different scenes. Welcome to try it out!

[kelvincai522]

@bennyguo Thank you for the new branch! I tested it with my own dataset and found the following in my Fedora with a V100 GPU (16GB vram):

• train/inv_s in the log never change (increase/decrease) while the loss is decreasing but dropped down to nan though
• I had to change model.texture.mlp_network_config.otype from FullyFusedMLP to VanillaMLP. Otherwise, tcnn would have Internal Error (I had to do that in previous versions as well).
• I had to disable lambda_distortion(_bg) by changing it to 0. Otherwise, it would error with "RuntimeError: max(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument."
• I had to change occupancy_grid_bg resolution from 256 to 128. Otherwise, Cuda is OOM.
• Had a warning during validation: data/users/kcai/instant-nsr-pl/utils/mixins.py:90: RuntimeWarning: invalid value encountered in true_divide | 0/2 [00:00<?, ?it/s] img = (img - img.min()) / (img.max() - img.min())
• While the img converge okay (not as good as nerf unbounded), it couldn't extract a mesh with the following error: File "/data/users/kcai/instant-nsr-pl/models/geometry.py", line 107, in isosurface vmin, vmax = mesh_coarse['v_pos'].amin(dim=0), mesh_coarse['v_pos'].amax(dim=0) IndexError: amin(): Expected reduction dim 0 to have non-zero size.

[bennyguo]

Thanks for the detailed information about your experiment! Is it possible to provide the dataset you used so that I could try it on my side? It would be very valuable!

[kelvincai522]

That would be great. Here is the dataset https://drive.google.com/file/d/1EPhaA7vg9P3neH-T7MiVUIKE9nwgU04T/view?usp=sharing

[bennyguo]

It seems like most of the errors were caused by divergence. By setting model.radius=1.5 and turning off all the regularizations, i.e. system.loss.lambda_distortion=0. system.loss.lambda_distortion_bg=0. system.loss.lambda_sparsity=0., I was able to fairly reconstruct the scene: I'll come up with an automatic strategy to determine model.radius and try to fix the stuffed area below the table.

[bennyguo]

About the error using FullyFusedMLP, you may have to run tiny-cuda-nn samples to see if the installation works.