Not able to produce good quality reconstructions when enforcing normal loss on RGBD data

[kaushikbalasundar]

Hello! Thank you for releasing this amazing codebase!

I wanted to try enforcing normal loss on an RGB-D dataset. To do this, I did the following:

1. I generated normals from the Omnidata model, and prepared the .npy files as necessary.
2. Since we have depth from the dataset, I discarded the predicted depth from the Omnidata model and only used the sensor depth for enforcing depth consistency in the loss ( I therefore set --pipeline.model.mono-depth-loss-mult 0).
3. I resized the depth images to be the same size as the output of the OmniData model. I got a shape mismatch after about 4800 training steps if I didn't do this.
4. I used the NeuS backbone with RGB-D data from synthetic neural-RGBD dataset - similar to the example provided in the docs
5. I trained this with 2048 rays / batch. The full command I used was as follows:

ns-train neus --pipeline.model.sdf-field.use-grid-feature True --trainer.steps_per_save=10000 --pipeline.model.sdf-field.hidden-dim 256 --pipeline.model.sdf-field.num-layers 2 --pipeline.model.sdf-field.num-layers-color 2 --pipeline.model.sdf-field.use-appearance-embedding True --pipeline.model.sdf-field.geometric-init True --pipeline.model.sdf-field.inside-outside True  --pipeline.model.sdf-field.bias 0.8 --pipeline.model.sdf-field.beta-init 0.3 --pipeline.datamanager.train-num-images-to-sample-from -1 --trainer.steps-per-eval-image 5000 --pipeline.model.background-model none --vis wandb --experiment-name with_normals_depth --pipeline.model.sensor-depth-l1-loss-mult 0.1 --pipeline.model.sensor-depth-freespace-loss-mult 10.0 --pipeline.model.sensor-depth-sdf-loss-mult 6000.0 --pipeline.model.mono-normal-loss-mult 0.05 --pipeline.model.mono-depth-loss-mult 0 --pipeline.datamanager.train-num-rays-per-batch 2048 --machine.num-gpus 1 sdfstudio-data --data data/custom/neural_rgbd_normals --include_sensor_depth True --include_mono_prior True --skip_every_for_val_split 30

The training curves are as follows:

I thought that enforcing normal consistency might improve results in large regions with less texture. However, the results I got without it looks much better. Here is an example for the kitchen scene - on the left is an image with normal loss enforced and on the right is without it. All other parameters are the same.

Any thoughts on what I could be doing wrong?

[niujinshuchong]

@kaushikbalasundar Maybe the normal from omnidata is noise? Could you try to set different weights for the normal loss and check what happens? And maybe also disable sensor depth loss and compare it to using normal loss only?