Training are very sensitive to network initialization

[gerwang]

Hello, Thank you for the great work! I really like the neat project structure so I am experimenting with NeRF on this codebase. However, I find the training is very sensitive to network initialization.

First, if I trained on the hotdog scene for 2000 iterations using default configs in the options/nerf_blender_repr.yaml, I would get a normal render from self.nerf, but an empty render from self.nerf_fine.

The command is

python train.py --group=nerf-debug --model=nerf --yaml=nerf_blender_repr --name=debug-hotdog-nerf-9 --data.scene=hotdog

The result shown in tensorboard is

After some investigation, I found that the reason lay in the different initialization weights of the two networks. Then I tried changing the random seed from 0 to 233, this time both networks rendered an empty scene.

The command is

python train.py --group=nerf-debug --model=nerf --yaml=nerf_blender_repr --name=debug-hotdog-nerf-10 --data.scene=hotdog --seed=233

The result is

Finally, I tried copying the self.nerf's weights to self.nerf_fine, using the following code

          if opt.nerf.fine_sampling:
              self.nerf_fine = NeRF(opt)
              self.nerf_fine.load_state_dict(self.nerf.state_dict()) // here

and set the random seed back to 0. This time the result was fine.

The command is

python train.py --group=nerf-debug --model=nerf --yaml=nerf_blender_repr --name=debug-hotdog-nerf-11 --data.scene=hotdog

The result is

Here, I want to post the results to discuss the general stability of NeRF training. I wonder if other NeRF repositories all have this kind of sensitivity to network initialization, or are we missing some initialization trick in this repo?

I believe this can relate to a more fundamental nature of NeRF. Do you have any ideas about this phenomenon? Thank you!

[chenhsuanlin]

Hi @gerwang, this is most probably related to large amounts of white background in this particular synthetic dataset. Please see this issue (bmild/nerf#29) from the original NeRF repo for discussions -- the authors mentioned a warm-up trick by optimizing only on the center cropped region in the first few iterations. This could be worked around by just trying different seeds to get a better initialization, but yes, I think it's a problem of NeRF in general; would be interesting to investigate.

[gerwang]

Thanks for your explanation and reference!

I found setting the density_noise_reg from 0 to 1e0, according to the original nerf repo, solved the convergence problem in my case. In the issue you referenced, the author encountered convergence problems when training on the more challenging mic scene, so I believe both the density_noise_reg and precrop contribute to the improvement of convergence.