When I train on the synthetic nerf dataset by running:
python train_single.py --eagerly --data_dir data/nerf_synthetic/lego --img_wh 128 --batch_size 2 --name lego --ray_chunks 2048
I get zero gradients after the second step in the first epoch:
2/50 [>.............................] - ETA: 2:13 - coarse_loss: 0.0676 - coarse_psnr: 11.9349 - coarse_ssim: 0.3363 - fine_loss: 0.0681 - fine_psnr: 11.9008 - fine_ssim: 0.33512023-03-17 23:17:19,600 | root | ERROR | Both Coarse and Fine Gradient are zero (nerf.py:442)
I tried different datasets and the coarse model, the fine model and sometimes both return zero gradients. This seems to be random for each run while using the same parameters. It is rare that both the coarse and fine model start converging correctly.
Even when letting the training run for all epochs, the model gets stuck after the first one.
When I train on the synthetic nerf dataset by running: python train_single.py --eagerly --data_dir data/nerf_synthetic/lego --img_wh 128 --batch_size 2 --name lego --ray_chunks 2048
I get zero gradients after the second step in the first epoch: 2/50 [>.............................] - ETA: 2:13 - coarse_loss: 0.0676 - coarse_psnr: 11.9349 - coarse_ssim: 0.3363 - fine_loss: 0.0681 - fine_psnr: 11.9008 - fine_ssim: 0.33512023-03-17 23:17:19,600 | root | ERROR | Both Coarse and Fine Gradient are zero (nerf.py:442)
I tried different datasets and the coarse model, the fine model and sometimes both return zero gradients. This seems to be random for each run while using the same parameters. It is rare that both the coarse and fine model start converging correctly.
Even when letting the training run for all epochs, the model gets stuck after the first one.
How can I fix this?