search

yenchenlin / nerf-pytorch

A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.
MIT License
5.47k stars 1.06k forks source link

NaN value happens in the middle of training #82

Open Beniko95J opened 2 years ago

Beniko95J commented 2 years ago

Hi, thanks for the re-implementation of NeRF in pytorch. I am trying to run this repo on my own dataset, but NaN value happens in the middle of training. This confuses me since the training looks good in the start. Has anyone met this issue before?

image

AY1997 commented 2 years ago

I also met the same problem. Did you solve it now?

yuanyue commented 1 year ago

Same question here.

YuvalMandel commented 1 year ago

I have the same problem, Did anyone manage to solve it?

yuanyue commented 1 year ago

I have the same problem, Did anyone manage to solve it?

I reduce the learning rate and it works. You can try it.

YuvalMandel commented 1 year ago

Thanks! I tried to reduce from 1e-6 to 1e-9 and also 1e-12, but it doesn't see, to be doing much of a difference. I will also add that the NaN's don't always appear at the start, and usually appear in later iterations.

silver-obelisk commented 1 year ago

I increased factor,and the problem was sloved.But it is not a good way to increase factor,that means low resolution

silver-obelisk commented 1 year ago

I have the same problem, Did anyone manage to solve it?

I reduce the learning rate and it works. You can try it.

I wonder what lr you set? I change 5e-4 to 5e-5 but unuseful.