search

liuyuan-pal / NeuRay

[CVPR2022] Neural Rays for Occlusion-aware Image-based Rendering
GNU General Public License v3.0
409 stars 32 forks source link

A question that loss values are all NaN #7

Closed ZYY-666 closed 2 years ago

ZYY-666 commented 2 years ago

Have you ever encountered any cases where all loss values are NaN values? I ran your code on shapeNet dataset, and the training was fine at first, but after a few thousand steps, loss would all be NaN, with a warning that the input tensor might have NaN or Inf. I wonder if you have any experience in this field？ I implemented shapeNet's dataset class myself, modeled after the other dataset classes in database.py. Among it, the masks are all set to 1, and the depth maps are all set to 0, because I use your Cost Volume model. The parameter I am not sure about is the background. I set the background to white, while your training and testing seem to use black. Will this matter?

liuyuan-pal commented 2 years ago

Hi, I've not encountered such a problem. Maybe, you can insert a line (require pip install ipdb)

if torch.sum(torch.isnan(<suspectible tensor here>))>0:
    import ipdb; ipdb.set_trace()

Then, it will stop when a NaN occurs and you will encounter the debug console model. You may print the variable to find a more exact line that causes the NaN.

liuyuan-pal commented 2 years ago

The background is black in all default training sets (I recommend black), though I think using a white background is OK.

ZYY-666 commented 2 years ago

Thank you very much! Your reply was very helpful to me.