model.eval causing nan values

[hpc100]

Thanks for sharing your work ! @Yukichiii @yuxiaoguo I tried to test your code on Semantic3D : In validation step, i get "nan" value in output.

• I checked points cloud data, and there is no "nan" in npy files
• I used this : print('Nan value .... ???? ', [k for k, v in model.named_parameters() if any(torch.isnan(v.ravel()))]). There is no nan values in train (model.train()) and validation (model.eval())
  • torch.where( torch.isnan(coord) == True), torch.where( torch.isnan(feat) == True), torch.where( torch.isnan(batch) == True) return empty list
  • So : neither nan value in data nor weigths/biais -> however output filled with nan

Do you have any idea where the problem could come from (layer norm, ....) ?

[jaswanthbjk]

Found any solution?

[hpc100]

No. @jaswanthbjk do you have the same problems ?

[jaswanthbjk]

Yes, I had the same problem.

But not anymore when I included normals in the features.

[hpc100]

@jaswanthbj So, you didn't get nan values with XYZ+RGB+Normals or it's for XYZ + Normals ? I tried both, and get nan values when model is set to eval mode. Which points clouds do you use for validation ? (me : domfountain_station1_xyz_intensity and untermaederbrunnen_station3_xyz_intensity) Have you tried intensity features ?

[jaswanthbjk]

@hpc100

Sorry for the confused reply,

I am still gettinig nans in eval mode. But not during training with RGB + Normals + XYZ, which is super weird for me.

[hpc100]

Found any solution ? @jaswanthbj Have you tried to evaluate the model on cpu // other GPU ?

[jaswanthbjk]

No, However, I run, It's resulting in nan values.

The dataloader is very different between train and val. Maybe digging around that might help solve the issue.

[Yukichiii]

The Nan value may be caused by half-precision. Could you please try to forward the model with full-precision? You can set fp16_mode=0 and use_amp=False.