search

charlesq34 / pointnet

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Other
4.64k stars 1.45k forks source link

Getting NAN after few iterations #239

Closed dhirajsuvarna closed 4 years ago

dhirajsuvarna commented 4 years ago

Hi,

I am using PointNet on Pytorch using this github to train my custom dataset.

However, after training around 10-20 batches of examples, the loss goes into NAN.

I highly doubt that it is a exploding gradient issue. (still trying to diagnose the problem)

Posting this problem here in expectation of any help or direction that you may provide.

PS: I have also posted this in the github of fxia11/pointnet.pytorch.

dhirajsuvarna commented 4 years ago

There were NAN's in the input to the network. These NaN's were generated after the Zero Mean and Normalization step for one model only. For time being I have removed that particular model from the dataset and was able to train.

Thank You.