search

hehefan / PointRNN-PyTorch

PointRNN, PointGRU and PointLSTM.
MIT License
56 stars 6 forks source link

NaN values for earth movers distance #1

Closed Jagyan closed 3 years ago

Jagyan commented 3 years ago

Hi @hehefan, I am trying to implement this complete architecture of PointRNN using Pytorch and the modules you have provided on the radar points of the nuscenes dataset. There are a few issues. Firstly, my loss values are going very high as I am using Chamfer's distance and Earth Movers distance for my loss as stated in the paper. The loss values are going to more than 100000. I do not know whether this is normal. Then at certain point, the states being returned from the RNN cells are giving very high values in the range of 10 raised to 36. Then, the loss values returned turn out to be Nan values. Any help from you would be highly appreciated as I am working under a deadline.

Thank you.

hehefan commented 3 years ago

Hi @Jagyan,

I have not used the PyTorch CD and EMD losses. So please first kindly check the losses to make sure they are correct. Second, please try to divide the loss by the number of points. Finally, using gradient clip may help you. Thank you.

Best regards.

Jagyan commented 3 years ago

Hi @hehefan, This is an urgent request as I have a deadline to complete a project where I am using your network. I am running the model on nuscenes data and the loss is fluctuating within a certain range and not going down. I have already implemented the range of [-5m, 5m]. Is there any link to a prepared dataset? What else are you using for data preparation?

Regards. Training loss