Loss become nan when at about 300 epoch

zeliu98 / Group-Free-3D

Group-Free 3D Object Detection via Transformers

MIT License

243 stars 33 forks source link

Loss become nan when at about 300 epoch #8

Closed adrien-Chen closed 3 years ago

adrien-Chen commented 3 years ago

Thanks for your excellent work!

I encountered a problem during the training. Since I only have one GPU, so I modified the train_dist.py to a single GPU version (just remove the codes about distribution). Screenshot from 2021-05-11 09-31-19

I want to know if is there anything else needs to be modified? And if you have any suggestion about this problem? Thanks very much!!