search

LeegoChen / PTC-Net

PTC-Net
MIT License
15 stars 0 forks source link

training loss become nan #2

Open ftd0828 opened 1 year ago

ftd0828 commented 1 year ago

Hi, thanks for your amazing work!but I encountered some problems while reproducing the work .The training loss became nan during training process. If there are possible solutions to my problem? Thanks in advance! 37%|███▋ | 148/400 [2:54:15<4:53:23, 69.85s/it]train loss: 0.3699 embedding norm: 23.580 Triplets (all/active): 95.9/23.4 Mean dist (pos/neg): 0.000/0.000

37%|███▋ | 149/400 [2:55:25<4:52:58, 70.03s/it]train loss: nan embedding norm: nan Triplets (all/active): 95.7/17.5 Mean dist (pos/neg): 0.000/0.000

LeegoChen commented 1 year ago

Sorry, I'm little busy these days. Have you solved the problem? or can you post the config.txt and the training code?

ftd0828 commented 1 year ago

Thanks for your replaying!I use the same training code and config files with this project,i am wondering whether it is caused by my cuda version 11.1