I get loss "nan". - Githubissues

jwyang / graph-rcnn.pytorch

[ECCV 2018] Official code for "Graph R-CNN for Scene Graph Generation"

727 stars 158 forks source link

Open jgyy4775 opened 4 years ago

jgyy4775 commented 4 years ago

When training, I got loss "nan".

What's the problem??

August66 commented 4 years ago

Me too, when training Relpn.so has this problem been solved？

jgyy4775 commented 4 years ago

@August66 I changed the gpu Titan X to 1080ti and it was fixed. I still don't know the exact cause. Sorry...

August66 commented 4 years ago

hello,How big is the memory of the new GPU? I used the memory of GPU GTX 1080 8G.

jgyy4775 commented 4 years ago

I used the memory of GPU GTX 1080ti 11G. And I used 2 multi GPU.