FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.

facebookresearch / detectron2

Detectron2 is a platform for object detection, segmentation and other visual recognition tasks.

https://detectron2.readthedocs.io/en/latest/

Apache License 2.0

30.01k stars 7.41k forks source link

FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged. #5080

Open ouyang11111 opened 1 year ago

ouyang11111 commented 1 year ago

Instructions To Reproduce the 🐛 Bug:

when I use VOC2012 dataset for my project shows the bug : "FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged." this means the loss is extremely big, how can I solve it?

Environment:

CUDA 12 RTX3090 detectorn2:V6.0 torch : 2.1.0.dev20230708+cu121

Hemanth21k commented 1 year ago

I had the same issue as well, but i got it while training voc 2007 on detectron2 from scratch. I think its fine because when you train model like detectron2 without pretrained weights its hard to converge.