Infinity losses during training

jd730 / EOPSN

[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

Other

52 stars 7 forks source link

Infinity losses during training #6

Open Hoff97 opened 2 years ago

Hoff97 commented 2 years ago

When running the training, I noticed that sometimes some losses (especially the box regression loss) can become Infinity. This seems to happen especially once many exemplars have been mined. I was wondering if you encountered this issue as well, and if you know of any fixes.

jd730 commented 2 years ago

Hi, I've never encountered that issue. what if just limiting the maximum number of exemplars at each time although box although I do not think that that is the primary reason for the Inf error?