search

csuhan / opendet2

Official code of the paper "Expanding Low-Density Latent Regions for Open-Set Object Detection" (CVPR 2022)
https://arxiv.org/abs/2203.14911
100 stars 11 forks source link

floatingPointError: Loss became infinite or NaN at iteration=166! #29

Open Santosh-Kumar-Singh opened 5 months ago

Santosh-Kumar-Singh commented 5 months ago

I keep getting this error every time I try to train the model. Apparently the loss values abruptly become NaN at a random iteration and training stops with this error.

Runtime Error: floatingPointError: Loss became infinite or NaN at iteration=166! loss_dict = {'loss_cls_ce': nan, 'loss_box_reg': nan, 'loss_ins_con': 0.0, 'loss_cls_up': nan}