search

shinya7y / UniverseNet

USB: Universal-Scale Object Detection Benchmark (BMVC 2022)
Apache License 2.0
422 stars 54 forks source link

loss nan error when set filter_empty_gt=False #14

Closed zhengye1995 closed 3 years ago

zhengye1995 commented 3 years ago

Hello, thanks for your outstanding work!

I want to train universeNet with the images which do not contain objects, so I set filter_empty_gt=False, but the loss is nan.

I have reduced the learning rate and tried to increase the warm-up step, but they are not work.

I have also tried to train my data with other models like cascade rcnn or faster rcnn with the same setting, the losses for them are both normal.

So, how can i fix this issue?

shinya7y commented 3 years ago

A possible solution is applying this fix https://github.com/open-mmlab/mmdetection/pull/3702 to gfl_head.py.

if len(pos_inds) == 0: weight_targets become 0, avg_factor may become 0, resulting in nan.

whut2962575697 commented 3 years ago

Hello，do you fix the issue? https://github.com/shinya7y/UniverseNet/issues/13

shinya7y commented 3 years ago

I updated the master branch with the abovementioned fix. If the issue persists, please let me know more details.

zhengye1995 commented 3 years ago

I updated the master branch with the abovementioned fix. If the issue persists, please let me know more details.

Thanks for your replay, it works. But I still need to reduced the learning rate and tried to increase the warm-up step, which may results in the performance is not well.

shinya7y commented 3 years ago

I close this issue. If there are performance issues (e.g., lower AP than the baselines ATSS and GFL), please open another issue.