search

fanq15 / FewX

FewX is an open-source toolbox on top of Detectron2 for data-limited instance-level recognition tasks.
https://github.com/fanq15/FewX
MIT License
346 stars 48 forks source link

NaN #31

Open Retiina opened 3 years ago

Retiina commented 3 years ago

Dear author, thanks for you great work. Currently I am trying to run your code but always report NaN error, the following is the error traceback, can you have a look? Thanks in advance!

detectron2/detectron2/modeling/proposal_generator/proposal_utils.py", line 103, in find_top_rpn_
proposals
    raise FloatingPointError(
FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.
Retiina commented 3 years ago

When I train with 4 GPU, NaN gone. However, when I train on 8GPU, it always yield NaN