search

ucbdrive / few-shot-object-detection

Implementations of few-shot object detection benchmarks
Apache License 2.0
1.08k stars 225 forks source link

FloatingPointError #104

Closed jinweiLiu closed 3 years ago

jinweiLiu commented 3 years ago

Hello! When I start training, the following issue occurred “FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.” What should I do?

diaosilei commented 3 years ago

Hello! When I start training, the following issue occurred “FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.” What should I do?

Hello! Have you solved your problem?

jinweiLiu commented 3 years ago

decrease the learning rate may help

diaosilei commented 3 years ago

decrease the learning rate may help

I've tried this method, but it didn't solve my problem. Although, thank you for your reply.

xinw1012 commented 3 years ago

Hi,

sorry for replying to the thread late. Could you let me know what your setup is? Like your Pytorch, detectron2, and cudnn version? The code was tested in Pytorch 1.6 and Detectron2 v0.2.1. Maybe you can check whether the problem can be resolved if an older version is used?

Also, if you have any additional information to share to resolve the issue, please let us know. We will look into the issue on our end as well.

Thanks, Xin

On Tue, Apr 13, 2021 at 8:36 PM jgsimard @.***> wrote:

@diaosilei https://github.com/diaosilei , @jinweiLiu https://github.com/jinweiLiu did you solve the problem ? I get the the same thing! The training starts but it diverges, even if I decrease the learning rate

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ucbdrive/few-shot-object-detection/issues/104#issuecomment-819204391, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVXTBADMJNWSOSJOUIL223TIUESVANCNFSM42QKUQRA .

jinweiLiu commented 3 years ago

you could try to set the learning rate lower发自我的荣耀手机-------- 原始邮件 --------发件人： jgsimard @.>日期： 2021年4月14日周三 11:36收件人： ucbdrive/few-shot-object-detection @.>抄送： jinweiLiu @.>, Mention @.>主 题： Re: [ucbdrive/few-shot-object-detection] FloatingPointError (#104) @diaosilei , @jinweiLiu did you solve the problem ? I get the the same thing! The training starts but it diverges, even if I decrease the learning rate

—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe.

jgsimard commented 3 years ago

Hi, I think it it a mismatch with detectron v.04. It works well with detectron v0.3 !

xinw1012 commented 3 years ago

Great news! @thomasehuang and I would also keep an eye on updating the codebase to the latest detectron2. It's a bit unfortunate that sometimes these library versions are not completely backward compatible.