search

facebookresearch / adaptive_teacher

This repo provides the source code for "Cross-Domain Adaptive Teacher for Object Detection".
Other
180 stars 35 forks source link

FloatingPointError: Predicted boxes or scores contain Inf/NaN #57

Closed anranbixin closed 10 months ago

anranbixin commented 1 year ago

Hello, could you please share the configuration file you ran out? I also had the following problems during the experiment (the problem occurred in the first iteration) : FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged. After trying through the problem set, changing the weights and updating the virtual machine environment, the problem was still not resolved. What is your operating environment? I tried single gpu and multi-GPU (4). My experimental environment is python3.8,torch=1.9,cuda=11.6,detectron2=0.5. (#9 #12 )I tried the method in the two questions above the question,but still didn't solve the problem.My attempt was at voc2012 to clipart.

huamiao1012 commented 1 year ago

I have the same question, and if the problem appears in the initial of experiment, may be you can try to install detectron2=0.3.

ouyang11111 commented 11 months ago

my problem appears in the initial but my CUDA version are not support low version of detectron2 (<0.6 not support) .how to fix ?

anranbixin commented 11 months ago

I am renting a server, so I did not solve this problem, but detectron=0.3 is able to configure the corresponding operating environment. My environment is cude10.2, Python 3.8, and torch 1.7.

ouyang11111 commented 11 months ago

now i will seek for a rent too. for it has already waste me a few days .

anranbixin commented 11 months ago

You can use https://www.gpushare.com/ The above server, detectron=0.3, needs to be downloaded and installed by oneself. Other instructions can be used:

  1. pip install torch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0
  2. pip install torch==1.7.0 torchvision==0.8.1 torchaudio==0.7.0 (Because there may be an issue with the torch and torch vision versions here)

The machine can be either a TITAN X-12G 1 card or a 2080 Ti-11G 1 card

ouyang11111 commented 11 months ago

thanks i will try

active-svg commented 10 months ago

请问解决了吗，这个问题，Predicted boxes or scores contain Inf/NaN. Training has diverged.，是如何解决的

ouyang11111 commented 10 months ago

我是租服务器解决的

一个老外给我的回复方案：

You can use https://www.gpushare.com/&nbsp;The above server, detectron=0.3, needs to be downloaded and installed by oneself. Other instructions can be used:

pip install torch==1.7.0 torchvision==0.8.0 torchaudio==0.7.0

pip install torch==1.7.0 torchvision==0.8.1 torchaudio==0.7.0 (Because there may be an issue with the torch and torch vision versions here)

The machine can be either a TITAN X-12G 1 card or a 2080 Ti-11G 1 card

jianfeiou @.***

 

------------------ 原始邮件 ------------------ 发件人: "facebookresearch/adaptive_teacher" @.>; 发送时间: 2023年10月19日(星期四) 下午4:27 @.>; @.**@.>; 主题: Re: [facebookresearch/adaptive_teacher] FloatingPointError: Predicted boxes or scores contain Inf/NaN (Issue #57)

请问解决了吗，这个问题，Predicted boxes or scores contain Inf/NaN. Training has diverged.，是如何解决的

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

active-svg commented 10 months ago

但是我装完detectron3之后又出现TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType错误，该怎么解决嘞