I want to train the network from scratch instead of a pre-trained model R-50.pkl. Then I annotate the weights in the config file.
However, I got an error.
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, kwargs)
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward
output = self.module(*inputs[0], *kwargs[0])
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(input, kwargs)
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/meta_arch/rcnn.py", line 517, in forward
proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/rpn.py", line 437, in forward
proposals = self.predict_proposals(anchors, pred_objectness_logits, pred_anchor_deltas, images.image_sizes)
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/rpn.py", line 469, in predict_proposals
self.training,
File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/proposal_utils.py", line 92, in find_top_rpn_proposals
"Predicted boxes or scores contain Inf/NaN. Training has diverged."
FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.
could you help to solve this problem or give me some suggestions?
I want to train the network from scratch instead of a pre-trained model R-50.pkl. Then I annotate the weights in the config file.
However, I got an error.
File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, kwargs) File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 886, in forward output = self.module(*inputs[0], *kwargs[0]) File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(input, kwargs) File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/meta_arch/rcnn.py", line 517, in forward proposals, proposal_losses = self.proposal_generator(images, features, gt_instances) File "/home/multiai3/anaconda3/envs/OWOD/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/rpn.py", line 437, in forward proposals = self.predict_proposals(anchors, pred_objectness_logits, pred_anchor_deltas, images.image_sizes) File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/rpn.py", line 469, in predict_proposals self.training, File "/home/multiai3/Jiuqing/OWOD-master/detectron2/modeling/proposal_generator/proposal_utils.py", line 92, in find_top_rpn_proposals "Predicted boxes or scores contain Inf/NaN. Training has diverged." FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.
could you help to solve this problem or give me some suggestions?