search

Layout-Parser / layout-model-training

The scripts for training Detectron2-based Layout Models on popular layout analysis datasets
195 stars 53 forks source link

Predicted boxes or scores contain Inf/NaN. Training has diverged. #23

Open MachineLearning4Work opened 6 months ago

MachineLearning4Work commented 6 months ago

I encountered an issue while running the training code using train_net.py. The error suggests a FloatingPointError in the file "proposal_utils.py" at line 103, indicating that predicted boxes or scores contain Inf/NaN, leading to training divergence.

File "/usr/local/lib/python3.10/dist-packages/detectron2/modeling/proposal_generator/proposal_utils.py", line 103, in find_top_rpn_proposals

raise FloatingPointError(

FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.

How can I resolve this issue?

xrimpy commented 1 month ago

Same error here