search

zylo117 / Yet-Another-EfficientDet-Pytorch

The pytorch re-implement of the official efficientdet with SOTA performance in real time and pretrained weights.
GNU Lesser General Public License v3.0
5.2k stars 1.27k forks source link

error training #726

Closed myasser63 closed 2 years ago

myasser63 commented 2 years ago

I am trying to train efficientdet0-do on my own dataset. But I get this nan for VAL. what does it mean

training command:

! python train.py -c 0 -p FFB.yml --head_only True --lr 5e-3 --batch_size 16 --load_weights /content/Yet-Another-EfficientDet-Pytorch/weights/efficientdet-d0.pth --num_epochs 100 --save_interval 100 

Val. Epoch: 0/100. Classification loss: nan. Regression loss: nan. Total loss: nan
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
Step: 15. Epoch: 1/100. Iteration: 8/8. Cls loss: 2.89493. Reg loss: 3.56300. Total loss: 6.45794: 100% 8/8 [01:10<00:00,  8.82s/it]
Val. Epoch: 1/100. Classification loss: nan. Regression loss: nan. Total loss: nan
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.
WARNING:root:NaN or Inf found in input tensor.