search

meituan / YOLOv6

YOLOv6: a single-stage object detection framework dedicated to industrial applications.
GNU General Public License v3.0
5.7k stars 1.03k forks source link

Why is the IOU_LOSS as nan? #319

Closed joel5638 closed 2 years ago

joel5638 commented 2 years ago

Im training with 69 images and 10 images for testing. Im trainng on my local pc.

im using python3 tools/train.py --batch 1 --conf configs/yolov6s.py --data data/dataset.yaml --device 0

Im getting iou_loss as nan as shown in the picture im sharing. Screenshot from 2022-07-13 13-17-41

qinhao14 commented 2 years ago

Hi, the possible reason is that the gt box of some samples is too large and the value overflows in the calculation of ciou loss. You can try to solve this problem by adding “loss = torch.nan_to_num(loss)” to the “IOUloss” function.

joel5638 commented 2 years ago

@qinhao14 where do i add that? right in the function?

qinhao14 commented 2 years ago

@joel5638 YOLOv6/yolov6/utils/figure_iou.py class IOUloss: image

joel5638 commented 2 years ago

while training. i dont see the 69 images being loaded. I see this. Is this normal. Screenshot from 2022-07-13 15-38-10

mtjhl commented 2 years ago

Yes, in you screenshot, it shows Final number of valid images: 69 / labels:69.