search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.75k stars 7.96k forks source link

Segmentation fault during training #4696

Open titanbender opened 4 years ago

titanbender commented 4 years ago

Hi,

After starting training of tiny yolo 3l and running for a minute, I always receive a segmentation fault. I've checked my dataset and the coordinates for x_center, y_center, width, and height are within 1.0 and 0.0. With a different dataset, I've managed to train the model fine.

Where could the error come from? I'm having difficulties debugging. Below is the print outs before the error occours:

0, .75R: 0.000000, count: 2, loss = 4282.251953, class_loss = 4250.005859, iou_loss = 32.246094
v3 (mse loss, Normalizer: (iou: 0.750000, cls: 1.000000) Region 30 Avg (IOU: -nan, GIOU: -nan), Class: -nan, Obj: -nan, No Obj: 0.487783, .5R: -nan, .75R: -nan, count: 0, loss = 16698.056641, class_loss = 16698.056641, iou_loss = 0.000000
v3 (mse loss, Normalizer: (iou: 0.750000, cls: 1.000000) Region 16 Avg (IOU: -nan, GIOU: -nan), Class: -nan, Obj: -nan, No Obj: 0.528772, .5R: -nan, .75R: -nan, count: 0, loss = 1220.703003, class_loss = 1220.703003, iou_loss = 0.000000
v3 (mse loss, Normalizer: (iou: 0.750000, cls: 1.000000) Region 23 Avg (IOU: 0.245025, GIOU: -0.112839), Class: 0.441727, Obj: 0.508659, No Obj: 0.494860, .5R: 0.000000, .75R: 0.000000, count: 3, loss = 4294.695801, class_loss = 4273.713867, iou_loss = 20.981934
v3 (mse loss, Normalizer: (iou: 0.750000, cls: 1.000000) Region 30 Avg (IOU: -nan, GIOU: -nan), Class: -nan, Obj: -nan, No Obj: 0.486767, .5R: -nan, .75R: -nan, count: 0, loss = 16572.892578, class_loss = 16572.892578, iou_loss = 0.000000
Segmentation fault
AlexeyAB commented 4 years ago

Do you compile with GPU=1 CUDNN=1 OPENCV=1 ?

I addes some fixes, try to download Darknet again.