Loss diverged when I trainging yolov4-tiny

beizhengren commented 4 years ago

Hi, @AlexeyAB I train my own dataset basing on yolov4-tiny.conv.29. And chang the anchors. The loss diverged at about 350'000 iterations. My labels have many small objects, I think whether the small objects result in the loss diverged? Some training info as following. Thanks a lot!

1. Training command

 ./darknet detector train my-voc.data my-yolov4-tiny.cfg yolov4-tiny.conv.29 -map -gpus 0 1
 CUDA-version: 9000 (10000), cuDNN: 7.4.1, GPU count: 2  
 OpenCV version: 3.3.1
yolov4-tiny-zerotech
 0 : compute_capability = 610, cudnn_half = 0, GPU: TITAN Xp 
net.optimized_memory = 0 
mini_batch = 32, batch = 64, time_steps = 1, train = 1 
   layer   filters  size/strd(dil)      input                output

2. The last 3 training log:

400198: 23.908436, 25.670296 avg loss, 0.000026 rate, 1.081508 seconds, 25612672 images, 0.032194 hours left
Loaded: 0.000019 seconds
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 30 Avg (IOU: 0.000000, GIOU: -0.176947), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 124, class_loss = 7.437500, iou_loss = 0.000039, total_loss = 7.437539 
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 37 Avg (IOU: 0.000000, GIOU: -0.211407), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 1522, class_loss = 62.625004, iou_loss = nan, total_loss = nan 
 total_bbox = 739629374, rewritten_bbox = 25.490606 % 
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 30 Avg (IOU: 0.000000, GIOU: -0.232000), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 54, class_loss = 3.312500, iou_loss = 0.000017, total_loss = 3.312517 
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 37 Avg (IOU: 0.000000, GIOU: -0.226382), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 1422, class_loss = 61.750000, iou_loss = nan, total_loss = nan 
 total_bbox = 739630850, rewritten_bbox = 25.490576 % 

 400199: 33.783440, 26.481609 avg loss, 0.000026 rate, 1.192034 seconds, 25612736 images, 0.031879 hours left
Loaded: 0.000016 seconds
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 30 Avg (IOU: 0.000000, GIOU: -0.265819), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 192, class_loss = 11.562500, iou_loss = 0.000062, total_loss = 11.562562 
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 37 Avg (IOU: 0.000000, GIOU: -0.161304), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 735, class_loss = 29.562500, iou_loss = nan, total_loss = nan 
 total_bbox = 739631777, rewritten_bbox = 25.490551 % 
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 30 Avg (IOU: 0.000000, GIOU: -0.219837), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 154, class_loss = 8.437500, iou_loss = 0.000061, total_loss = 8.437561 
v3 (iou loss, Normalizer: (iou: 0.07, cls: 1.00) Region 37 Avg (IOU: 0.000000, GIOU: -0.167445), Class: 0.000000, Obj: 0.000000, No Obj: 0.000000, .5R: 0.000000, .75R: 0.000000, count: 882, class_loss = 30.375000, iou_loss = nan, total_loss = nan 
 total_bbox = 739632813, rewritten_bbox = 25.490522 % 

 400200: 19.986563, 25.832104 avg loss, 0.000026 rate, 1.085733 seconds, 25612800 images, 0.031563 hours left

3.The loss chart

4. I use the command ./darknet detector train ... -show_imgs . The result is correct:

5. My cfg file

please click to see cfg file

```sh [net] # Testing #batch=1 #subdivisions=1 # Training batch=64 subdivisions=2 width=608 height=608 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1 learning_rate=0.00261 burn_in=1000 max_batches = 400200 policy=steps steps=350000,380000 scales=.1,.1 [convolutional] batch_normalize=1 filters=32 size=3 stride=2 pad=1 activation=leaky [convolutional] batch_normalize=1 filters=64 size=3 stride=2 pad=1 activation=leaky [convolutional] batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky [route] layers=-1 groups=2 group_id=1 [convolutional] batch_normalize=1 filters=32 size=3 stride=1 pad=1 activation=leaky [convolutional] batch_normalize=1 filters=32 size=3 stride=1 pad=1 activation=leaky [route] layers = -1,-2 [convolutional] batch_normalize=1 filters=64 size=1 stride=1 pad=1 activation=leaky [route] layers = -6,-1 [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky [route] layers=-1 groups=2 group_id=1 [convolutional] batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky [convolutional] batch_normalize=1 filters=64 size=3 stride=1 pad=1 activation=leaky [route] layers = -1,-2 [convolutional] batch_normalize=1 filters=128 size=1 stride=1 pad=1 activation=leaky [route] layers = -6,-1 [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky [route] layers=-1 groups=2 group_id=1 [convolutional] batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky [convolutional] batch_normalize=1 filters=128 size=3 stride=1 pad=1 activation=leaky [route] layers = -1,-2 [convolutional] batch_normalize=1 filters=256 size=1 stride=1 pad=1 activation=leaky [route] layers = -6,-1 [maxpool] size=2 stride=2 [convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky ################################## [convolutional] batch_normalize=1 filters=256 size=1 stride=1 pad=1 activation=leaky [convolutional] batch_normalize=1 filters=512 size=3 stride=1 pad=1 activation=leaky [convolutional] size=1 stride=1 pad=1 filters=36 activation=linear [yolo] mask = 3,4,5 anchors = 3, 5, 6, 9, 13, 19, 29, 31, 31, 66, 119,166 classes=7 num=6 jitter=.3 scale_x_y = 1.05 cls_normalizer=1.0 iou_normalizer=0.07 iou_loss=ciou ignore_thresh = .7 truth_thresh = 1 random=0 resize=1.5 nms_kind=greedynms beta_nms=0.6 [route] layers = -4 [convolutional] batch_normalize=1 filters=128 size=1 stride=1 pad=1 activation=leaky [upsample] stride=2 [route] layers = -1, 23 [convolutional] batch_normalize=1 filters=256 size=3 stride=1 pad=1 activation=leaky [convolutional] size=1 stride=1 pad=1 filters=36 activation=linear [yolo] mask = 0,1,2 anchors = 3, 5, 6, 9, 13, 19, 29, 31, 31, 66, 119,166 classes=7 num=6 jitter=.3 scale_x_y = 1.05 cls_normalizer=1.0 iou_normalizer=0.07 iou_loss=ciou ignore_thresh = .7 truth_thresh = 1 random=0 resize=1.5 nms_kind=greedynms beta_nms=0.6 ``` <\details>

lflel commented 4 years ago

Hi @beizhengren , @AlexeyAB I have a similar problem with the avg loss.

Im not sure if its the same problem.

beizhengren commented 4 years ago

@lflel Hello, I have resolved the problem by using the cfg file as following https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny_3l.cfg

AlexeyAB / darknet

Loss diverged when I trainging yolov4-tiny #6508