Something is going wrong with yolo_v3_tiny_pan3_aa_ae_mixup_scale_giou.cfg

[keko950]

Hi @AlexeyAB, I've been training my dataset for months, testing and trying out different cfgs, but, lately, i trained two models with yolo_v3_tiny_pan3_aa_ae_mixup_scale_giou.cfg and im receiving strange results..

This dataset have more than 90% mAP with a model trained with panet, but with tiny pan3 aa aemixuip cfg something went wrong. Any idea?

This is the content of cfg for 2 classes:

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=64
subdivisions=32
width=832
height=832
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.0
exposure = 1.0
hue=.1
mixup=1
#blur=5

learning_rate=0.001
burn_in=15000
max_batches = 100000

policy=sgdr
sgdr_cycle=1000
sgdr_mult=2
steps=4000,8000,12000,9000
#scales=1, 1, 0.1, 0.1

[convolutional]
batch_normalize=1
filters=16
size=3
stride=1
pad=1
activation=swish

#[convolutional]
#batch_normalize=1
#filters=16
#size=3
#stride=2
#pad=1
#activation=swish

# remove
[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=swish

[maxpool]
size=2
stride=2
antialiasing=1

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=swish

[maxpool]
size=2
stride=2
antialiasing=1

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=swish

[maxpool]
size=2
stride=2
antialiasing=1

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=swish

[maxpool]
size=2
stride=2
antialiasing=1

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=swish

[maxpool]
size=2
stride=1

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=swish

###########

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=swish

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=swish
assisted_excitation=4000

########### to [yolo-3]

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=swish

[upsample]
stride=2

[route]
layers = -1, 8

###########

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=swish

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=swish

########### to [yolo-2]

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=swish

[upsample]
stride=2

[route]
layers = -1, 6

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=swish

########### features of different layers

[route]
layers=2

[maxpool]
size=16
stride=16

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

[route]
layers=4

[maxpool]
size=8
stride=8

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

[route]
layers=4

[maxpool]
size=8
stride=4
stride_x=4
stride_y=8

[convolutional]
batch_normalize=1
filters=64
size=1
stride=2
stride_x=2
stride_y=1
pad=1
activation=swish

[route]
layers=4

[maxpool]
size=8
stride=8
stride_x=8
stride_y=4

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
stride_x=1
stride_y=2
pad=1
activation=swish

[route]
layers=4

[maxpool]
size=8
stride=8

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

[route]
layers=6

[maxpool]
size=4
stride=4

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

[route]
layers=6

[maxpool]
size=4
stride=2
stride_x=2
stride_y=4

[convolutional]
batch_normalize=1
filters=64
size=1
stride=2
stride_x=2
stride_y=1
pad=1
activation=swish

[route]
layers=6

[maxpool]
size=4
stride=4
stride_x=4
stride_y=2

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
stride_x=1
stride_y=2
pad=1
activation=swish

[route]
layers=8

[maxpool]
size=2
stride=2

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

[route]
layers=10

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=swish

[route]
layers=-1, -3, -6, -9, -12, -15, -18, -21, -24, -27

[maxpool]
maxpool_depth=1
out_channels=64
stride=1
size=1

########### [yolo-1]

[upsample]
stride=4

[route]
layers = -1,24

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=swish

[convolutional]
size=1
stride=1
pad=1
filters=35
activation=linear

[yolo]
mask = 0,1,2,3,4
anchors =  7, 13,   5,110,  33, 19,  19, 78,  22,105,  29,110,  52, 83,  38,135,  61,117,  80,102,  86,151, 125,153
classes=2
num=12
jitter=.3
ignore_thresh = .7
truth_thresh = 1
iou_normalizer=0.25
cls_normalizer=1.0
iou_loss=giou
scale_x_y = 1.05
random=0

########### [yolo-2]

[route]
layers = -6

[upsample]
stride=2

[route]
layers = -1,19

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=swish

[convolutional]
size=1
stride=1
pad=1
filters=35
activation=linear

[yolo]
mask = 4,5,6,7,8
anchors =  7, 13,   5,110,  33, 19,  19, 78,  22,105,  29,110,  52, 83,  38,135,  61,117,  80,102,  86,151, 125,153
classes=2
num=12
jitter=.3
ignore_thresh = .7
truth_thresh = 1
iou_normalizer=0.25
cls_normalizer=1.0
iou_loss=giou
scale_x_y = 1.1
random=0

########### [yolo-3]

[route]
layers = -12

[route]
layers = -1,14

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=swish

[convolutional]
size=1
stride=1
pad=1
filters=28
activation=linear

[yolo]
mask = 8,9,10,11
anchors =  7, 13,   5,110,  33, 19,  19, 78,  22,105,  29,110,  52, 83,  38,135,  61,117,  80,102,  86,151, 125,153
classes=2
num=12
jitter=.3
ignore_thresh = .7
truth_thresh = 1
iou_normalizer=0.25
cls_normalizer=1.0
iou_loss=giou
scale_x_y = 1.2
random=0

[AlexeyAB]

You must train more than 19 000 iterations = ( burn_in=15000 + assisted_excitation=4000 )

Do you use the latest repo?

Try to train yolov3-tiny.cfg, what mAP can you get after 4000 - 8000 iterations?

[keko950]

Just tried now to train with a clean panet model, and same result..

Yes, im using the latest repo. Just before trying it, im going to test another dataset, just in case..

[AlexeyAB]

This is normal, you will get very low mAP before 19 000 iterations = ( burn_in=15000 + assisted_excitation=4000 )

[keko950]

Sorry, i fotgot to tell you that this last one is a panet model, not even tiny..

[AlexeyAB]

What is the panet? Can you attach cfg-file?

[keko950]

It is yolo_v3_spp_pan_scale.cfg.txt as referenced in: https://github.com/AlexeyAB/darknet/issues/3114

Sure:

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=64
subdivisions=24
width=512
height=512
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 0.5
exposure = 1.0
hue=.1

learning_rate=0.0001
burn_in=1000
max_batches = 100000

#policy=steps
#steps=80000,90000
#scales=.1,.1

policy=sgdr
sgdr_cycle=1000
sgdr_mult=2
steps=4000,6000,8000,9000
#scales=1, 1, 0.1, 0.1

[convolutional]
batch_normalize=1
filters=32
size=3
stride=1
pad=1
activation=leaky

# Downsample

[convolutional]
batch_normalize=1
filters=64
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=32
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=128
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=64
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=256
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=512
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

# Downsample

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=2
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
filters=1024
size=3
stride=1
pad=1
activation=leaky

[shortcut]
from=-3
activation=linear

######################

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

### SPP ###
[maxpool]
stride=1
size=5

[route]
layers=-2

[maxpool]
stride=1
size=9

[route]
layers=-4

[maxpool]
stride=1
size=13

[route]
layers=-1,-3,-5,-6

### End SPP ###

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

########### to [yolo-3]

[route]
layers = -4

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 61

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=512
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=512
activation=leaky

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

########### to [yolo-2]

[route]
layers = -4

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1, 36

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=256
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=256
activation=leaky

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

########### to [yolo-1]

########### features of different layers

[route]
layers=1

[reorg3d]
stride=2

[route]
layers=5,-1

[reorg3d]
stride=2

[route]
layers=12,-1

[reorg3d]
stride=2

[route]
layers=37,-1

[reorg3d]
stride=2

[route]
layers=62,-1

########### [yolo-1]

[convolutional]
batch_normalize=1
filters=128
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=4

[route]
layers = -1,-12

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=256
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=28
activation=linear

[yolo]
mask = 0,1,2,3
anchors = 8,8, 10,13, 16,30, 33,23,  32,32, 30,61, 62,45, 64,64,  59,119, 116,90, 156,198, 373,326
classes=2
num=12
jitter=.3
ignore_thresh = .7
truth_thresh = 1
scale_x_y = 1.05
random=0

########### [yolo-2]

[route]
layers = -7

[convolutional]
batch_normalize=1
filters=256
size=1
stride=1
pad=1
activation=leaky

[upsample]
stride=2

[route]
layers = -1,-28

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=512
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=28
activation=linear

[yolo]
mask = 4,5,6,7
anchors = 8,8, 10,13, 16,30, 33,23,  32,32, 30,61, 62,45, 64,64,  59,119, 116,90, 156,198, 373,326
classes=2
num=12
jitter=.3
ignore_thresh = .7
truth_thresh = 1
scale_x_y = 1.1
random=0

########### [yolo-3]

[route]
layers = -14

[convolutional]
batch_normalize=1
filters=512
size=1
stride=1
pad=1
activation=leaky

[route]
layers = -1,-43

[convolutional]
batch_normalize=1
size=3
stride=1
pad=1
filters=1024
activation=leaky

[convolutional]
size=1
stride=1
pad=1
filters=28
activation=linear

[yolo]
mask = 8,9,10,11
anchors = 8,8, 10,13, 16,30, 33,23,  32,32, 30,61, 62,45, 59,119,   80,80, 116,90, 156,198, 373,326
classes=2
num=12
jitter=.3
ignore_thresh = .7
truth_thresh = 1
scale_x_y = 1.2
random=0

[AlexeyAB]

• How many images in your training dataset?
• Check the mAP on Training dataset.

[keko950]

Nevermind, you were right, dataset issue. Didn't check the train.txt correctly, it was incomplete. Running now it again, but i think that was the problem (Only 3000 images were been used for training)

[isgursoy]

@keko950 progress?