Training with single class Yolov4-csp

[fede-qopius]

Hello, I am having several trouble on training a model with a single class. I followed some tutorial I found online and I changed the configuration file yolov4-csp changing all the filter before the yolo layer and the number of classes inside the Yolo layer as follow:

[convolutional]
        batch_normalize=1
        size=3
        stride=1
        pad=1
        filters=256
        activation=mish

        [convolutional]
        size=1
        stride=1
        pad=1
        filters=18
        activation=linear

        [yolo]
        mask = 0,1,2
        anchors = 12, 16, 19, 36, 40, 28, 36, 75, 76, 55, 72, 146, 142, 110, 192, 243, 459, 401
        classes= 1
        num=9
        jitter=.3
        ignore_thresh = .7
        truth_thresh = 1
        random=1
        scale_x_y = 1.05
        iou_thresh=0.213
        cls_normalizer=1.0
        iou_normalizer=0.07
        iou_loss=ciou
        nms_kind=greedynms
        beta_nms=0.6

I also changed steps and max batches at the beginning of the config file:

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=64
subdivisions=8
width=512
height=512
channels=3
momentum=0.949
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.00261
burn_in=1000
max_batches = 6000
policy=steps
steps=4800,5400
scales=.1,.1

As I wrote in a previous issue I think I solved a instability issue but still I am having problem with the test part of the training because the YOLO output sometime return NaN. Do you have any clue about that? Thank you

[WongKinYiu]

hello, do you mean output return nan, or loss return nan. i have not meet the first case. for second case, add some eps in bbox_iou function can solve the problem. https://github.com/WongKinYiu/ScaledYOLOv4/issues/145

[fede-qopius]

I meant again the Loss + some part of the output probably because the shape of the images in the validation it is different from the shape of the image during training. I think it is the only difference on the code. Why the validation image shape is not resized as the trained images?

Moreover the model seems to not learning and it continue to behave very strangely.

      0/99     3.81G    0.1159    0.3143         0    0.4302        22       416         0         0 0.0001042 3.125e-05       nan       nan         0
      1/99     3.83G    0.1114    0.1864         0    0.2978        22       416         0         0         0         0       nan       nan         0
      2/99     3.83G    0.1107    0.1741         0    0.2848         8       416         0         0 7.042e-05 7.042e-06       nan       nan         0
      3/99     3.83G    0.1102    0.1808         0     0.291        13       416         0         0 0.0003041  3.89e-05       nan       nan         0
      4/99     3.83G    0.1095    0.1813         0    0.2907        11       416  0.006394   0.09483  0.001297 0.0001768       nan       nan         0
      5/99     3.83G    0.1106    0.1848         0    0.2954         8       416         0         0         0         0       nan       nan         0
      6/99     3.83G    0.1117    0.2155         0    0.3272        21       416         0         0         0         0       nan       nan         0
      7/99     3.83G    0.1109    0.2619         0    0.3728        12       416         0         0         0         0       nan       nan         0
      8/99     3.83G    0.1112    0.2914         0    0.4026        10       416         0         0         0         0       nan       nan         0
      9/99     3.83G    0.1109     0.338         0    0.4489         8       416         0         0  0.009552  0.001581    0.1116    0.3138         0
     10/99     3.83G    0.1111    0.3417         0    0.4528         8       416         0         0   0.01478   0.00203    0.1109    0.3544         0
     11/99     3.83G    0.1108    0.3401         0    0.4509        10       416         0         0   0.01044  0.001636    0.1107    0.3215         0
     12/99     3.83G    0.1106    0.3362         0    0.4469        27       416         0         0  0.008028  0.001208    0.1106    0.2543         0
     13/99     3.83G    0.1107    0.2738         0    0.3845        32       416         0         0   0.01208  0.001849    0.1106    0.2562         0
     14/99     3.83G     0.111    0.3142         0    0.4252        24       416   0.01841    0.4698   0.01761  0.002823    0.1104    0.3078         0
     15/99     3.83G    0.1107    0.3138         0    0.4245         9       416   0.01569    0.4009   0.01724  0.002518    0.1104    0.2791         0
     16/99     3.83G    0.1111    0.2969         0     0.408        15       416         0         0   0.01484  0.002528    0.1103     0.219         0
     17/99     3.83G    0.1106    0.2481         0    0.3587        21       416         0         0   0.02012  0.002935    0.1103    0.2229         0
     18/99     3.83G     0.111    0.2685         0    0.3795        19       416         0         0  0.008988   0.00155    0.1104    0.2591         0
     19/99     3.83G    0.1107    0.3468         0    0.4575        10       416   0.01097    0.3147  0.007157   0.00117    0.1106    0.3206         0
     20/99     3.83G    0.1111    0.4279         0     0.539         6       416   0.02017    0.4914   0.01344  0.002081     0.111    0.3353         0
     21/99     3.83G    0.1113    0.4294         0    0.5407         8       416   0.01628    0.4181   0.01465  0.002042    0.1108    0.3208         0
     22/99     3.83G    0.1099    0.4421         0    0.5521         7       416         0         0  0.003778 0.0005248    0.1109     1.364         0
     23/99     3.83G    0.1114    0.4572         0    0.5686        29       416 0.0001778   0.00431  0.002721  0.000472     0.111     4.267         0
     24/99     3.83G    0.1111    0.4503         0    0.5614        21       416         0         0         0         0     0.112     15.18         0
     25/99     3.83G    0.1106    0.4124         0    0.5231        10       416         0         0         0         0     0.112     20.32         0
     26/99     3.83G    0.1096    0.4094         0     0.519         9       416         0         0         0         0    0.1117     22.05         0
     27/99     3.83G    0.1105    0.4268         0    0.5372        15       416         0         0         0         0    0.1123     34.41         0
     28/99     3.83G    0.1104    0.4216         0     0.532        17       416 0.0001042   0.00431  7.12e-07 2.136e-07    0.1111     40.54         0
     29/99     3.83G    0.1109    0.4476         0    0.5585        25       416  0.001354   0.05603 7.529e-05 1.051e-05    0.1107      64.2         0
     30/99     3.83G    0.1108     0.429         0    0.5399        10       416  0.009167    0.3793   0.00392 0.0006237     0.111      73.9         0
     31/99     3.83G    0.1109    0.4293         0    0.5402        12       416   0.01355    0.4655  0.007208  0.001102    0.1108     50.27         0
     32/99     3.83G    0.1095     0.434         0    0.5434         5       416         0         0         0         0    0.1112     24.53         0
     33/99     3.83G    0.1106    0.4032         0    0.5138         5       416         0         0         0         0    0.1112     11.18         0
     34/99     3.83G    0.1112    0.3966         0    0.5078        22       416  0.001667   0.06897 0.0001093 1.511e-05    0.1109     4.109         0
     35/99     3.83G    0.1105    0.3781         0    0.4886        10       416   0.00778     0.319  0.002509 0.0003963    0.1107      1.75         0
     36/99     3.83G     0.111    0.3823         0    0.4933        18       416  0.009536    0.3448  0.003467 0.0005455    0.1106    0.7712         0
     37/99     3.83G    0.1108    0.3709         0    0.4817        16       416         0         0  0.004179 0.0006936    0.1106    0.4985         0
     38/99     3.83G    0.1107    0.3644         0    0.4751        14       416         0         0    0.0109  0.001602    0.1105    0.4055         0
     39/99     3.83G    0.1106    0.3602         0    0.4707        11       416   0.01641    0.4655   0.01232  0.001821    0.1106    0.3605         0
     40/99     3.83G    0.1107    0.3694         0    0.4801         8       416   0.01662    0.4784   0.01434  0.002538    0.1106    0.3567         0
     41/99     3.83G    0.1108    0.3826         0    0.4934         7       416   0.01659    0.4784   0.01458  0.002108    0.1106    0.3657         0
     42/99     3.83G    0.1108    0.4031         0    0.5139        28       416   0.01634    0.4698    0.0145  0.002224    0.1106     0.365         0
     43/99     3.83G    0.1104    0.3761         0    0.4865        11       416   0.01518    0.4353  0.009867  0.001284    0.1107    0.3581         0
     44/99     3.83G    0.1108     0.361         0    0.4718        26       416   0.01111     0.319   0.00578 0.0009425    0.1106    0.3528         0
     45/99     3.83G    0.1107     0.362         0    0.4727        14       416   0.01105    0.3147  0.005935 0.0009475    0.1106    0.3467         0
     46/99     3.83G    0.1112     0.368         0    0.4792        37       416   0.01145    0.3276  0.005867  0.001094    0.1106    0.3484         0
     47/99     3.83G    0.1107     0.367         0    0.4777        10       416   0.01158    0.3362  0.005795 0.0009532    0.1105    0.3463         0
     48/99     3.83G    0.1106    0.3612         0    0.4718         7       416         0         0  0.005276 0.0008596    0.1105    0.3414         0

As you can see validation losses start with Nan values.

Annotations seem correct too.