mAP changes with --batch-size

[imyhxy]

Hi, Dude. I have noticed that the AP changes a little with different --batch-size, it this reasonable?

Notice that this isn't related to the CUDA randomness. And the following two evaluation have a 0.3 AP gap on the coco128 dataset. When evaluate on the full coco val set, the gap is closer but still there.

Test 1:

val: data=data/coco128.yaml, weights=['weights/yolov5s.pt'], batch_size=16, imgsz=640, conf_thres=0.001, iou_thres=0.6, task=val, device=, single_cls=False, augment=False, verbose=True, save_txt=False, save_hybrid=False, save_conf=False, save_json=False, project=runs/val, name=coco, exist_ok=False, half=False
YOLOv5 🚀 v6.0-4-gb754525 torch 1.9.0 CUDA:0 (NVIDIA GeForce RTX 2060, 5934.5625MB)

Fusing layers... 
Model Summary: 213 layers, 7225885 parameters, 0 gradients
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████| 128/128 [00:00<?, ?it/s]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|██████████| 8/8 [00:01<00:00,  4.11it/s]
                 all        128        929      0.643      0.572      0.621      0.406
              person        128        254      0.727      0.724      0.773      0.503
             bicycle        128          6      0.619        0.5      0.544      0.339
                 car        128         46      0.677      0.348      0.465      0.206
          motorcycle        128          5      0.733        0.6      0.812      0.611
            airplane        128          6          1      0.909      0.995      0.705
                 bus        128          7      0.606      0.714      0.729      0.622
               train        128          3      0.714          1      0.995      0.615
               truck        128         12      0.542      0.333      0.441      0.261
                boat        128          6      0.484      0.333      0.444      0.172
       traffic light        128         14      0.726      0.194      0.255      0.171
           stop sign        128          2      0.588        0.5      0.828      0.663
               bench        128          9      0.735      0.444      0.528      0.228
                bird        128         16      0.752          1      0.969      0.617
                 cat        128          4      0.866       0.75      0.836        0.7
                 dog        128          9      0.827      0.667      0.839      0.537
               horse        128          2      0.561          1      0.995      0.672
            elephant        128         17      0.945      0.882       0.94      0.662
                bear        128          1       0.56          1      0.995      0.995
               zebra        128          4       0.84          1      0.995      0.908
             giraffe        128          9      0.773      0.889      0.866      0.599
            backpack        128          6          1       0.33      0.461      0.183
            umbrella        128         18      0.711      0.556      0.695      0.374
             handbag        128         19      0.535      0.105       0.16      0.112
                 tie        128          7      0.802      0.571      0.662      0.413
            suitcase        128          4          1      0.738      0.995      0.509
             frisbee        128          5      0.573        0.8      0.798      0.689
                skis        128          1      0.534          1      0.995      0.398
           snowboard        128          7       0.89      0.714      0.768      0.555
         sports ball        128          6      0.578        0.5      0.584       0.32
                kite        128         10      0.656      0.575      0.545      0.184
        baseball bat        128          4      0.454        0.5      0.307     0.0892
      baseball glove        128          7       0.35      0.429      0.355      0.194
          skateboard        128          5      0.522        0.4       0.69       0.49
       tennis racket        128          7      0.474      0.571      0.551      0.313
              bottle        128         18      0.599      0.498      0.479       0.28
          wine glass        128         16      0.532      0.688      0.649      0.396
                 cup        128         36      0.797      0.361      0.489      0.321
                fork        128          6      0.488      0.321      0.303       0.21
               knife        128         16      0.587      0.625      0.685      0.461
               spoon        128         22      0.568      0.409      0.485       0.25
                bowl        128         28      0.675      0.571      0.606      0.428
              banana        128          1      0.142          1      0.142     0.0569
            sandwich        128          2          0          0      0.147      0.118
              orange        128          4          1          0      0.523      0.208
            broccoli        128         11      0.286      0.221      0.246      0.211
              carrot        128         24      0.512        0.5      0.572      0.391
             hot dog        128          2      0.481      0.943      0.497      0.465
               pizza        128          5      0.599          1      0.824      0.571
               donut        128         14       0.69          1      0.963      0.862
                cake        128          4      0.698          1      0.995      0.678
               chair        128         35      0.433      0.525      0.475      0.228
               couch        128          6      0.681      0.333      0.788      0.461
        potted plant        128         14      0.782      0.786      0.779      0.435
                 bed        128          3          1          0      0.696      0.234
        dining table        128         13      0.585      0.462       0.43      0.224
              toilet        128          2      0.407        0.5      0.532      0.476
                  tv        128          2      0.628          1      0.995      0.796
              laptop        128          3          1          0      0.288       0.15
               mouse        128          2          1          0     0.0289     0.0231
              remote        128          8      0.632      0.625      0.639      0.489
          cell phone        128          8      0.721      0.331       0.41      0.225
           microwave        128          3      0.332          1      0.995       0.83
                oven        128          5      0.279        0.4      0.422      0.248
                sink        128          6      0.358      0.167      0.279      0.171
        refrigerator        128          5      0.702        0.8      0.816      0.505
                book        128         29      0.589      0.198      0.292      0.124
               clock        128          9       0.78      0.778      0.905      0.593
                vase        128          2      0.177          1      0.663      0.597
            scissors        128          1          1          0     0.0144    0.00288
          teddy bear        128         21      0.825      0.449      0.586      0.353
          toothbrush        128          5      0.714      0.514      0.678      0.144

Test 2:

val: data=data/coco128.yaml, weights=['weights/yolov5s.pt'], batch_size=64, imgsz=640, conf_thres=0.001, iou_thres=0.6, task=val, device=, single_cls=False, augment=False, verbose=True, save_txt=False, save_hybrid=False, save_conf=False, save_json=False, project=runs/val, name=coco, exist_ok=False, half=False
YOLOv5 🚀 v6.0-4-gb754525 torch 1.9.0 CUDA:0 (NVIDIA GeForce RTX 2060, 5934.5625MB)

Fusing layers... 
Model Summary: 213 layers, 7225885 parameters, 0 gradients
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████| 128/128 [00:00<?, ?it/s]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100%|██████████| 2/2 [00:03<00:00,  1.55s/it]
                 all        128        929      0.674       0.53       0.62      0.409
              person        128        254       0.78      0.677      0.773      0.502
             bicycle        128          6      0.733        0.5      0.563      0.349
                 car        128         46      0.795      0.348      0.474      0.208
          motorcycle        128          5      0.735        0.6      0.762      0.558
            airplane        128          6          1      0.838      0.995      0.722
                 bus        128          7       0.64      0.714      0.718       0.61
               train        128          3      0.761          1      0.995      0.615
               truck        128         12      0.605       0.25      0.463      0.253
                boat        128          6       0.59      0.333      0.426      0.157
       traffic light        128         14      0.567      0.143      0.242      0.156
           stop sign        128          2      0.624        0.5      0.828      0.663
               bench        128          9      0.761      0.333      0.527      0.228
                bird        128         16      0.839      0.875      0.966      0.615
                 cat        128          4      0.918       0.75      0.845      0.704
                 dog        128          9       0.84      0.586      0.847      0.561
               horse        128          2      0.621          1      0.995      0.672
            elephant        128         17      0.954      0.882       0.94      0.667
                bear        128          1      0.595          1      0.995      0.995
               zebra        128          4      0.853          1      0.995      0.952
             giraffe        128          9      0.779      0.778       0.86      0.573
            backpack        128          6      0.556      0.167      0.399      0.171
            umbrella        128         18      0.746      0.556      0.687       0.37
             handbag        128         19      0.605      0.105      0.152      0.108
                 tie        128          7      0.994      0.571      0.663      0.415
            suitcase        128          4      0.759        0.5      0.895      0.499
             frisbee        128          5      0.634        0.8      0.798      0.664
                skis        128          1      0.599          1      0.995      0.398
           snowboard        128          7      0.943      0.714      0.769      0.555
         sports ball        128          6      0.637        0.5      0.583      0.319
                kite        128         10      0.634      0.521      0.541      0.183
        baseball bat        128          4      0.399        0.5       0.28      0.121
      baseball glove        128          7      0.305      0.317      0.356      0.195
          skateboard        128          5       0.71      0.504      0.738      0.548
       tennis racket        128          7      0.548      0.571      0.551      0.308
              bottle        128         18      0.657      0.426      0.472      0.285
          wine glass        128         16      0.735      0.875      0.834      0.405
                 cup        128         36       0.84      0.361       0.49      0.293
                fork        128          6      0.331      0.167      0.246       0.19
               knife        128         16      0.842      0.625      0.679      0.447
               spoon        128         22      0.782      0.409      0.521      0.268
                bowl        128         28       0.74      0.571      0.625       0.46
              banana        128          1          0          0      0.142     0.0995
            sandwich        128          2          0          0      0.106     0.0804
              orange        128          4          1          0       0.55       0.22
            broccoli        128         11      0.319      0.182      0.332      0.287
              carrot        128         24      0.672      0.458      0.597      0.337
             hot dog        128          2      0.352      0.555      0.497      0.465
               pizza        128          5      0.619          1      0.824      0.569
               donut        128         14        0.7          1      0.963      0.863
                cake        128          4       0.71          1      0.945      0.757
               chair        128         35       0.51      0.535      0.493      0.233
               couch        128          6      0.623      0.333       0.74      0.407
        potted plant        128         14      0.782      0.714       0.78      0.428
                 bed        128          3          1          0      0.665      0.185
        dining table        128         13      0.768      0.462       0.47      0.281
              toilet        128          2      0.454        0.5      0.531      0.475
                  tv        128          2      0.716          1      0.995      0.796
              laptop        128          3          1          0      0.256      0.131
               mouse        128          2          1          0     0.0277     0.0222
              remote        128          8      0.746      0.625      0.639       0.49
          cell phone        128          8      0.618      0.214       0.41      0.225
           microwave        128          3      0.373          1      0.995       0.83
                oven        128          5      0.352        0.4      0.416      0.243
                sink        128          6      0.336      0.167      0.259      0.166
        refrigerator        128          5      0.678        0.8      0.814      0.439
                book        128         29      0.626      0.138      0.289      0.123
               clock        128          9      0.813      0.778      0.901      0.596
                vase        128          2      0.231          1      0.663      0.597
            scissors        128          1          1          0     0.0158    0.00316
          teddy bear        128         21      0.884      0.366       0.59      0.362
          toothbrush        128          5          1      0.499      0.653      0.396

[glenn-jocher]

@imyhxy rectangular batch shapes are computed as a function of the --batch-size argument for minimum padding, so yes as your --batch-size varies then padding will vary, causing results to vary.