slight decrease in mAP with random = 1 flag

[Rahul-Venugopal]

Hi ,

I have trained yolov3 to detect three custom objects with following number of images

person (~1000 training images and ~ 150 validation images) forklift(~300 training images and ~ 100 validation images) cartons(~300 training images and ~ 100 validation images)

I have trained for 20500 iterations

First I used flag RANDOM = 0 and mAP on validation set is

calculation mAP (mean average precision)...
296
 detections_count = 450, unique_truth_count = 388  
class_id = 0, name = person,     ap = 78.76 % 
class_id = 1, name = forklift,   ap = 84.49 % 
class_id = 2, name = box,    ap = 70.08 % 
 for thresh = 0.25, precision = 0.89, recall = 0.75, F1-score = 0.81 
 for thresh = 0.25, TP = 291, FP = 36, FN = 97, average IoU = 69.11 % 

 mean average precision (mAP) = 0.777743, or 77.77 % 

Also the avg loss is ~0.0657

Then I used flag RANDOM = 1 and in cfg the parameters are

batch=64 subdivisions=16 width=416 height=416 channels=3 momentum=0.9 decay=0.0005 angle=0 saturation = 1.5 exposure = 1.5 hue=.1

learning_rate=0.001 burn_in=1000 max_batches = 20500
policy=steps steps=17000,17500
scales=.1,.1
...

jitter=.3 ignore_thresh = .7 truth_thresh = 1 random=1

When I calculate mAP for this weights with RANDOM = 1 , I get mAP

Total BFLOPS 65.304 Loading weights from 00_training_results/r1_map_valid.txt/weights/yolov3-obj_r1_final.weights... seen 64 Done!

calculation mAP (mean average precision)... 296 detections_count = 470, unique_truth_count = 388
class_id = 0, name = person, ap = 77.89 % class_id = 1, name = forklift, ap = 84.45 % class_id = 2, name = box, ap = 69.01 % for thresh = 0.25, precision = 0.89, recall = 0.75, F1-score = 0.81 for thresh = 0.25, TP = 290, FP = 35, FN = 98, average IoU = 69.58 %

mean average precision (mAP) = 0.771177, or 77.12 % Total Detection Time: 31.000000 Seconds

and avg loss is ~0.08

Is it expected the mAP and avg loss to improve with RANDOM = 1 flag ? What could be the reason for lower mAP and higher avg loss with random = 1 flag ? Should I continue for more training ? I am not seeing much variation in avg loss for last few 500 iterations and does it improve if I I train for more iterations ?

My chart after 19700 iterations with flag Random = 1 is attached below

[AlexeyAB]

@Rahul-Venugopal Hi,

Is it expected the mAP and avg loss to improve with RANDOM = 1 flag ? What could be the reason for lower mAP and higher avg loss with random = 1 flag ?

There are at least 2 reasons:

1. small validation dataset, so mAP value isn't accurate
2. there is no veriety of sizes of objects in your validate dataset, so you don't need random=1

I am not seeing much variation in avg loss for last few 500 iterations and does it improve if I I train for more iterations ?

Usually you should train 2x-4x more iterations with random=1 than with random=0.

It shouldn't always increase mAP. random=1 can decrease mAP in some cases.

random=1 is necessary for training the weights-file that can be used for detection with different network resolutions, just like we can do with the default yolov3-file: https://pjreddie.com/media/files/yolov3.weights As you can see there is the same URL to the same weights file for YOLOv3-320, YOLOv3-416, YOLOv3-608: https://pjreddie.com/darknet/yolo/ Use the same yolov3.weights just you change width= height= in yolov3.cfg

[Rahul-Venugopal]

Thanks for replying @AlexeyAB

1.small validation dataset, so mAP value isn't accurate

  What is the recommended number of validation set images(per class) , if I have around 600 training 
  images per class ?

Thanks for all information . I will try to change the modifications as you suggested