Why mAP doesnt increse after 17300 iterations but average loss still descrese ?

[astronaut71]

Hi

Im using the yolov3-spp.cfg model withbatch=64 subdivisions=64 width=832 height=832 random=1 and anchors calculated for 832x83. Im trying to detect workers on the construction field. My data set is the following

• 1 class
• 923 images for training and 116 images for validation
• Images resolution are different as were taken from different cameras such as: 2112 x 1584, 1920x1080, 3264x1836
• So workers are dynamic objects that can change the locations/positions (as they can stand/seat and work or can move)
• Training images are not same , changes in objects dimensions depending from which distance were taken.
• Detection images are almost the same as training images.

So Im using this obj.data file

classes = 1
train  = /home/admini/darknet/darknet/data/train.txt
valid  = /home/admini/darknet/darknet/data/test.txt
names = /home/admini/darknet/darknet/data/workers.names
backup = /home/admini/darknet/darknet/backup

So the mAP will be calculated on the test.txt data which are 116 images. But after 17300 iterations the mAP is not incresing more then 55% but average loss is still decreasing as can see from the image

So this I got when last time mAP was calulated.

Label file name is too short:
Can't open label file. (This can be normal only if you use MSCOCO):
Label file name is too short:
Can't open label file. (This can be normal only if you use MSCOCO):

detections_count = 760, unique_truth_count = 862
class_id = 0, name = person, ap = 49.32% (TP = 344, FP = 82)

for thresh = 0.25, precision = 0.81, recall = 0.40, F1-score = 0.53 for thresh = 0.25, TP = 344, FP = 82, FN = 518, average IoU = 57.21 %

IoU threshold = 50 %, used Area-Under-Curve for each unique Recall mean average precision (mAP@0.50) = 0.493236, or 49.32 % Total Detection Time: 3.000000 Seconds

Set -points flag: -points 101 for MS COCO -points 11 for PascalVOC 2007 (uncomment difficult in voc.data) -points 0 (AUC) for ImageNet, PascalVOC 2010-2012, your custom dataset

mean_average_precision (mAP@0.5) = 0.493236 Saving weights to /home/admini/darknet/darknet/backup/yolov3-spp-832_last.weights

How can improve the mAP? is it because have only 116 images for validation?

Hope can help

[AlexeyAB]

@astronaut71 Hi,

High fluctiation of mAP happens due to low number of validation images (116 images only).

Try to train more, and if mAP doesn't increase, reduce 10x times learning_rate and train more.

[astronaut71]

I can include more validation images. Would be the data set of 1100 training and 300 validation images ok? What is the optimal data set in my case? should I reduce 10x times learning_rate when have around 330-350 validation images?

[AlexeyAB]

In Training dataset - you should have 4000 images = 2000 images with objects + 2000 images without objects In Validation dataset you should have 200 - 800 images.

https://github.com/AlexeyAB/darknet#how-to-improve-object-detection

for each object which you want to detect - there must be at least 1 similar object in the Training dataset with about the same: shape, side of object, relative size, angle of rotation, tilt, illumination. So desirable that your training dataset include images with objects at diffrent: scales, rotations, lightings, from different sides, on different backgrounds - you should preferably have 2000 different images for each class or more, and you should train 2000*classes iterations or more

desirable that your training dataset include images with non-labeled objects that you do not want to detect - negative samples without bounded box (empty .txt files) - use as many images of negative samples as there are images with objects

[astronaut71]

what do you mean with

your training dataset include images with non-labeled objects that you do not want to detect - negative samples without bounded box (empty .txt files) - use as many images of negative samples as there are images with objects

[astronaut71]

Validaton data set also should include images images without objects? Can be validation set like 670 images = 320 images with objects + 350 images without objects?

[AlexeyAB]

what do you mean with

your training dataset include images with non-labeled objects that you do not want to detect - negative samples without bounded box (empty .txt files) - use as many images of negative samples as there are images with objects

If you want to detect only apples. Then you should have 2000 images with labeled apples + 2000 images with non-labeled plum, pear, mango ... So the network will know that "plum, pear, mango" are not objects.

Validaton data set also should include images images without objects? Can be validation set like 670 images = 320 images with objects + 350 images without objects?

Yes.

[imr555]

Thank you again for such a guided explanation. But still for the negative sample should the text file be completely empty or have its class number. Like for the example above, if 0 was apple it would have a txt with 0 and bbox coordinate s and 1 was non-apple, shouldn't negative txt have a 1 at its start? Sorry for such a layman question and thnx in advance

[AlexeyAB]

@imr555

If there are not objects that you want to detect in the image, then txt file should be empty as described there: https://github.com/AlexeyAB/darknet#how-to-improve-object-detection

desirable that your training dataset include images with non-labeled objects that you do not want to detect - negative samples without bounded box (empty .txt files) - use as many images of negative samples as there are images with objects

[imr555]

Thank You very much and which cfg would you suggest for highest inference on cpu for training on one class?One last question, is it possible to prune weights if one was to make custom cfgs? Again thanks in advance, it's amazing what you are doing, @AlexeyAB

[cyrineee]

@AlexeyAB for multiple classes : should we add negative samples ( an empty txt)? because if a picture include an object for example apple and the other one Car ,than the Car is not a negative sample for the apple ?

[bhargavravat]

@cyrineee : As far as negative samples are concerned you should add those specific images which you don't want to detect. Lets say for example if you don't want to detect car , then you should add images of only car followed by its blank txt. The point which is to be keep in mind is

• You should add only car , not anything else

• Let s say you added one picture that consists of car along with some other object , then it will directly effect your detection accuracy.

[cyrineee]

@bhargavravat yes but i have 30 classes that i want to detect and any other object that's different to those 30 classes i don't want to detect .How can i can add negative samples if it's infinit number ?

[AlexeyAB]

yes but i have 30 classes that i want to detect and any other object that's different to those 30 classes i don't want to detect .How can i can add negative samples if it's infinit number ?

Add as much as possible. First add images with backgrounds and objects which are very similar to your objects. For example, you want to detect cars, but don't want to detect trucks, then add images with trucks as negative samples.

[astronaut71]

Hi @AlexeyAB

I have done what you said. Now I have 1660 images for training ( 1000 are with labeled objects and 660 are images where no workers). Then have 580 images for testing( 350 with objects, and the rest are without). As you advice im using the Spatial-full-model yolov3-spp.cfg model with batch=64 subdivisions=64 width=832 height=832 random=1 and anchors calculated for 832x83.

the cfg. file is

batch=64
subdivisions=64
width=832
height=832
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.001
burn_in=1000
max_batches = 90000
policy=steps
steps=4800,5400
scales=.1,.1

then in each yolo layer `random=1`and in the last one `max=200`. But still after 20 000 iteration maximum mAP is 85% and average loss still decreasing (last one around  0.5). 

Please any advice how to get better mAP? 85% is not so good for the detector

[AlexeyAB]

@astronaut71 Hi,

and anchors calculated for 832x83.

Can you show your anchors?

---

max_batches = 90000 policy=steps steps=4800,5400

If you set max_batches = 90000, then you should change steps=4800,5400 https://github.com/AlexeyAB/darknet#how-to-train-to-detect-your-custom-objects

change line steps to 80% and 90% of max_batches, f.e. steps=4800,5400

[astronaut71]

yes. this are the anchors .

calculating k-means++ ... iterations = 83 avg IoU = 75.24 %

Saving anchors to the file: anchors.txt anchors = 15, 51, 31, 61, 22, 89, 37,103, 38,166, 67,137, 60,266, 104,210, 97,380

[shartoo]

The author is in a bad life time now,though i wanna get the answer.