search

AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.71k stars 7.96k forks source link

Low loss and low MAP for damage detection #4359

Open MarcPoulin1 opened 4 years ago

MarcPoulin1 commented 4 years ago

When I train a YOLOv3 to detect damage on images of cars I get a low average loss but also a really low MAP. The model is overfitting a lot (see below). I have trained the model on 500 images of damaged car. chart

Any suggestion on how to reduice overfitting for this type of detection? I can increase the number of photos and change the config file. Also since the damage doesn't have any aspect ratio, what are the optimal anchors for this problem? If I increase the number of anchors will it help on the validation set?

AlexeyAB commented 4 years ago

What mAP do you get on Training dataset?

MarcPoulin1 commented 4 years ago

For the last weights I get this: image

Thanks for your help!

AlexeyAB commented 4 years ago

Your model is trained very well.

https://github.com/AlexeyAB/darknet#how-to-improve-object-detection

for each object which you want to detect - there must be at least 1 similar object in the Training dataset with about the same: shape, side of object, relative size, angle of rotation, tilt, illumination. So desirable that your training dataset include images with objects at diffrent: scales, rotations, lightings, from different sides, on different backgrounds - you should preferably have 2000 different images for each class or more, and you should train 2000*classes iterations or more

MarcPoulin1 commented 4 years ago

Yes I understand this. The problem is that the object that I am trying to identify has too many variations. Damage is not a well defined object by clear features like cars, animals and others. My question is what do you suggest in order to have a better object detector? Should I only increase the number of images or can I optimize some parameters in the config file?

AlexeyAB commented 4 years ago

You use bad images in your training dataset.

Did you get Training and Validation datasets by evenly randomly dividing a Single dataset in a Training 80% and Validation 20% of images? You should do this.

MarcPoulin1 commented 4 years ago

Yes the training and validation sets come from a random split 80%/20% of the same dataset.

MarcPoulin1 commented 4 years ago

I have used the same code for another project that worked perfectly. Here is the python code for the spit (I masked some parts of the code):

split train/validation for early_stopping and create text files with relative paths

import os import random import numpy as np

image_folder = MASKED!!!

Select only images

image_list = [f for f in os.listdir(image_folder) if f.split('.')[-1] in ["jpg","jpeg"]]

Split 80% train/20% validation

random.seed(5) np.random.shuffle(image_list)

split_index = int(0.8*len(image_list))

We dont want the same claim number in both training and validation (image title starts with a 9 digit code)

while image_list[split_index-1][:9]==image_list[split_index][:9]: split_index+=1

training,validation = image_list[:split_index],image_list[split_index:]

def write_file(data,name):

train/test/validation folder

data_folder = MASKED!!!!
data = ['data/obj2/' + f for f in data]
f = open(os.path.join(data_folder,name),'w+')
f.write('\n'.join(data))
f.close()

write_file(training,'train2.txt') write_file(validation,'valid2.txt')

AlexeyAB commented 4 years ago

So I think that your dataset is too small.

Also what mAP@0.05 can you get by using ./darknet detector map ... -iou_thresh 0.05 command?

MarcPoulin1 commented 4 years ago

image At 0.05 I get a better mAP! What does it mean? Thank you very much for your help!

AlexeyAB commented 4 years ago

Accuracy on the validation dataset is very low even at low iou_thresholds. While the accuracy on the training dataset is very high. This means that your training set is not representative - there are very few objects in it that you want to detect. Just collect 4-8x more images and follow these rules: https://github.com/AlexeyAB/darknet/issues/4359#issuecomment-557659270

MarcPoulin1 commented 4 years ago

Thank you! I will increase the number of images and post if I need some help later.

maria-mh07 commented 4 years ago

I'm training yolov3 with COCO dataset (only 5 categories):

Problem: In training the error decreases but mAP is always 0.0%

COCO dataset has already been used for this type of training, so I don't use bad images in my dataset training, right? Any ideas? please help

AlexeyAB commented 4 years ago

@maria-mh07

Aleksei91 commented 4 years ago

Yes I understand this. The problem is that the object that I am trying to identify has too many variations. Damage is not a well defined object by clear features like cars, animals and others. My question is what do you suggest in order to have a better object detector? Should I only increase the number of images or can I optimize some parameters in the config file?

I do not think that damage is object. In COCO terms it is "stuff", I think it can be better use semantic segmentation networks like BiSeNet and others (but its much harder to annotate images). https://paperswithcode.com/sota/real-time-semantic-segmentation-on-cityscapes But I'm not really sure, you should try and compare. If you have only 1% test result you definitely should increase your dataset.

maria-mh07 commented 4 years ago

graf

AlexeyAB commented 4 years ago

@maria-mh07

Show bad.list and bad_labels.list files

maria-mh07 commented 4 years ago
AlexeyAB commented 4 years ago

@maria-mh07

maria-mh07 commented 4 years ago
AlexeyAB commented 4 years ago

@maria-mh07

class_id [0-4], but some txt-annotations are empty. I also think that in train.txt there are some repeated routes. I don't know if this affects.

This is normal.

set GPU=1 CUDNN=1 in the Makefile and do

make clean
make
maria-mh07 commented 4 years ago

Ok thank you very much @AlexeyAB Sorry for this silly question but how do I do it on windows?

AlexeyAB commented 4 years ago

https://github.com/AlexeyAB/darknet#how-to-compile-on-windows-using-cmake-gui

maria-mh07 commented 4 years ago

Thanks!! =)

maria-mh07 commented 4 years ago

I did a training with 128 images, I used yolo_mark for the annotations and it seems to be going well. chart So I think there is somthing wrong with my .txt annotations or train.txt and valid.txt files.

AlexeyAB commented 4 years ago

Does it affect if I have different numbers of decimals?

No, it's normal

0 0.197656 0.700234 0.0 0.0

How can be object with zero size? It's bad.

I have repeated paths to the same imagen in valid.txt/train.txt files.

Its normal.

Check your dataet by using Yolo_mark and run training with flag -show_imgs

maria-mh07 commented 4 years ago
maria-mh07 commented 4 years ago

Would you look at my conversion scrip of the bounding boxes, please? I would really appreciate it because I don't know what I'm doing wrong filter_train.zip Thank you very much for all the help

AlexeyAB commented 4 years ago

@maria-mh07

I can see the bounding box but the image is shifted

It's normal. This is due data augmentation. Your dataset is correct.

Train more using much more images. https://github.com/AlexeyAB/darknet#when-should-i-stop-training

maria-mh07 commented 4 years ago

ok thank you very much for the help! =)