search

zylo117 / Yet-Another-EfficientDet-Pytorch

The pytorch re-implement of the official efficientdet with SOTA performance in real time and pretrained weights.
GNU Lesser General Public License v3.0
5.21k stars 1.27k forks source link

Low mAP and poor detection of cones dataset #669

Open IzzeddinTeeti opened 3 years ago

IzzeddinTeeti commented 3 years ago

Hi there,

I used an annotated cones dataset. It was annotated using supervisly. Then I used this repo to convert the annotations to COCO format. I trained effienctdet-d0 using this command python train.py -c 0 -p cones --data_path ../cones_dataset/good/ --batch_size 64 --lr 1e-5 --num_epochs 100 --load_weights weights/efficientdet-d0.pth knowing that I have 800 training and 340 validation images. While training I got a very high classification loss. I run COCO_eval.py and got the following results:

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.243 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.351 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.295 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.014 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.323 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.669 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.114 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.287 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.293 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.017 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.422 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.758

mAP are quit low, so I run efficientdet_test.py to visualise the results, and got the following image, which shows that most of the cones were not detected, and some cones have two different labels:

img_inferred_d0_this_repo_0

Does anyone have any advice on how to have better mAP and better inference results?

Ronald-Kray commented 3 years ago

@IzzeddinTeeti Hi, have you tried head only true? I have got decent results on D0, D1, D2, not D3 yet:)> You should do 2 steps training 1st: head only true, 2nd: head only False. Please see the tutorials.

SiBensberg commented 3 years ago

Probably Formula Student right? You should definitly do what @Ronald-Kray said. Additionally, try scaling the anchors down in params.yaml if you did not already. Had a major improvement with it. Despite I was using the official EfficientDet implementation on Tensorflow. In this repo the smalles anchor is 32 pixels regarding the input size, which can be too big for cones. Also I would suggest to try without any augmentation, because obviously the objects are heavily dependent on color. The "double detections" could vanish when training more epochs.

IzzeddinTeeti commented 3 years ago

@IzzeddinTeeti Hi, have you tried head only true? I have got decent results on D0, D1, D2, not D3 yet:)> You should do 2 steps training 1st: head only true, 2nd: head only False. Please see the tutorials.

Thank you, I tried doing this, got better results but only slightly better:

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.285 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.381 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.342 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.009 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.397 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.799 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.125 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.303 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.303 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.008 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.435 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.84

IzzeddinTeeti commented 3 years ago

Probably Formula Student right? You should definitly do what @Ronald-Kray said. Additionally, try scaling the anchors down in params.yaml if you did not already. Had a major improvement with it. Despite I was using the official EfficientDet implementation on Tensorflow. In this repo the smalles anchor is 32 pixels regarding the input size, which can be too big for cones. Also I would suggest to try without any augmentation, because obviously the objects are heavily dependent on color. The "double detections" could vanish when training more epochs.

Yes, I am a formula student. You mean anchors rations right? knowing that my input size is 640, how did you calculate the number 32?

SiBensberg commented 3 years ago

@IzzeddinTeeti I thought you were the standard for D0 which would be 512. The smallest anchor is actually 8 but in backbone.py the anchors gets multiplied by 4.
So you could try anchors_scales: '[0.5, 0,75, 1 ]' maybe you could even add more scales than 3.

IzzeddinTeeti commented 3 years ago

@SiBensberg, thank you so much, I have way better results after modifying the anchors_scales. I am also trying to apply your second advise which is suppressing the augmentation, but I am stuck at that.

SiBensberg commented 3 years ago

Nice! I am happy to hear that.

Transforms for the train dataset would be here: https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/blob/15403b5371a64defb2a7c74e162c6e880a7f462c/train.py#L114-L116

as far as I understand there is just a flip augmentation implemented in this repo which you can leave the way it is. But you can obviously try implement others as they could improve your performance. I just made some bad experiences with color hue or jitters.

zylo117 commented 3 years ago

head_only = False can lead to better performance but you must have more data to train or it overfits