search

zylo117 / Yet-Another-EfficientDet-Pytorch

The pytorch re-implement of the official efficientdet with SOTA performance in real time and pretrained weights.
GNU Lesser General Public License v3.0
5.2k stars 1.27k forks source link

Evaluating provided D4 model #671

Closed SiBensberg closed 3 years ago

SiBensberg commented 3 years ago

Did somebody try evaluating the provided weights for D4? For some reason the model performs very bad for me while the other models meet the expected performance.

MAP[0.5:0.95] D4: 0.160
MAP[0.5:0.95] D6: 0.505
Ronald-Kray commented 3 years ago

@SiBensberg Hi I have the same issues. I have got decent results on D0, D1, D2. Also good results for --head_only True on D3, D4. However, when I'm using --head_only False on D3, D4. performance is getting worse:) of course I've done second step training as I did. I don't know why....

SiBensberg commented 3 years ago

Ok so probably it is not me. Strange behavior, maybe somebody can throw some light on this.

Ronald-Kray commented 3 years ago

@SiBensberg um do you know how much do I have to have GPU's memory for training D3, D4, D5? I couldn't find the exact required memory for that.

SiBensberg commented 3 years ago

I could see the following for the batchsize of 1: D3 5,5gb D4 8,7gb D5 11,1GB Could not do D6 because my 1080 can not handle it anymore.

Ronald-Kray commented 3 years ago

@SiBensberg I'm measuring the required memory! Thanks! I will also check it out and let you know about it!

Ronald-Kray commented 3 years ago

@SiBensberg Hey, I guess I got the answer to the question. The author said the With such a small batch size, the training will be very unstable. Maybe batch 1 or 2 produces a bad result. Please refer to the below issues https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/issues/162

SiBensberg commented 3 years ago

I already made that experience. My issue is that I can not reproduce the MAP for the provided D4 model.

zylo117 commented 3 years ago

@SiBensberg did you mean the original coco pretrained weights provided by me?

SiBensberg commented 3 years ago

@SiBensberg did you mean the original coco pretrained weights provided by me?

Yes. I have bad results with the provided D4 weights. I also tested D0 D1 and D6 they provide good results.

my question is if somebody can reproduce it or if it is an error from on my side.

zylo117 commented 3 years ago

@SiBensberg update to latest code? I can't reproduce.

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.485
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.679
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.524
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.323
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.533
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.633
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.364
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.576
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.614
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.446
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.664
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.745