Evaluation differs from what reported

[dzambrano]

Dear @pierluigiferrari,

Thank you so much for this repo, I found it really useful. I am trying to evaluate the models you provided and I can't achieve exactly the same numbers are shown in the documentation.

1. First of all, for the evaluation of the 300 model, the ssd300_evaluation.ipynb example, you load the weights of a model fine tuned ( weights_path = 'path/to/trained/weights/VGG_VOC0712_SSD_300x300_ft_iter_120000.h5' ) which should have the scales for MS COCO. However, the scales for Pascal are used.
2. Moreover, when I evaluated your script with the correct settings, I do not achieve the same results reported.

• VGG_VOC0712_SSD_300x300_iter_120000 achieved 76.5 instead of 77.5 mAP;
• VGG_VOC0712_SSD_300x300_ft_iter_120000 achieved 80.6 instead of 81.2 mAP;
• VGG_VOC0712_SSD_512x512_iter_120000 achieved 78.9 instead of 79.8 mAP;
• VGG_VOC0712_SSD_512x512_ft_iter_120000 achieved 82.7 instead of 83.2 mAP.

I used the same settings you described in ssd300_evaluation.ipynb (with the relative changing in scales or input image where needed). My system runs on Tensorflow 1.12 and cuDNN 7.0.

Am I doing something wrong? Is there a specific parameter that needs to change in your example?

[DonghoonPark12]

I didn't try all, but same with me VGG_VOC0712_SSD_300x300_ft_iter_120000 achieved 80.6 instead of 81.2 mAP. In evalutation code, there are two evaluation metric. 'Previous metric' and 'Recent metric'. I think code author wrote using recent VOC metric, cause, recent one give us higher mAP than previous one in my model. I didn't try using VGG_VOC0712_SSD_300x300_ft_iter_120000.h5 though. My guess.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[eypros]

I am having the same issue also. I cannot get the reported mAP neither for the provided ported weights from Caffe nor for the pretrained model from scratch.

• The reported mAP should be 0.772 according to the original caffe or 0.775 according to the repo. Also, how is that to use the caffe ported pretrained weights and get 0.3% higher performance?

• The reported mAP for the provided model trained from scratch should be 0.771 while I get 0.761. Any explanation on that divergence?