search

balancap / SSD-Tensorflow

Single Shot MultiBox Detector in TensorFlow
4.11k stars 1.89k forks source link

evaluation mAPs don't match reported numbers #77

Open starcrest opened 7 years ago

starcrest commented 7 years ago

I downloaded the models from the evaluation section and followed the instructions there to run eval_ssd_network.py.

I get the following numbers: CHECKPOINT_PATH=./checkpoints/VGG_VOC0712_SSD_300x300_iter_120000.ckpt AP_VOC07/mAP[0.59928032818577548] AP_VOC12/mAP[0.60921401213750548]

CHECKPOINT_PATH=./checkpoints/VGG_VOC0712_SSD_300x300_ft_iter_120000.ckpt AP_VOC07/mAP[0.74313309022098] AP_VOC12/mAP[0.76659688181365315]

CHECKPOINT_PATH=./checkpoints/VGG_VOC0712_SSD_512x512_ft_iter_120000.ckpt AP_VOC07/mAP[0.77656339926263329] AP_VOC12/mAP[0.80273876556779]

So that's mAP of 0.599, 0.743, and 0.777. The readme reports mAP of 0.778, 0.817, and 0.837 respectively. Am I misunderstanding? Can you help me understand the discrepancy?

Thanks a lot for making this project available!

casperthuis commented 7 years ago

Have the exact same results here. Would also be interested in the cause of this.

iwcvanderlinden commented 7 years ago

It seems this might be because the readme reported values are actually evaluated on the train data. See #48

huunhan312 commented 7 years ago

@iwcvanderlinden The default checkpoint can reach 82% on VOC07 test data. So, the result maybe is real. However, the given weight factors do not match the reported results.

I run the available weight in directory checkpoints named "ssd_300_vgg.ckpt". The result on VOC07 Test are AP_VOC07/mAP[0.82107575232572283] AP_VOC12/mAP[0.8562692165386977]

CangHaiQingYue commented 6 years ago

@huunhan312 I run the "ssd_300_vgg.ckpt" got the result on VOC07_test,mAP = 0.822. But after a month I rerun “ssd_300_vgg” on both voc07_test and VOC07_trian, got mAP = 0.736. I have no idear about this. Can you help me.

CharlesShang commented 6 years ago

guess the ssd_300_vgg.ckpt model is converted from caffe-ssd model... caffe-ssd

huangh12 commented 6 years ago

The evaluation results I obtained:

ssd_300_vgg.ckpt AP_VOC07/mAP[0.82107502332759208] AP_VOC12/mAP[0.8562692241613149]

VGG_VOC0712_SSD_300x300_iter_120000.ckpt AP_VOC07/mAP[0.59928033396046509] AP_VOC12/mAP[0.60921385591790489]

VGG_VOC0712_SSD_300x300_ft_iter_120000.ckpt AP_VOC07/mAP[0.74313215436144464] AP_VOC12/mAP[0.76659691986281664]

VGG_VOC0712_SSD_512x512_ft_iter_120000.ckpt AP_VOC07/mAP[0.77656339500846039] AP_VOC12/mAP[0.80274009098080834]

TingZhang08 commented 6 years ago

@huangh12 Could you help me? I'm confused that how to compute AP_VOC12/mAP. Actually, I only use VOC07 TEST to test? Thank you?

huangh12 commented 6 years ago

@TingZhang08 Hello, just run the eval_ssd_network.py as the readme.md said, you should get the mAP results once finished. On the principle behind mAP calculation, you can refer to this website for details.