search

tensorflow / models

Models and examples built with TensorFlow
Other
77.05k stars 45.77k forks source link

Object detection ssd_mobilenet_v2 evaluation on COCO #5106

Closed snownus closed 6 years ago

snownus commented 6 years ago

Get the release pretrained model ssd_mobilenet_v2 evaluation on COCO, how to evaluate? Seems the official code only provides evaluation on Open Image Challenges 2018. I have tried model_main.py, but doesn't work. python model_main.py --pipeline_config_path=experiments/coco/configs/pipeline.config --checkpoint_dir=models/ssd_mobilenet_v2/ --model_dir=output/experiments/coco/ssd_mobilenet_v2/ --run_once=True --alsologtostderr

@pkulzc

pkulzc commented 6 years ago

Your command looks right, what message did you get ?

snownus commented 6 years ago

It doesn't find the latest checkpoint path. I have addressed it. remove tf.train.latest_checkpoint by using the latest one manually. It works now.

May I know which COCO to evaluate? COCO2014 or COCO2017 for evaluating the official performance.

Thanks a lot for instant reply.

@pkulzc

pkulzc commented 6 years ago

Eval input path should be in your config file.

snownus commented 6 years ago

@pkulzc , Yes, currently I use COCO2014 to test, the performance is 24.1, while the official one is 22. So I want to know which COCO dataset the your side use? Is it COCO2017? The official website hasn't told which COCO dataset to measure the performance.

netanel-s commented 6 years ago

I believe the difference is that the official metric is on COCO's test set, while I assume you tested on COCO's val set.

snownus commented 6 years ago

@netanel-s , ok. Thanks.

One more question, that, does the SSD-MobilNetV1 and SSDlite-MobileNetV2 has the same preprocessing as in https://github.com/tensorflow/models/blob/master/research/object_detection/object_detection_tutorial.ipynb. I run SSDlite-MobileNet-V2 (Only change the model to SSDlite-MobileNetV2 download tar.gz) and looks like that the performance is worse than SSD-MobileNet-V1.

I have tried to check the preprocessing for v2 and v2, and haven't found difference.

netanel-s commented 6 years ago

Is the claim that the performance is worse based only on the given test images of the tutorial? These two examples might not be representative of the models' performances. Try evaluating both on COCO's val set or at check the results on much more examples.

snownus commented 6 years ago

@netanel-s , the issue is addressed. Thanks.