search

tensorflow / tpu

Reference models and tools for Cloud TPUs.
https://cloud.google.com/tpu/
Apache License 2.0
5.21k stars 1.77k forks source link

MNasNet pre-trained model accuracy mismatch? #666

Open tigert1998 opened 4 years ago

tigert1998 commented 4 years ago

I downloaded the saved_model format mnasnet model from https://storage.googleapis.com/mnasnet/checkpoints/mnasnet-a1.tar.gz. A text file in the folder claims the given tflite and saved_model model has the top1 accuracy of 75.180286% but I don't know the preprocessing method so I cannot reproduce it.

I have already tried the exact same preprocessing method in https://github.com/tensorflow/tpu/blob/master/models/official/mnasnet/preprocessing.py and tested it under ImageNet 2012 validation set but it only gave 73.534% top1 accuracy.

mgolub2 commented 4 years ago

I think 75% would be the test accuracy, so it's not that weird the validation set would be different?

I'm pretty curious how the test accuracy was computed at all though, since the imagenet test server has been down for quite some time (> 1 year? ), at least for ILSVR2012 and ILSVR2017.