search

pytorch / hub

Submission to https://pytorch.org/hub/
1.38k stars 240 forks source link

Reported accuracy mismatch for `vgg11_bn` #313

Open nps1ngh opened 1 year ago

nps1ngh commented 1 year ago

The hub documentation here: https://pytorch.org/hub/pytorch_vision_vgg/ reports the vgg11_bn top-1 error to be 26.70. That is, a top-1 accuracy of 73.30. This is almost as good as the resnet34 reported here: https://pytorch.org/vision/main/models/generated/torchvision.models.resnet34.html

But the main problem is that torchvision reports a top-1 accuracy of 70.37 for the it here: https://pytorch.org/vision/main/models/generated/torchvision.models.vgg11_bn.html

I guess the latter reported accuracy is correct (70.37) and the one reported here by hub is incorrect?

Or am I missing something? Is this possibly related to https://github.com/pytorch/vision/issues/223? (It's a really old issue though.)

Thanks!

NicolasHug commented 1 year ago

Thanks for the report @nps1ngh . Good catch!

For all torchivsion models, the correct reference for accuracies should be this table: https://pytorch.org/vision/main/models.html#table-of-all-available-classification-weights (or the specific model pages like the one you gave above, which has the same info). Unfortunately, the ones reported on the torchhub website may go out of date. I opened https://github.com/pytorch/hub/issues/318 to try to think of solutions / mitigations.

Meanwhile, it's probably worth updating the VGG table with the latest correct values. LMK if you would you like to open a PR for that. Thank you!

nps1ngh commented 1 year ago

Sure, I'll open one!