ImageNet V2 evaluation

[samuelstevens]

One of the proposed benefits of WIT-400M and LAION-400M is that they lead to very strong robustness across distributions. This is typically measured by comparing ImageNet 0-shot performance to ImageNet V2 0-shot, ImageNet-R 0-shot, etc.

Did you evaluate the MetaCLIP models on distribution shifts of ImageNet? Even evaluating on simply ImageNet V2 would give a good idea of the models' robustness. Thanks!

[howardhsu]

we have ImageNet variants eval averaged in table 8 of appendix. For ImageNet v2, MetaCLIP has: L14-400M: 69.8%, L14-1B: 72.5% L14-2.5B: 72.6% (vs OpenAI CLIP L14-400M 69.8%, OpenCLIP L14-400M: 65.4%).

[samuelstevens]

Thanks!