Closed ytaek-oh closed 4 months ago
ytaek-oh
commented
4 months ago As a reference, my evaluation results on valid models:
| arch | coco_i2t | coco_t2i | flickr_i2t | flickr_t2i | winogavil | IN1k |
|---|---|---|---|---|---|---|
| ViTamin-L-336px | 64.28 | 47.13 | 89.60 | 74.50 | 46.69 | 81.61 |
| ViTamin-L-384px (not reproduced) | 42.08 | 36.57 | 67.60 | 61.94 | 44.90 | 72.00 |
| ViTamin-XL-384px | 68.02 | 50.08 | 92.00 | 78.34 | 43.36 | 82.88 |
| ViTamin-XL-256px | 67.32 | 49.47 | 90.60 | 77.62 | 43.84 | 82.24 |
Beckschen
commented
4 months ago Thanks so much! I have updated the model initialization (config.json) of ViTamin-L-224px, ViTamin-L2-224px, and ViTamin-L-256px. I have also updated the model of ViTamin-XL-336px (was failed to push to HF), and fixed the model jienengchen/ViTamin-L-384px (due to positional embedding) as well.
I have used the mentioned evaluation pipeline to evaluate jienengchen/ViTamin-L-384px, and can reproduce the results with zero-shot ImageNet-1k score of 81.79% for now.
Your suggestions are super helpful. Please let me know if you have further questions.
ytaek-oh
commented
4 months ago Thank you so much for your prompt reply and resolving the issue!
I have downloaded the updated version from huggingface, and corresponding evaluation results are as below. I found that L2 models from huggingface still have the same issue of loading pretrained checkpoints, but when I loaded them locally (borrowing your term, open_clip interface), they worked fine.
=== Results of ViTamin-L-224px model ===
ImageNet zero-shot accuracy: 80.752 # official report: 80.8
=== Results of ViTamin-L-256px model ===
ImageNet zero-shot accuracy: 81.184 # official report: 81.2
=== Results of ViTamin-L-336px model ===
ImageNet zero-shot accuracy: 81.606 # official report: 81.6
=== Results of ViTamin-L-384px model ===
ImageNet zero-shot accuracy: 81.794 # official report: 81.8
=== Results of ViTamin-XL-256px model ===
ImageNet zero-shot accuracy: 82.292 # official report: 82.3
=== Results of ViTamin-XL-336px model ===
ImageNet zero-shot accuracy: 82.696 # official report: 82.7
=== Results of ViTamin-XL-384px model ===
ImageNet zero-shot accuracy: 82.882 # official report: 82.9
=== Results of ViTamin-L2-224px model ===
ImageNet zero-shot accuracy: 0.1 # huggingface interface: not using pretrained checkpoint
ImageNet zero-shot accuracy: 80.896 # openclip interface, official report: 80.9
=== Results of ViTamin-L2-336px model ===
ImageNet zero-shot accuracy: 0.1 # huggingface interface: not using pretrained checkpoint
ImageNet zero-shot accuracy: 81.426 # openclip interface, official report: 81.5
=== Results of ViTamin-L2-256px model ===
ImageNet zero-shot accuracy: 0.1 # huggingface interface: not using pretrained checkpoint
ImageNet zero-shot accuracy: 81.79 # openclip interface, official report: 81.8
=== Results of ViTamin-L2-384px model ===
ImageNet zero-shot accuracy: 0.1 # huggingface interface: not using pretrained checkpoint
ImageNet zero-shot accuracy: 82.066 # openclip interface, official report: 82.1The full log including warnings when loading L2 models via huggingface:
At this point, my issue is cleared, and I will choose open_clip interface to load the models. Thank you again,
Best regards,
Beckschen
commented
4 months ago Thanks for the validation. I am happy that your issues are cleared. I will look into the HuggingFace for L2 variants.
Dear authors, thank you for sharing the codes and checkpoints with your amazing project! I really appreciate that it is easily accessible via huggingface.
While testing the list of models provided in the main page of your repo, some of the models do not work correctly as detailed below. Could you possibly take a look with your model's availability?
jienengchen/ViTamin-L-224px,jienengchen/ViTamin-L2-224pxjienengchen/ViTamin-L-256pxjienengchen/ViTamin-L2-256px,jienengchen/ViTamin-L2-336px,jienengchen/ViTamin-L2-384pxFor example,
This results in nearly zero performance due to un-initialized models.
jienengchen/ViTamin-XL-336pxThe remaining models were successfully initialized and running evaluation was possible, which results in numbers aligned to your report except for
jienengchen/ViTamin-L-384px. When tested on zero-shot imagenet classification, thejienengchen/ViTamin-L-384pxcheckpoint gives 72.0, not your number 81.8.You may take a quick test on zero-shot imagenet with the provided attachment below. It tries to download model checkpoints from the corresponding hf repository.
Prototype code for zero-shot imagenet classification: clip_zeroshot_imagenet.zip
How to run:
Reproduced:
ViTamin-L-336px: 81.61, ViTamin-XL-256px: 82.24, ViTamin-XL-384px: 82.88
Not reproduced:
ViTamin-L-384px: 72.00