search

ml-jku / cloob

Other
151 stars 11 forks source link

Model checkpoints #15

Open namin-an opened 3 months ago

namin-an commented 3 months ago

Hello,

Are the checkpoints (e.g., CLOOB - ResNet50) that you provided the models pretrained on CC or YFCC? When I ran your demo notebook, I got the zero-shot classification performance of 3.66% for CLOOB - ResNet50, which seems to be similar to results in Table 1 (3.06, CLOOB trained on CC) than Table 2 (28.9 CLOOB trained on YFCC).

fuersta commented 3 months ago

Hello, the checkpoints are trained on YFCC. You should get the correct performance. Your results seem to be wrong. Did you try the same for other zeroshot datasets? What are the results?

namin-an commented 3 months ago

image image image

Hello,

Thank you for your reply.

The attached images are the code that I used to evaluate your pre-trained model (CLOOB—ResNet50) on the Birdsnap test dataset (1,855 images). I have also tried the Country211 dataset, which gave me lower results than the one reported in the paper.

Could you please go through my code and see if there are mistakes that I made? :( I just changed the file name from clip to clip_cloob and downloaded the model checkpoint (https://ml.jku.at/research/CLOOB/downloads/checkpoints/cloob_rn50_yfcc_epoch_28.pt) and placed it under my checkpoint directory.

namin-an commented 3 months ago

BTW, I got the same score (18.8) as the one in the paper for CIFAR100.

fuersta commented 3 months ago

Did you try cloning the repo again and without changing anything running the notebook again? I did the experiments with a clean repo and could get the same results as stated in the paper.