search

mlfoundations / patching

Patching open-vocabulary models by interpolating weights
MIT License
89 stars 7 forks source link

About the CIFAR100 finetune score #3

Closed InugYoon closed 1 year ago

InugYoon commented 1 year ago

Hello, I tried to check the CIFAR100 dataset fine-tune performance.

When I finetuned and evaluated both with CIFAR100 dataset, (Without any code modification, I used below command. I followed the command written in Readme for MNIST, except changing #epochs and #batch_size) python src/patch.py --train-dataset=CIFAR100 --epochs=1 --lr=0.00001 --batch-size=16 --model=ViT-L/14 --eval-datasets=CIFAR100 --results-db=results.jsonl --save=models/patch/ViTL14 --data-location=~/data --alpha 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

The result I obtained was Classification head for ViT-L/14 on CIFAR100 exists at models/patch/ViTL14/head_CIFAR100.pt Loading classification head from models/patch/ViTL14/head_CIFAR100.pt Files already downloaded and verified Files already downloaded and verified CIFAR100 Top-1 accuracy: 0.922

When I see the paper, in Figure 14, it seems that the best fine-tune performance on CIFAR100 is below 80%. So I am little surprised the high score. Is it due to the train set difference? Or did I do any mistake?

Thank you.

gabrielilharco commented 1 year ago

Hi @InugYoon, thanks for the question! In the paper (e.g. in Fig 14), CIFAR100 is a "supported" task, which we only use for control. In other words, we do not fine-tune on CIFAR100 for that figure, those numbers are zero-shot with respect to CIFAR100. Hope this helps!

InugYoon commented 1 year ago

Oh, I missed that part! Thank you for kind explanations :)