Full-tuning and Linear accuracy is strange

[sungbinson]

I'm now proceeding with a reimplementation of full-tuning and Linear accuracy. However, unlike the paper, both full-tuning and linear have performance below 50%. Can you tell me why?

[JieShibo]

Hi.

The results of full fine-tuning, linear probing, and VPT are borrowed from VPT paper and can be reproduced with this repo.

I guess the main reasons for the performance gap are that

• VPT paper uses SGD instead of AdamW for linear, and searches the base learning rate and weight decay for each single task. The optimal hyper-parameters can be found here.
• VPT paper does not normalize the input images, while our pipeline follows NOAH paper and uses normalized inputs (see here).

[sungbinson]

Thanks!!! I have a one question about optimal hyper-parameters I know vtab is a dataset for domain adaptation In optimal hyper-parameters.csv, cifar100 batchsize is 2,048(linear), but vtab training set has a 1,000 training sample is it possible? i think that they use full training sample(cifar100 has a 50,000 dataset)

[JieShibo]

Sorry, I have no idea. This file is provided by VPT’s author @KMnP here. Maybe you can ask her or raise an issue in this repo.

[KMnP]

Hi @sungbinson, we set 2048 for all linear probing experiments since all the experiments including other larger datasets. In the case of VTAB-1k, setting 2048 means we use the entire training data (800 during validation, 1000 during training) for one batch. Let me know if you have other questions, or you can raise an issue in VPT repo like @JieShibo suggested.