Optimal hyperparameters

[ptomaszewska]

Hello! The paper contains information that the optimal hyperparameter values for each experiment can be found in Appendix C. However, there is no such information in Appendix C. Could you share the optimal hyperparameter values for each experiment to save compute power required for grid search?

[KMnP]

Thanks for catching that! We were to put in the Appendix but cannot fit the table in a page. The hyperparameter values used (prompt length for VPT / reduction rate for Adapters, base learning rate, weight decay values) in Table 1-2, Fig. 3-4, Table 4-5 can be found here: Dropbox / Google Drive.

I also update the repo to include this information in this commit.

Let us know if you have any other questions!

[zhaoedf]

Thanks for catching that! We were to put in the Appendix but cannot fit the table in a page. The hyperparameter values used (prompt length for VPT / reduction rate for Adapters, base learning rate, weight decay values) in Table 1-2, Fig. 3-4, Table 4-5 can be found here: Dropbox / Google Drive.

I also update the repo to include this information in this commit.

Let us know if you have any other questions!

1. why only three run since you have 5 seeds in demo.ipynb, e.g. eurosat. So for promptdeep and vit b16 and what are the seeds of these 3 runs?
2. what is the meaning of '0.1' in 'promptdeep_0.1'

[Xiangmoshihuang]

Thanks for catching that! We were to put in the Appendix but cannot fit the table in a page. The hyperparameter values used (prompt length for VPT / reduction rate for Adapters, base learning rate, weight decay values) in Table 1-2, Fig. 3-4, Table 4-5 can be found here: Dropbox / Google Drive.

I also update the repo to include this information in this commit.

Let us know if you have any other questions!

Hi, does promptdeep_0.1 mean PROMPT.DropOut? I found the hyperparameter values of "vtab-structured: dmlab" in the Demo.ipynb are different from the "vtab_all_release.csv".

[KMnP]

@Xiangmoshihuang @zhaoedf promptdeep_0.1 indeed means vpt-deep with dropout 0.1.

@Xiangmoshihuang The lr values in "vtab_all_release.csv" represents the base learning rate, which is the same one in the demo for "vtab-structured: dmlab" (# base_lr = 1.0). Please note that train.py takes in actual learning rate values, instead of the unscaled base learning rate values in CSV files.

[KMnP]

@zhaoedf

why only three run since you have 5 seeds in demo.ipynb, e.g. eurosat. So for promptdeep and vit b16 and what are the seeds of these 3 runs?

In the demo, we provided seeds used for the ensemble experiments in Appendix B (page 21 of the paper). We did not set explicit seed values for the experiments in Table 1, which reports mean accuracy over 3 runs, as stated in Sec4.1. Both set of experiments use same hyper-parameters for the final runs.

[zhaoedf]

@zhaoedf

why only three run since you have 5 seeds in demo.ipynb, e.g. eurosat. So for promptdeep and vit b16 and what are the seeds of these 3 runs?

In the demo, we provided seeds used for the ensemble experiments in Appendix B (page 21 of the paper). We did not set explicit seed values for the experiments in Table 1, which reports mean accuracy over 3 runs, as stated in Sec4.1. Both set of experiments use same hyper-parameters for the final runs.

thx, i understood

[Xiangmoshihuang]

@KMnP Thanks!

[ZYHCMU]

@KMnP Hi,

If I understand correctly, About FGVS, you select the model with the best train or Val accuracy by searching 9 Lr and 4 Wd (fix optimal prompt length here?) and then report its corresponding test accuracy. About VTAB, you do the above thing first, and then re-train the whole 1k training data with the best hyper params in the previous step to report the test accuracy.

If yes, do you have the best hyper params for each dataset so I can re-run table 14 quickly? (it’s a little painful to find the best params through your large grid search tables) Thanks.

[pgsld23333]

@zhaoedf

why only three run since you have 5 seeds in demo.ipynb, e.g. eurosat. So for promptdeep and vit b16 and what are the seeds of these 3 runs?

In the demo, we provided seeds used for the ensemble experiments in Appendix B (page 21 of the paper). We did not set explicit seed values for the experiments in Table 1, which reports mean accuracy over 3 runs, as stated in Sec4.1. Both set of experiments use same hyper-parameters for the final runs.

@KMnP Hello, I would like to know is such a 3 replicate experiment conducted for the dataset in FGVC? Thank you very much!

[thomascong121]

Hello, I think the google drive or the dropbox link does not work now.