Question on validation set on 1k experiment

[jp7c5]

Hello. Thanks for sharing this nice work. I read the code and the paper, but I am not sure about the setup of validation set on 1k experiment. In Section 3 of the main paper, it is written that "Note, for VTAB-1k we define custom train set with 800 examples and validation set with 200 examples.", and in appendix I, I see that "Note that when training on 1000 examples, we perform model selection using the regular validation set for each task.". So my question is that on VTAB-1k experiment, to select the best hyperparameters, did you use validation set of size 200 or the whole validation set for each task?

[ghost]

Thanks for spotting, you have found a bug in Appendix I. For VTAB-1k, we select the hyperparameters using the 200 validation examples. Other hyperparameter search strategies are fine, provided, of course, that only the 1000 labelled examples are used for both training and hyper selection.

[jp7c5]

I see. Thanks for clarifying this!