A concerned question about adding a new baseline.

[FanqingM]

a good job! I have a very concerned question that I hope can be answered. Reading the article, the standard for measuring model capabilities should be the test set of the dataset? What if we directly select the sample in the train dataset that is most similar to the test set as the baseline result? For example, if they are all text, directly embed them into vectors and then calculate the sample that is most similar to the test set.

[xiamengzhou]

Hi! Thanks for your question.

• We select examples that are similar to the validation examples (which are also the in-context learning examples) from the training set, and we have the representation data selection baseline presented in table 3
• In machine learning generally, it's prohibited to use any information from test set (including inputs and outputs) as it's basically cheating, and we adhere to this rule by using a small validation set for data selection

Maybe I misunderstood something here.. Let me know if it helps address your question.

[FanqingM]

Thank you for your response. It seems that i have a mistake about table 3. Now I get it! Great work!