Cross Validation

[bayou3]

Hi, I read the training paragraph in your paper, I find that there is no description about cross validation in it. Because my dataset is not large as msmarco, I care about this problem. I process my dataset as this way: I have a human-judged file, which list which docid is relevant, which is not. Then I use these labellbed docid to generate the triple file, namely query, positive_doc, negative_doc. And I also have an initial ranked list, which lists top n docids for each query. I use this initial ranked list to get a dev.tsv file for prediction phase. Should I need to cross validation during the training phase? And how to modify the training code? Or Is it right that the way I do?

[rodrigonogueira4]

For cross-validation, you should split training and dev set at the query level:

• Suppose you have 100 training queries, generate five disjoint sets, each with 20 queries (and their respective relevant (positive) documents).
• Train the first model using the first 4 sets (i.e., 80 queries), and evaluate on the fifth set.
• Train the second model using the last 4 sets, and evaluate on the first set.
• Repeat the process to train another three models. The final dev set performance will be the average of these 5 dev set performances.