search

UKPLab / gpl

Powerful unsupervised domain adaptation method for dense retrieval. Requires only unlabeled corpus and yields massive improvement: "GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval" https://arxiv.org/abs/2112.07577
Apache License 2.0
315 stars 39 forks source link

Evaluation data format #19

Open Matthieu-Tinycoaching opened 1 year ago

Matthieu-Tinycoaching commented 1 year ago

Hi,

1/ How should the evaluation data format be as passed in the evaluation_data argument? Could you provide me some example of evaluation data and how it should be formatted?

2/ How does the evaluation work on these data? What are the tests passed and labels used?

Thanks!

adrienohana commented 1 year ago

Haven't figured out how it works yet, I tried to feed some evaluation data with a folder containing : corpus.jsonl, queries.jsonl and qrels/train.tsv . That doesn't work... nothing happens.

Would be nice have some training metrics that show what's happening. Plotting the loss maybe ? Or evaluating the data every hundred steps... Seems that my metrics keep improving way after 100k steps (beir metrics are NDCG, MAP, Precision and Recall @ K)