How many negative passages should you set per query when eval Cross Encoder on MS Marco Dev?

UKPLab / sentence-transformers

State-of-the-Art Text Embeddings

https://www.sbert.net

Apache License 2.0

14.85k stars 2.44k forks source link

How many negative passages should you set per query when eval Cross Encoder on MS Marco Dev? #1170

Open BugApe opened 3 years ago

BugApe commented 3 years ago

And can you provide a eval code that reproduces these cross Encoder MRR@10 results on MS Marco Dev? https://sbert.net/docs/pretrained-models/ce-msmarco.html

nestorp commented 2 years ago

I would also love to know the answer to this.

nreimers commented 2 years ago

Attached the evaluation script. It runs on the MSMARCO Top-1k passage re-ranking files: For each query, we have 1k passages given which need to be re-ranked

eval_msmarco_rerank_ce.py.txt

nestorp commented 2 years ago

Amazing! Thanks for the quick response @nreimers !