search

Georgetown-IR-Lab / cedr

Code for CEDR: Contextualized Embeddings for Document Ranking, accepted at SIGIR 2019.
MIT License
155 stars 28 forks source link

consider using pytrec_eval #2

Closed cmacdonald closed 4 years ago

cmacdonald commented 5 years ago

Would you consider using pytrec_eval for validation purposes, instead of trec_eval?

I think there are several advantages of doing so:

  1. You don't have to ship/rely on trec_eval, which might not be complied for the correct platform. In contrast, I think that relying on pytrec_eval means that its compiled as appropriate.

  2. This means that if we provide a "dataset" that contains labels as input, we dont need to have a separate qrels file.

  3. You don't need to fork a separate process to run trec_eval, so it should be faster.

seanmacavaney commented 5 years ago

Hi Craig,

Thanks-- these are all good points. I was initially using pytrec_eval, but switched because I needed custom metric parameters (which are not yet supported) for some of my experiments.

cmacdonald commented 5 years ago

ah interesting motivation. You might want to have an evaluator base class, that can be instantiated for both scenarios.

seanmacavaney commented 4 years ago

Done with #25