search

hltcoe / patapsco

Cross language information retrieval pipeline
Other
18 stars 6 forks source link

switch to a different scoring library #27

Closed cash closed 2 years ago

cash commented 2 years ago

pytrec-eval is a pain to install because it downloads some code from the web and compiles it. pyterrier has a version of the library that is a wheel with pre-compiled library. I've tested it with Linux and it works great. Need to test with Windows and Mac and then we can switch over. This will also make it easier to run CI on Github.

eugene-yang commented 2 years ago

I would suggest using ir-measures. I believe it is more robust than the current pyterrier one. From what I know, they are also planning on revising that wheel.

cash commented 2 years ago

@eugene-yang ir-measures depends on the pyterrier one which is why I chose it since it has fewer dependencies and we don't need the additional functionality provided by ir-measures

eugene-yang commented 2 years ago

I believe ir-measures only adds cwl-eval>=1.0.10 and pyndeval>=0.0.2 aside from pytrec-eval-terrier. Will using ir-measures be more robust and easier to support more evaluation metrics in the future?

cash commented 2 years ago

pyterrier-eval-terrier does everything we need. It is a drop in replacement for pytrec-eval. In the future, if we need additional metric support, we can switch over to ir-measures.