search

tira-io / tira

The source code for the TIRA Shared Task Platform
https://www.tira.io
MIT License
14 stars 8 forks source link

Feature test persist run file #603

Closed janheinrichmerker closed 6 months ago

janheinrichmerker commented 10 months ago

The persist_and_normalize_run method currently creates invalid run files when the run DataFrame's score column contains nan or None values. The proposed tests are meant to detect exactly this bug, in order to be fixed.

mam10eks commented 10 months ago

Ah, yes, now I see, this is only the test, not the implementation. I would suggest to add another small test where the rank is available, or? And maybe it should fail if no rank and no score is available, because then we would likely mask some weird errors?

janheinrichmerker commented 10 months ago

Yes, the method should probably fail with an exception if no score (or no rank) is given. I recall that tools like trec_eval would usually look only at the scores.

janheinrichmerker commented 6 months ago

I don't have time to work on this unfortunately