search

terrierteam / ir_measures

provides a common interface to many IR measure tools
https://ir-measur.es/
Apache License 2.0
75 stars 7 forks source link

Recalculate cutoff for Judged measure. #48

Closed hscells closed 1 year ago

hscells commented 1 year ago

The judged measure will report an incorrect value if the size of the run is less than the cutoff. I don't think this should be expected behavior and this PR recalculates the cutoff if it is less than the size of the run for the Judged measure.

seanmacavaney commented 1 year ago

Nice catch @hscells! The change looks good. Mind adding a test case for it? It doesn't look like Judged has any tests at this time (?), but there are some examples tests for other measures you can use as a starting point.

hscells commented 1 year ago

Yeah, sure, let me get to that now :-)

hscells commented 1 year ago

I also see no reason why the Judged measure needs to have the cutoff required. Surely this can be made optional?

seanmacavaney commented 1 year ago

Excellent point, I agree!

hscells commented 1 year ago

I think that should do it! :-)

seanmacavaney commented 1 year ago

Awesome, thanks Harry!

Are you keen to get this out to pypi asap, or are you fine waiting until the next release?

hscells commented 1 year ago

That's fine, can wait until next release 🙂

Thanks!