osirrc / jig

Jig for the Open-Source IR Replicability Challenge (OSIRRC)
13 stars 3 forks source link

Images reporting expected effectiveness metrics #100

Open lintool opened 5 years ago

lintool commented 5 years ago

If we can find a standardized way for images to report their expected effectiveness metrics (AP, NDCG, etc.) - the jig could check and verify scores automatically.

That way, we'd be able to do end-to-end regression testing on entire IR systems!

I think this would be a pretty cool feature...

Thoughts?