Image should clearly communicate run ids

osirrc / jig

Jig for the Open-Source IR Replicability Challenge (OSIRRC)

13 stars 3 forks source link

Image should clearly communicate run ids #116

Open lintool opened 5 years ago

lintool commented 5 years ago

We should standardize a mechanism for each image to clearly communicate the run(s) its produces... at least a human-readable runid.

This issue came up as I was preparing the results table for the overview paper.

JMMackenzie commented 5 years ago

I would like to add that using the filename of the run is not what one would expect (or I guess, not what I would expect) since the run itself has a specific column for the runid

lintool commented 5 years ago

Sorry! This is a bit of laziness myself as the runs are actually gzipped and I was gathering stats on the web... :)

In my defense, tho, doesn't TREC require that the file name be the same as the runid at submission time?

JMMackenzie commented 5 years ago

Interesting point, I'm not 100% sure about that but I do recall having some oddly named run files so that might be right! Perhaps we should enforce the same thing?

lintool commented 5 years ago

Agreed!