search

trecrts / trecrts.github.io

TREC RTS homepage
http://trecrts.github.io/
8 stars 1 forks source link

TREC 2016 evaluation period #2

Closed lintool closed 8 years ago

lintool commented 8 years ago

I'd like to start by discussing the evaluation period - this is the period where all the participant systems will need to run live. The constraints are that NIST starts assessing August 16. SIGIR is July 17-21, and I assume people will want to run the evaluation after SIGIR. So, this gives us a relatively small window to work with.

QUESTION: Last year, we had an evaluation window of 10 days. For participants last year, what did you think? Too long? Too short? Just right?

As a concrete proposal, if we used the same evaluation duration, what about August 1 to August 10, 2016?

As an aside, interesting events that are happening this summer: The 2016 Republication National Convention: July 18–21, 2016 - I bet there'd be interesting tweets, but I don't think participants will want to be running evaluations during SIGIR. Summer Olympics are Aug 5-21, so the evaluation period will cover opening ceremony.

Any comments, suggestions, etc. about either the evaluation period or the task in general?

mossaab0 commented 8 years ago

I think August is a better option. I'd avoid starting on Monday, as that'll be Sunday in some other places. I suggest August 2 - 11. I'd also like to have a dry run period to make sure that the communication with the broker works as expected. That could be within the last week of July.

lintool commented 8 years ago

Okay, August 2 - 11, 2016 is the revised evaluation date.

Per Ellen's directive from NIST, we need to start nailing down the guidelines concretely. Unless we hear any more discussion before (Friday) April 29 - this will be the plan.

lintool commented 8 years ago

It's settled!