search

trec-dd / trec-dd-assessing

Tools for NIST's assessors working TREC DD
MIT License
0 stars 0 forks source link

Marking documents "complete" with no relevant passages breaks things #15

Open gvcormac opened 9 years ago

gvcormac commented 9 years ago

If you mark a document complete, it disappears from the UI list. But no judgement is sent to Waterloo, so Waterloo continues to return the same document. The upshot is that you can "disappear" all your documents and never see any more.

I do not think it is a good idea for Waterloo to similarly "disappear" these documents. They need some affirmative disposition.

shawn67 commented 9 years ago

If a doc is considered to have no relevant passages , the assessors are expected to click discard, but I'll add a check here.