search

naver / splade

SPLADE: sparse neural search (SIGIR21, SIGIR22)
Other
751 stars 84 forks source link

bug: TREC 2020 qrel_binary.json, score 1 should be treated as negative instead of positive #63

Open zhiyuanpeng opened 5 months ago

zhiyuanpeng commented 5 months ago

Hi,

For TREC, 0,1 are negative, 2,3 are positive. However, in qrel_binary.json, 1 is mapped to positive (1). For instance, in qrel.json:

{"23849": {"1020327": 2, "1034183": 3, "1120730": 0, "1139571": 1

in qrel_binary.json:

{"23849": {"1020327": 1, "1034183": 1, "1120730": 0, "1139571": 1,

doc 1139571 should be mapped to 0 in qrel_binary.json.