Closed kashpop closed 5 years ago
(Sorry for the late response... I don't personally own this repo so I didn't get notification for the issue)
You are right. These evidence are irrelevant. I just checked the data and this seems like an annotation error. For some reason 3 out of 4 crowdworkers think this is a valid evidence.
I will remove this particular evidence in the next release of the dataset (which will be available in a few hours).
I observe some inconsistencies in the dataset, especially the way evidence id is assigned in
perspectrum_with_answers_v0.2.json
. For example, take the first json object fromperspectrum_with_answers_v0.2.json
:Extracting the perspective ids: 3695, 24076 and 24077 (from
perspective_pool_v0.2.json
), here is what I get:However, when I try to get the evidence text using evidence ids: [2611, 7481, 2934] (from
evidence_pool_v0.2.json
), here is what I get:None of these evidence are relevant to the claim or perspectives.
How are the perspectives and evidence linked with each other? Am I missing something?