Health misinfo: Selecting pools of documents to judge need to include the unclear or neutral documents

[profsmucker]

At the moment, when we select pools of documents for health misinfo, we divide them into two pools per topic. In the case of 2022, we are dividing them into yes and no pools based on the judgment of the assessor. In the case of 2021, we are dividing them using the supportive or dissuades judgment in combination with the topic's answer to convert into yes and no pools.

For 2021: In the case of "Neutral" documents, these should be added to the pool of documents that are "correct". For example, topic 101 "Will wearing an ankle brace help heal achilles tendonitis?", the correct answer is "unhelpful". If we convert to this a yes/no answer, the answer is "no". Thus, the pool of documents that are correct are all documents labelled "Dissuades", for an "unhelpful" topic should be something that we dissuade people from taking. For topic 101, we'd add "neutral" docs to the "no" pool for preference judging. "Neutral" documents neither support nor dissuade, and thus they are considered "harmless". In terms of the graded preference scores given (the scores that range from -3 to 12, see https://github.com/trec-health-misinfo/resources21 ), you'll see that "neutral" documents are always given a positive score. For "helpful" topics, the neutral docs are put in with "supportive" docs, and for "unhelpful" topics, the "neutral" docs are put in with "dissuades" documents.

For 2022: In the case of "unclear" documents (see: https://docs.google.com/document/d/1qnvgi9Fmm_S8KOhdiogBbJY26hkN3BVKaGhWK5HcZGg/edit?usp=sharing ), which are documents not judged "yes" or "no", we want to put them into the pool with the correct documents. Thus, if the correct answer is "yes" for a topic, then all "unclear" and "yes" documents form a pool to be judged. If the correct answer is "no", then all "unclear" and "no" documents form a pool.

Because of this change, we will want to also decorate the topic question with the following:

(Documents say 'yes' or 'unclear'.) (Documents say 'no' or 'unclear'.)

and not just "yes" or "no", for our pools may now also have "unclear" documents added to them.

[linhnhifm]

I have assigned the issue to myself and will edit the script.

[linhnhifm]

I have edited the script to address the issue above. The script (conversion.py) has been merged.