Annotation of attention test did not match

[dnaiel0809]

Hi, Thanks for implementing this handy tool. It saved me so much time to conduct research.

However, I found that the attention test (testing pages) feature did not work properly. I ran the offensiveness sample project then added the "testing": ["surveyflow/testing.jsonl"] and set the "test_question_per_annotator": 1 in the yaml file.

The question of attention test appeared correctly but the annotation was like any other task's annotation. From my understanding, the annotation in this case should be the one that we assigned in the testing.jsonl file.

e.g., {"id":"match_finding","text":"This is a test question, please choose: ", "choices": ["1", "2"]}

Please let me know how can I fix it. Again, thanks so much for your hard work.

[Jiaxin-Pei]

Hi @dnaiel0809,

Sorry for the confusion. The potential inputs for attention test questions are designed to be the same as the annotation task, to make it less distinguishable from a normal annotation task.

It seems that the presented setting is missing [test_question_choice] in the text field.

Here is a correct example for an attention test.

{"id":"test_question","text":"This is a test question, please select [test_question_choice].", "choices": ["1", "2", "3", "4", "5"]}

[test_question_choice] is the placeholder where the desired option will be displayed. choices defines all the potential options and the system will automatically choose one from it. In the example above, ["1", "2", "3", "4", "5"] will be randomly used to replace [test_question_choice] and the annotators are required to choose the displayed option as an attention test.

So in this case, you should follow the example above. However, if you have other input options, you could edit the "choices" field, for example, if you have a sentiment analysis task:

{"id":"test_question","text":"This is a test question, please select [test_question_choice].", "choices": ["Positive", "Negative", "Neutral"]}

[dnaiel0809]

Hi @Jiaxin-Pei,

Thanks for the clarification. It is understandable that attention test questions are designed to be the same as the annotation task. However, in our case, we have multiple annotations in a single task. Therefore, it does not make so much sense if we have multiple annotations in an attention test question. For example:

I wondering if is there any workaround that can make the attention question page like any other pre_annotation page which is defined separately in jsonl file under the surveyflow.

Thank you so much!!!

[Jiaxin-Pei]

Hi @dnaiel0809, while it is not currently supported, we can make some modifications to support testing questions as a surveryflow page. However, we would need some time to implement and test this feature. Do you have a timeline for your study?

[dnaiel0809]

Hi, @Jiaxin-Pei

No worries. We decided to change our study design to fit into the system, so modifications were unnecessary. I close the issue for now. Thanks!