Support separate layout templates for surveyflow and annotation pages

[AmelieW]

Hi, This feature would be super useful for our task! Is there any workaround for the current version that I could use now to present annotators with separate annotation schemes/new page with tasks? Essentially, I would like to show an instances multiple times, but with varying annotation schemes. Any pointer would be much appreciated!

[davidjurgens]

This is a pretty cool idea and probably something we should support better in the long run. For your use case, do you want the item to be shown with different annotation schemes one after another (i.e., the item stays the same for a few pages in a row) or would these be shuffled?

@Jiaxin-Pei I think you could probably do this with an additional configuration option that lets the user specify multiple schemes but also say which ones are shown at a time like annotation_scheme_sequence = [...] where the list contains tuples with the names of the schemes to be shown at the same time (or also supports just a list of names if it's one at a time)

[AmelieW]

Ideally, we would want them to do Task A/annotation scheme A for let's say 5 instances, then we go to Task B/annotation scheme B and we show the same 5 instances again. Does that make sense? I think an annotation_scheme_sequence would be awesome, that way we could also add a page with instruction in between the tasks similar to what I can now do with the survey flow.

[rmovva]

Hello, is this feature still open? I would like to have a different HTML layout for the annotation examples and for the surveyflow, if possible.

Also, I noticed that on my surveyflow pages, the filename is showing up in the top left of the page, and I'm not sure why. I think usually it's a bordered box, which is gone because of how I changed the layout file.

For context, here's my current YAML file, and this is my custom layout file.

Thanks so much for your work on Potato, it's been very useful!

[rmovva]

To add some more detail, the main reason why I would like a separate HTML page for surveyflow and annotation is so that I can instructions on the annotation page, but not on the surveyflow page. For example, see here, on the commented out sections, like

<div class="alert alert-info" role="alert">
  <p>You will be shown a <strong>prompt</strong> that a user sent to an AI chatbot and two possible responses, <strong>Response A</strong> and <strong>Response B</strong>. Please read the prompt in full. Then, you will decide which of the two responses is better. You should judge the responses based on <strong>whatever criteria you think are important.</strong></p>
  </div>

Is there a way I can include these messages on the annotation pages without them showing up on the surveyflow pages?

Another option is to have these messages only on the Instructions page. However, I'm not sure how to add blocks of highlighted text to my instruction.jsonl file in the appropriate format.

A final question: is there a way to include some example data instances in the pre-annotation surveyflow that accept jsonl data files in the same format as the main annotation data?

[Jiaxin-Pei]

Hi @rmovva,

I just pushed an update to support different HTML layouts for surveyflow and annotation pages. Please check out the documentation here.

For both surveyflow pages and annotation pages, the input text supports HTML grammar, so you can actually use the alert message when you define the survey files (either in text or choices) and make sure you use pure_display as the schema.

The filename location seems to be affected by your custom header file instead of the HTML layout file. You could hide the file name by defining the surveyflow page with a dictionary in the .yaml file. Check out this link for detailed instructions.

Let me know if you have any questions! Please make sure you upgrade potato-annotation to 1.2.1.6 to see all these latest features.

[rmovva]

Thanks very much @Jiaxin-Pei, I will give this a try soon. I appreciate the quick change and will let you know if I have further questions.

One thing I was wondering: is there a way to have a pre-study phase with a few example questions that show users how the task will look, but without screening them out?

[Jiaxin-Pei]

@rmovva I think the easiest way is to display the screenshots/gif of the task in the annotation pages. You can check out this example project to see how to do that. In short, you can embed gifs/images/videos in any surveyflow page.

Another way to do that is, just like how the consent question is set up, you can add several pages of example questions as pre-annotation survey pages. You can add the correct answer in the question text, and ask the annotator to choose the right option to proceed (just like the consent questions).

[davidjurgens]

@Jiaxin-Pei, I think @rmovva's idea is to have actual questions which could be useful as a pre-study phase—e.g., questions where you show the answer after the user clicks something and explains the answer or why their choice wasn't correct (something like a training warm up).

On Wed, Feb 14, 2024 at 5:30 PM Jiaxin Pei @.***> wrote:

@rmovva https://github.com/rmovva I think the easiest way is to display the screenshots/gif of the task in the annotation pages. You can check out this example project https://github.com/davidjurgens/potato/tree/master/project-hub/reading_comprehension to see how to do that. In short, you can embed gifs/images/videos in any surveyflow page.

Another way to do that is, just like how the consent question is set up, you can add several pages of example questions as pre-annotation survey pages. You can add the correct answer in the question text, and ask the annotator to choose the right option to proceed (just like the consent questions).

— Reply to this email directly, view it on GitHub https://github.com/davidjurgens/potato/issues/39#issuecomment-1944840538, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHO4LLGZNHRSH3RWVAGZUTYTU3ILAVCNFSM6AAAAAAWEKYIYOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBUHA2DANJTHA . You are receiving this because you commented.Message ID: @.***>

[rmovva]

Thanks both. Jiaxin's suggestion of a screenshot is sufficient for now, but yes, I think a training phase would be helpful. Specifically I think the training phase should use the annotation layout, not the surveyflow layout -- or, have a customizable option for a "training" layout.

It would be nice if the training phase allows you to input data in the same format as the annotation phase ("id", "text", etc). And then maybe there could also be some logic where the user can specify, for each example, what a pop-up window should display when the user selects each choice.

This isn't urgent though, so feel free to ignore.

I was also wondering if I could use the prestudy test as an approach to show the user some examples, except set a low passing threshold / give them a chance to go back and change their answers if they don't pass. But I wasn't quite sure how the prestudy test works - it looks like that part of the docs is still under construction (e.g., does it use the annotation template or the surveyflow template).

Thanks, Raj

[Jiaxin-Pei]

@rmovva Currently the prestudy test is majorly used to filter users and there is no option to retake the test. I could help to add an option for that in the coming weeks! If you want to take a look at how it currently works, you may check out this example-project. I will add more guidelines in the doc for this function! Thanks a lot for spotting that.

In the longer term, we would like to support a separate training process(layout) different from the prestudy test. In the training process, users are allowed to change their answers and receive feedback regarding each of the choices. The deployer can choose to whether add a separate test after this training.