Experiment mode: Every Word's a button

[johnwdubois]

Background

• Rezonator should support the user who wants to create linguistic experiments.
• Initially, experiments may be performed directly in Rezonator as installed on the user's computer.
• For crowd-sourcing, experiments may be managed via Firebase and/or Mechanical Turk, Prolific.ai, or a similar web-based service.
• For example, one kind of experiment involves asking users to identify Intonation Units in a conversation. Users listen to the audio for each Turn, and then divide the Turn into Intonation Units. Thus, instead of showing them the Intonation Units from the beginning, this would require them to listen and then decide which word is the last word of each Intonation Unit.

What to do Create an "Experiment mode". Once the user selects this mode:

1. For Experiment Mode, turn off all distracting options, buttons, and other elements (similar to playing a game of Rezzles). For example, the Nav window should be turned off by default; also the Tool panel, Help panel, and the main Menu. Show only the words on the Main (Center) screen, along with their speaker labels, plus some special buttons that are specific to Experiment mode (see below), and messages telling the user what to do in the experiment.
2. In the Rezonator Main screen, display each Turn as a long paragraph of words (with or without word-wrap).
3. Every Word acts as a button, which will perform an action as specified for the current Experiment.
4. For the basic experiment design, each Word has a variable (or Field in the Word grid), called UnitEndWord. By default, for each word the value of UnitEndWord = FALSE.
5. The user is shown a message: "Click on the last word of the first intonation unit, and press [RETURN]".
6. When the user clicks on a Word, this highlights the word. When the user presses [RETURN], this makes the Word become the last Word of a newly created Unit (i.e. it sets the value of UnitEndWord = TRUE for this Word).
7. The newly created Unit should now be displayed on a single line of the Main screen. (The remaining words of the Turn are pushed to the next line, where they are displayed as before - as a paragraph with word wrap.)
8. The user is shown a new message: "Click on the last word of the next intonation unit, and press [RETURN]".
9. The user continues until they are satisfied that all of the Intonation Units on the screen have been correctly identified. They then press the button [CONTINUE] at the bottom right corner of the screen.
10. If the user changes their mind about an Intonation Unit, they can click on the last Word, and press [DELETE]. This will set the value of UnitEndWord = FALSE for this Word. This then causes the display to update accordingly.

Further development

1. Each time the user performs an action in the Experiment, record the time in milliseconds (measured from when the current screen was first displayed to the user), as follows:
   • UnitEndWordClickTime: when the user clicks on the word. (Allow for an ordered list of times, if the user clicks on the same word more than once.)
   • UnitEndWordCreateTime: when the user presses [ENTER] to create the new Unit
   • UnitEndWordDeleteTime: when the user presses [DELETE] to eliminate the Unit.
2. To make it easier for the user, Rezonator can supply dedicated playback buttons, designed to play just the relevant piece of audio.
   • To the left of each Line on the Main screen, there should be a [PLAY] button. This button plays the audio for the current Line.
   • When the user creates a new Unit (which then appears on a new Line), a new button is generated to the left of the Line, which plays the audio for the Line.
   • For this to work, each word needs to know its WordEnd time. This is then used to set the UnitEnd time for the new user-created Unit.
3. Another variation. As soon as the user creates a new Intonation Unit, show a pop-up dialogue, asking them to "Select the EndNote for this Unit". Display options for Continuing, Final, Appeal [Question], and Break [Cut-off, Truncation], marked by comma, period, question mark, and double hyphen. Allow also "None", for Units that consist of only laughter, or only breath, etc.
4. _Crowdsourcing option. Develop the capacity for crowd-sourcing, managing the tasks and workers for these experiments via Mechanical Turk, Prolific.ai, or a similar service.

Additional context

1. There can be 2 conditions for this experiment:
   • Words only: The user has access only to the words, and has to guess where the Intonation Units end, based on just the words
   • Words and audio: The user has access to both the words and the audio, and has to decide on where the Intonation Units end based on listening to the audio.

Related to

814

737

482

653

[kayaulai]

If we are going to make this interface, perhaps it can be applied to data that is imported and organised in terms of turns, but with no unit information? After all, I'm sure there will be a lot of conversational transcripts that don't do IUs, and since we're providing an interface for separating turns into IUs, why not allow annotators to use it as well, and not just participants in our (somewhat particular) experiment?

[johnwdubois]

Yes, this should be a tool available to everyone, not just for experiments. If the user has a transcription that shows turns but no IUs, they could use Rezonator to mark the IUs. It could also be used for other units. it could also be used as a tutorial, with a hidden gold standard segmentation into IUs. The user does their best, then gets feedback.