ASK comprehension wh- questions during READ

[JackMostow]

From @JackMostow on February 2, 2017 1:57

Insert occasional random generic wh- questions at end of page to scaffold comprehension

Copied from original issue: synaptek/RoboTutor#221

[JackMostow]

Such questions scaffolded children's reading comprehension in at least one published experiment, and will be easy to implement simply by playing the spoken questions (already translated into Swahili and narrated) at the end of page.

[JackMostow]

1. We can administer a generic free-form spoken-response question simply by playing its narration and waiting for kid to start and finish answering-- how long?

a. Without listening: wait a fixed duration for an answer, e.g. [5] seconds, then go on

+: easy to implement -: will go on even if kid doesn't answer -: will interrupt if kid is still speaking

b. With listening: i. Wait to detect audio input ii. Reprompt after [3 seconds] of silence; if still no speech, go on iii. Once speech starts, wait till kid is silent for [1 second] -: requires ASR, or at least speech/silence detection -: will interrupt if kid pauses but hasn't finished speaking iv. Interrupt after [7 seconds] in case audio input is background or off-task speech

2. (How) visually indicate waiting for spoken response?

a. Don't -- just rely on audio prompt +: easiest

b. Gaze at user in Persona +: easy -: not specific to questions or speech input

c. Display "listening" icon, e.g. photo of Leonora listening +: specifically elicits speech -: more work

3. (How) indicate speech detected?

a. Don't +: easiest +: only option if not listening -: no acknowledgement of user action

b. Change Persona to indicate listening ?: How? Nod?

c. Backchannel +: no graphics required -: not immediate (backchanneling needs to wait for a brief silence to avoid interrupting)

4. (How) visually indicate time limit to answer?

a. Don't +: easiest -: unclear

b. Clock countdown -: may be unfamiliar to kids unfamiliar with analog clocks

c. Shrinking bar +: can use as convention for other time-limited input

d. Dimming page +: makes clearer that page will go away -: disruptive?

e. Numeric countdown +: leverage numeracy -: distracting

f. Other?

5. (How) to acknowledge spoken answer?

a. Don't +: easiest

b. Verbal thanks ("Asante")

c. Let's go on +: appropriate whether or not kid answered

[JackMostow]

Ulani, Wednesday, June 20, 2018 11:35 AM:

Hi Jack,

Thanks for the updated files! I've looked over the links and I think I have a better understanding of how I should approach the task, but I wanted to confirm it with you first.
This is my understanding of my task, please correct me if wrong: 1) For next week, I'll be implementing generic wh- questions in Swahili for Swahili stories, drawing upon translations from Swahili translations -> QUESTIONS 2) The stories that I will be implementing questions for are all the ones in this directory: https://github.com/RoboTutorLLC/RTAsset_Publisher/tree/master/RTAsset_Stories_SW_Set1/assets/story/sw 3) The questions should be randomly inserted, one question after each page. 4) The questions should be implemented in JSON, like in mcq.json. We cannot run the questions with the story in RoboTutor yet to see them in action because interleaving hasn't been implemented yet.

My additional questions are: 1) For me to ask questions relevant to the story content, I think I will need to know what the Swahili stories are saying. I've tried Google Translate, whose usefulness has its limits. It'd be great if you have translations that you could link me to :)

2) In the email after our meeting, point 2a, you said that "For now (the questions) can run by itself, but once Kevin implements interleaving it can intervene during a story.". I'm not quite sure what "it can run by itself" means, is there a way to run the questions in a .json file by itself (without interleaving with stories) in RoboTutor? If so, I know how to run the tasks that you showed me yesterday (story.echo, story.read, etc.) but I'm not quite sure how to run just the questions alone, it'd be great if you could briefly provide instructions or a short demo. This would be very helpful to test my implementation.

3) Is it possible to borrow a tablet from you guys for development purposes? The Android device I have (Amazon Fire Tablet 1st Generation, 2011) is quite old and outdated, and it requires a usb cable to be able to connect to my computer for file transfer. I only have the charging cable, which doesn't work for file transfers. If you guys have a cable I can borrow that instead, but it might be better for me to just borrow an up to date Android device from you guys.

Let me know if you would also like me to contact Kevin or Judith with regards to either of these questions. Thanks again!

Best, -Ulani.

[JackMostow]

Ulani - We (try to) use GitHub for discussions of issues in order to keep them organized, so I posted your email above and will reply here.

Understanding of task: 1) Yes, specifically the generic questions. 2) Yes. Some or all of them may also be in our Google Drive, but I don’t know if the folder names will be the same because at some point we reordered the stories within each level (1, 2, …). 3) Yes, but: a. Maybe just every other page for now. We may alternate them with other types of questions (cloze, picture-matching) once available. b. Avoid asking the same question too often. E.g. order them randomly, then cycle through them. 4) Not exactly. a. The data source for generic questions can be simply the ordered list of items, where each item has attributes for: i. Question type “generic” (for when this data source is interleaved with others ii. Page number (to facilitate interleaving with the story) iii. The English question (the animator graph uses English names for Swahili narrations) iv. The Swahili question (just to be safe) v. Whether to display the text while asking the question b. The animator graph calls Next to get the next item, plays the question aloud, waits for a reply, and goes on. Simply adapt an existing animator graph by deleting the irrelevant portion. c. The Java component needs: i. a Next method ii. a method to display a suitable frame pulled from XPRIZE – Home > AUDIO ASSETS > SWAHILI VIDEOS > Leonora Kivuva facial expressions while:

1. asking the question (e.g. Leonora speaking),
2. awaiting a reply (e.g. Leonora listening with hand to ear), and
3. hearing speech (e.g. Leonora hearing) iii. optionally, a method to wait for the kid’s reply
4. the simplest solution is just to play a [5 second] recording of silence
5. optionally, a method to detect speech vs. silence d. If Kevin’s interleaving method can leave the preceding text page in place, RoboTutor can display: i. Just the text page, with nothing extra ii. The text page plus a smaller version of 4.c.ii.

Additional questions:

1. We don't have translations of the Swahili stories. The generic questions should be text-independent.

2. It's true that you could implement the questions simply as a story, eventually presented in HIDE mode. You'd need to generate a storydata.json for it.

3. Yes, on my office table are a Google Pixel C (no keyboard) and data cable I can lend you.

Kevin and Judith are on the list of assignees for this issue, so they should get updates. You can address them by name, e.g. @kevindeland, to call a comment to their attention.

[JackMostow]

@uhq1 - English-named narrations of the Swahili questions are in Converted English Names - QUESTIONS. If you use the Swahili-named files, you can compile them into a regular story to present using READ. Or you could simply insert the questions into the story using local HIDE mode once Kevin implements it.

[kevindeland]

Resolved in a previous commit.