1.8.9.1 change mode within story

[JackMostow]

This issue succeeds READ add PARROT, HIDE, REVEAL #262.

Allow story mode to change sentence by sentence, so that we can, e.g., insert spoken prompts.

Which is easier to implement and use?

a. Treat story mode as a variable that storydata.json can assign a new value. +: simple +: concise -: context-dependent because its scope lasts until the next assignment

b. Treat story mode as a nestable modifier to wrap around one or more pages or sentences. +: context-independent insofar as it's not affected by changing the mode of the preceding text -: verbose -: modifier could nest around multiple pages, which might be awkward

[JackMostow]

This is an architectural change, so it involves Kevin, but may be appropriate for Octav to implement.

[JackMostow]

If animator graph for story does a case split on mode, or mode is a feature, storydata.json can signal which words to narrate from which package, e.g. generic prompts vs. story text.

[JackMostow]

Octav - Please implement per-sentence modes. Ulani - As consumer for this feature, please let Octav know which syntax will make your task easier.

[octavpo]

Let me see if I understand the task correctly. So we want to be able to switch between the six modes (HEAR, ECHO, READ, etc), at the sentence level, right? So it seems the simplest thing is to add a 'mode' attribute at the desired levels in storydata.json files. That is sentence, page, and possible whole tutor. Although for the whole tutor there's already a mode in the dev_data.json, so one question is if we get rid of that or still consider it as the default mode. It has the advantage that you could use the same tutor in different modes, I don't know if that's something useful.

[octavpo]

If animator graph for story does a case split on mode, or mode is a feature, storydata.json can signal which words to narrate from which package, e.g. generic prompts vs. story text.

Not sure I understand this comment. Are we saying we also want to communicate the mode as a feature to the graph? And what it would do with it?

[JackMostow]

So I assume, because the animator graph already has mode-specific edges: "edges": [ {"constraint": "STORY_STARTING", "edge": "BEGIN_STORY"}, {"constraint": "ECHO_LINEMODE", "edge": "ECHO_LINE_NODE"}, {"constraint": "PARROT_LINEMODE", "edge": "PARROT_LINE_NODE"}, {"constraint": "STORY_COMPLETE", "edge": "NEXT_SCENE"}, {"constraint": "PAGE_COMPLETE", "edge": "NEXT_PAGE_NODE"}, {"constraint": "PARAGRAPH_COMPLETE", "edge": "NEXT_PARA_NODE"}, {"constraint": "LINE_COMPLETE", "edge": "NEXT_LINE_NODE"}, {"constraint": "", "edge": "NEXT_WORD_NODE"} ]

However, I'm not sure to what if any degree they determine mode-specific behavior. If they do, this seems like the appropriate place to specify it.

[octavpo]

Right I forgot about that. It does need to do things somewhat different because it needs to read different prompts at different times. I've already implemented a 'prompt' feature some time ago, by which the json file could indicate what prompt to use for parrot mode, maybe we want to generalize it? Not sure what's the intended use for other modes.

[JackMostow]

(How) does 'prompt' differ from HIDE mode, i.e. play narrated prompt without displaying its text? - Jack

[octavpo]

I think what you're calling 'prompt' above is the sentence read by the tutor. The prompt feature we have now can be used to customize the introductory sentences like 'Listen carefully' or 'Please listen and repeat after me'. I don't know if any tutor actually uses it.

[octavpo]

This mode is called tutorVariant currently, and it's encoded in dev_data.json with values like 'story.hear', 'story.hide', ... If we still want to use that, maybe we should also name the feature in storydata.json 'tutorVariant' or just 'variant' for consistency, and use the same values. Or else we can reduce all the 'story' variants to just 'story' and have a 'mode' in storydata.json?

[JackMostow]

We already reuse the same story in different modes, and I'd like to be able to continue doing so, so we still need an externally specifiable global mode.

We also need the ability to specify per-sentence modes for various purposes, such as scripted prompts.

If a story in a global mode contains a sentence with its own mode, the sentence-specific mode should override the global mode, because:

1. We need some conflict resolution policy in case it happens.
2. This one is simple.
3. It probably makes more sense than letting the global mode override the local mode.

More generally, any grainsize component should be able to specify its own mode that overrides the mode of the component that it's part of in the overall hierarchy:

• story
• page
• paragraph
• sentence

Let's avoid per-utterance or per-word modes: we don't need them or the complications they'd add.

[octavpo]

With the current structure of the storydata.json files we cannot store any additional data at the paragraph level, as a paragraph is stored just as an array of sentences. We'd need to change it into an object to be able to add more parameters. It's probably not too difficult in terms of code changes, but all json files would need to be regenerated to the new format. Should I do it?

I assume the story level would be what we have now in dev_data.json, we don't want to add another story level setting in storydata.json, right?

[JackMostow]

Then omit paragraph-specific modes -- we can get by with just page- and sentence-specific modes. If you're asking whether storydata.json can specify a story mode instead of what's in dev_data.json, I agree that it should not, so as to avoid confusion. Thanks. - Jack

[octavpo]

I have an implementation of the mode changes, wasn't as easy as I thought. Not on GitHub yet, but tested. A few questions:

• It uses a 'variant' property, let me know if you want to rename it to 'mode' or something else.
• It doesn't repeat prompts, it probably needs to, as old prompts might not be valid anymore, the question is when: at the start of each sentence, or maybe only when mode changes? Repeating between all sentences might be tedious if mode doesn't change.
• Related to that it looks like we might need to also be able to customize prompts at the sentence level. Currently we only have that at the story and page level, should I add that?

[JackMostow]

The property name shouldn't matter. Prompts can be implemented simply as sentences in HEAR mode. Please clarify "repeat prompts" with a concrete example.

[JackMostow]

Just to clarify: mode should have scope nested within the story - page - sentence hierarchy. Thus a sentence in HEAR mode would not affect the mode of any other sentence or page.

[octavpo]

Prompts can be implemented simply as sentences in HEAR mode.

I assume you're thinking about HIDE mode, so they're not displayed. That's true, but they'd be stored in a different place in the assets hierarchy, and we'd to need to repeat them for each tutor. It might be nicer to have them specified as a sentence 'prompt' feature, especially since we already have it for pages, should be easy to extend, and it would be symmetrical with 'variant'.

Please clarify "repeat prompts" with a concrete example.

Let's say a story has two sentences, first in HEAR mode, second in READ mode. Currently the tutor would only play the prompt 'Listen carefully' at the start of the tutor. But that prompt is not valid for the second sentence. So it should play the other prompt 'Please read aloud' (or maybe a custom one) before the second sentence.

On the other hand if both sentences are in HEAR mode it should probably not repeat the prompt before the second sentence.

Just to clarify: mode should have scope nested within the story - page - sentence hierarchy. Thus a sentence in HEAR mode would not affect the mode of any other sentence or page.

Yes that's how I've implemented it.

[JackMostow]

Good catch. Yes, I meant HIDE, not HEAR.

Good point about prompts for mode changes, but:

1. Mode is specified hierarchically, so we need to define mode change carefully. a. A sentence in a different mode than the previous sentence: yes
   b. A sentence in a different mode than the page that contains it: not necessarily if it's the first sentence on the page

2. At which point(s) would you specify the page/sentence prompts for a mode, applicable how broadly? a. In the animator graph, for all text in that mode? b. At the top level of storydata.json, for every page/sentence in that story? c. In the page/sentence, for that page/sentence?

Thanks. - Jack

[octavpo]

Mode is specified hierarchically, so we need to define mode change carefully. a. A sentence in a different mode than the previous sentence: yes b. A sentence in a different mode than the page that contains it: not necessarily if it's the first sentence on the page

Yes we can define the 'effective mode' as being the one that was actually applied for each sentence after combining the three specification levels according to the priority rules. And then repeat prompts only on changes of this effective mode (and of course the first sentence would be always considered an effective mode change).

At which point(s) would you specify the page/sentence prompts for a mode, applicable how broadly? a. In the animator graph, for all text in that mode? b. At the top level of storydata.json, for every page/sentence in that story? c. In the page/sentence, for that page/sentence?

Default prompts are specified and played in the animator graph. Custom prompts are specified in storydata.json and still played by the graph. Specifying prompts in storydata.json would follow the same rules as specifying modes, with specifying them at three levels and playing the one associated with the effective mode. So you could specify modes as you see fit, and then either let the graph play the default prompts for each mode at effective mode change points, or also specify custom prompts for those modes.

And we already have part of the custom prompts implemented. What we don't have now (besides the previous issue of repeating prompts at mode changes) is a few things:

• We don't have prompts at sentence level, only at page and tutor levels, so we'd need to add those.
• The prompts at page level are currently only applied for parrot mode, we'd generalize them to be applied for all modes.
• Custom prompts at each level would only apply for the mode specifying at that level.

[JackMostow]

My intuition about prompts is that we don't yet have good intuitions about them so we should wait until we have actual rather than theoretical cases of needing them. In the meantime let's use HIDE to give any prompts we actually need, and rely on experience to reveal what if anything we really need. Or define PROMPT like HIDE except using the default sound package instead of the story-specific one.

[JackMostow]

1. I think this is a use case for PROMPT. Judith had requested: "Generic solution to show cover page in stories that occludes text and gives spoken-only mode-specific instructions (prevents them from reading before hearing instructions)".

2. Also, to further distinguish modes, can you display their text differently, as follows?

READ: as at present, Roman font in black, green, and red HEAR: same Roman font but blue PARROT: show in blue while reading to kid, then in black to prompt kid to repeat ECHO: show like READ when kid reads, and like HEAR when RoboTutor reads HIDE (and PROMPT): none ;-)

[octavpo]

I've implemented repeating the prompts on mode changes. It wasn't as easy as it seemed first, I had to shuffle things around in the graph to have them play at the right time. However now after testing it I'm wondering if this is the right thing. In general it works fine, but there might be a few issues:

• Some modes use the same prompt (HEAR and HIDE; READ, ECHO, and REPEAT), so if they follow each other the prompt would still be repeated.
• For PARROT mode there are two prompts, one before the reading phase, and another one before the listening phase. So technically the last heard prompt is not valid for the reading phase of the next parrot sentence, so maybe the prompt should be repeated.
• The prompts for successive sentences might be different even with no mode change, if we end up customizing them at the sentence level.

So now I'm wondering if we should rather implement a more adaptive strategy, where the prompt is repeated whenever it's different from the last heard prompt, rather than based on mode changes.

Btw I also found that the second prompt for PARROT('Repeat after me') is actually missing from the assets, can you remind me where I can find these sound files to add it?

[octavpo]

I'm trying to keep different issues in different messages. I've also implemented the different color modes, that was actually easy. One question though: for the reading phases (that is when the tutor reads the sentences), when sentences are supposed to be blue, do you want the whole sentence to be blue from the start, or only the read part of it, so the color would still switch from black to blue as each word is read? Now I've implemented the first choice, but it's easy to switch. I still kept the underlining of the current word, so there's still an indication of which word is spoken, but not as vivid as before, since there's no color change.

[octavpo]

I haven't implemented the PROMPT mode. It's not clear to me it's the right thing to do, for several reasons:

• Now that we change modes between sentences and we need to repeat prompts, we'd have to introduce one of these PROMPT mode sentences each time we need to repeat the prompt.
• Even so we won't be able to introduce it between the two phases of PARROT mode, we'd need to replace it with a HEAR mode followed by a READ mode for the same sentence. And it will have to hide the sentence between the two modes, otherwise kids might still start reading it.
• We'd also need to suppress the regular prompt reading, or we'll get the prompt repeated twice (or possibly two different prompts).
• Not sure why Judith wants to avoid kids starting to read before hearing the instructions. If it's because the audio is lost, in my testing the first word is not lost. The second one is, but it seems to be because of the strict alignment issue that we're planning to fix through different means (issue #348). So we should do that first and then see if it's still an issue.

[JackMostow]

I'll respond here to your comments above rather than edit them to insert my responses, since AFAIK, GitHub doesn't give any other way to insert comments in a previous post.

Your adaptive strategy sounds sensible, simple, and worth trying to see if it works OK in practice.

The Swahili narration of "Repeat after me" is Rudia baada yangu.wav in RT11-POSEIDON Assorted pro 132106.156 03262018.

When RoboTutor starts to reads a sentence, please change the whole sentence from gray to blue instead of to black. Then proceed as at present, i.e. underline each word while reading it and then turn it green.

Judith wants "to show cover page in stories that occludes text and gives spoken-only mode-specific instructions (prevents them from reading before hearing instructions)." Otherwise they won't hear the instructions, which might be to listen rather than to read. So use HIDE mode for the instructions. Can you put them in the animator graph so we don't have to edit every storydata.json?

@judithodili - How's the following for what to say at the start of each type of story? Story type;

• READ: "Please read aloud."
• HEAR: "Please listen to RoboTutor read." OR JUST "Please listen"?
• PARROT: "Please listen and repeat after me."
• ECHO: "Please read aloud, and then listen to RoboTutor read." OR JUST "Please read aloud, and then listen."?
• HIDE: none
• None: none (rely on explicit HIDE for story-specific initial prompt)

[judithodili]

Do we have to be so polite with the “please”? I’ll swing this by Filipo .... we might just have to give the instruction directly without the please so they get it before they tune out.

-

• READ: "Please read aloud."
• HEAR: "Please listen"?
• PARROT: "Repeat after me."
• ECHO: "Please read aloud”
• HIDE: none
• None: none (rely on explicit HIDE for story-specific initial prompt)

--

Regards, Judith Odili Uchidiuno www.judithu.com

[JackMostow]

I'll ask Leonora to record with "Please" (Tafadhali), "Now" (Sasa), and nothing, and get her advice.

[amyogan]

I concur with no 'please' - the prompts are so long the kids get impatient!!

[JackMostow]

In English, "Repeat after me" sounds OK, but "Read aloud." sounds abrupt, "Now read aloud" sounds OK, and "Sasa" is shorter than "Tafadhali".
But we need whatever's shortest without sounding rude in Swahili. I'll ask Leonora when we record today. We can record all 3 versions and check with Filipo if Tanzanian norms of politeness differ from Kenyan norms.

[JackMostow]

@judithodili - What if any additional prompts from new or revised activities need translated and narrated?

[amyogan]

Kids don't get told please there :) But I agree Filipo should be able to confirm!

[JackMostow]

@octavpo and @uhq1 - In case you're not already doing so, please coordinate so that Ulani can use Octav's latest implementation of sentence-specific modes to implement comprehension questions.

Ulani - When can I see each of these demos? (1) generic questions (I hear you already demo'd; is there more to do?) (2) cloze questions (coming soon?) (3) picture-matching questions (aiming for when?)

My priorities for picture-matching are:

1. listening/reading text comprehension: "Touch the picture that shows what you just heard/read." Make sure there's flexibility in terms of whether and when to display the text and the pictures, e.g. first display just the text, then just the pictures, in order to avoid contention for the visual channel.

2. listening/reading word comprehension (aka "vocabulary"): same prompt, but the text is one word, and the pictures are objects like those in the alphabet story and the animals in FaceLogin. Can reuse the very same code, perhaps with 3-4 pictures instead of 2. Practices retrieving meaning of a given printed word. Tests knowledge of the word.

3. Vocabulary: "Touch the word illustrated in the picture." same task in the opposite direction. Requires new code to display a list of words to pick from, but you may be able to reuse code from cloze. Mapping picture to word provides marginal value beyond mapping word to picture. Practices retrieving name of a given object and then recognizing its printed form. Gets more mileage out of the same number of pictures, but allows answering without knowing the illustrated word if the kid can eliminate the others.

4. Vocabulary: "Match the words to the pictures." Requires even more code and a UI/UX used nowhere else in RoboTutor. This activity is popular in schools and printed workbooks, which @judithodili values but I don't. I don't know or see any extra pedagogical value for it unless it's more engaging. It's harder to implement in code than simple multiple choice. From an assessibility standpoint, it conflates knowledge of one word with knowledge of another -- e.g. if you know 2 of the 3 words, you can match the unknown word by process of elimination based on your knowledge of the others.

In other words, I view cloze questions and text comprehension picture-matching as must-haves, word picture-matching as low-hanging fruit, and the others two as optional.

[octavpo]

I've implemented the changes about colors and when to repeat prompts (when the current prompt is different than the last one).

We already have prompts for all these modes, so it's not clear to me why we're trying to create new ones. The current prompts are (at least by file names, no idea what the swahili audio is actually saying):

• READ, ECHO, REVEAL: "Please read aloud"
• HEAR, HIDE: "Listen carefully"
• PARROT: "Please listen and repeat after me" at start, and "Repeat after me" after the reading phase.

We can change them easily (in the program I mean) if you think they're not good, but you guys are talking like that's something new.

[octavpo]

About the idea of reading the prompt before showing the text: I understand what Judith wants, but I don't see how this would work with the new idea that we might change modes from sentence to sentence, so we might need to repeat prompts between successive sentences. If we only do it for the first prompt in the story (or even on a page), fine, but it's inconsistent with how the rest of the prompts would work. If we do it for all prompts, that means each time we want to say a prompt we hide all text, say the prompt, and then show the text back. Is that what you guys want?

[JackMostow]

"Please read aloud" is too long in Swahili. "Read to me" is shorter and more personable.

Listen. | Sikiliza. | HEAR Read to RoboTutor. | Somea RoboTutor. | READ, ECHO

Their Swahili narrations are in RT11-POSEIDON MORE PROMPTS 160726.491 07202018.

HIDE should not have a prompt -- it's used for prompts.

Please clarify '... and "Repeat after me" after the reading phase.' What "reading phase"? "Repeat after me" belongs before RoboTutor says the thing to repeat.

@judithodili - You just want something for the first page, right? The problem is that if the text appears before the prompt, kids jump the gun and start reading it without hearing the instructions? The solution is to play the prompt before showing the text?

@octavpo - I don't understand your issue. Isn't this behavior equivalent to starting each story with "Please read aloud" in HIDE mode?

[judithodili]

‘Read to me’ is very risky. Robotutor is not human and referring to it with human terms may have really negative effects with this population. XPRIZE reported that some people already think it is witchcraft... do you want to risk making this problem worse?

Hearing the instruction before seeing the text would be nice but not worth losing sleep over if it’s too much work. Depending on the verdict on how we assess performance, it might not be an issue even if they start reading early. --

Regards, Judith Odili Uchidiuno www.judithu.com

[amyogan]

I thought we ended up going with 'Read to Robotutor'?

[JackMostow]

Oops, I misdescribed it as "Read to me," but the actual prompt is indeed "Read to RoboTutor." In fact I'm very leery of using the first person to refer to RoboTutor. "Repeat after me" didn't bother me; should it?

If RoboTutor, like Project LISTEN's Reading Tutor, displays each sentence only when it's ready to read -- i.e. hides future sentences rather than showing them in gray -- then HIDE will say the prompt before showing the text. What if any good reason is there to display future text in gray rather than not at all?

[octavpo]

HIDE should not have a prompt -- it's used for prompts.

HIDE was originally designed for a different purpose, see issue #262, that's why it has a prompt, although I'm not convinced the prompt is valid for the original purpose. If you're positive we don't need it anymore for the original purpose I can remove the prompt, but then it might also be clearer to rename it to PROMPT, or just add a PROMPT mode.

[octavpo]

Please clarify '... and "Repeat after me" after the reading phase.' What "reading phase"? "Repeat after me" belongs before RoboTutor says the thing to repeat.

The way PARROT is currently implemented, it's supposed to play two prompts, one before it reads the sentence and another one before it listens to the kid. If you think the first prompt is enough I can remove the second one. Or maybe a different prompt is better at that place? You were suggesting something like "Now YOU say it".

[octavpo]

@octavpo - I don't understand your issue. Isn't this behavior equivalent to starting each story with "Please read aloud" in HIDE mode?

The problem is it's not clear either how it should work or how we want to implement it. We have two ways to play sentences with no visible text, either as prompts from the graph, or as sentences in HIDE mode.

If it's a sentence in HIDE mode, it works like this: you add the text as the first sentence of the story, either on a separate page or on the same page as the rest of the sentences, depending on what image you want kids to see while reading the prompt. Then the next sentence comes, with a different mode, so a different prompt. So the system will play that prompt, so you're getting two prompts in sequence. Besides, you also need to add the audio file to the proper folder, changing the audio source is actually difficult.

If I understand correctly, your comments seem to indicate that's not what you want, because you don't want to add the prompt sentences to each story, and neither the audio to each story folder. So then it's a prompt, that's the main difference between prompts and story sentences, that they're specified in the code and the audios are in a different place. It's also that there's a whole tracking mechanism that's use for sentences that's absent for prompts. I cannot play a sentence in any mode when all the needed parameters are actually not present in the story.

So if we're playing it as a prompt, it seems like what we want different is that it's played before the sentences are displayed rather than after, as it currently is. That can be done, probably relatively easily, but there are still a few issues. One is what about the image? Should it be visible or not?

And the other issue is consistency of behavior. This request was made when we had only one prompt at the start of the story. But with the implementation of mode switching that's not true anymore, we could potentially have a different prompt played before each sentence. So do we implement this new behavior only for the first prompt in the story? Why would the first prompt work differently than the others? If seeing the text while the prompt is played is an issue, it seems like it should be an issue for all of them, so we should rather implement it for all.

If we do that then it will work like this: the first sentence comes, it's prompt is played before the text is displayed (maybe while the image is visible). Then the narration does something with the sentence, depending on mode. Then the next sentence comes. If it's the same mode, no prompt is played and the narration goes on. If it's a different mode, then it's prompt is played while hiding all the text, then it will show the text back and the narration goes on. So each time the mode changes the whole page text is hidden while the prompt is played. Is that what you guys want? Just wanted to make sure you're aware of it before I spend the time implementing it.

I also feel I'm spending too much time on something that has potentially little benefit, I should move to other things, there's a long list.

[JackMostow]

The way PARROT is currently implemented, it's supposed to play two prompts, one before it reads the sentence and another one before it listens to the kid. If you think the first prompt is enough I can remove the second one. Or maybe a different prompt is better at that place? You were suggesting something like "Now YOU say it".

@judithodili and @amyogan -

1. Should we rephrase "Repeat after me" to avoid using the first person, e.g. to "Repeat after RoboTutor"?
2. After reading something to the kid, does PARROT need to say "Now YOU say it" or should it just wait?

[JackMostow]

@octavpo - I agree that this task is getting out of hand in an effort to anticipate all cases and handle them correctly. I'm not sure what to do about it. At minimum, keep the current story-level modes working, and allow sentence-level modes, especially to allow story-specific prompts. If it helps, distinguish these two constructs:

1. HIDE: read the story text to the kid, using sentence narrations from the story folder.
2. PROMPT: tell the kid what to do, using narrated prompts from the prompts folder.

Rather than anticipate all cases, implement whatever seems reasonable, to fix later when and if needed.

@amyogan and @judithodili - Do you concur? Any other advice on how to move on expeditiously?

[octavpo]

The PROMPT mode should be ready now, but it's hard to test because I have no idea where these prompt audio files need to go and how to deliver them to the tablet. I don't see the old ones anywhere in CodeDrop1_Assets. I see them somewhere in RTAsset_Publisher/RTAsset_Audio_SW_Set1/, but then I don't know how to deliver them to the tablet, my understanding now is all deliveries happen from CodeDrop1_Assets. So if somebody has more information that it would be good.

Also since HIDE is not going to be used for prompts anymore, we need a prompt for it. I assume it should be the same as HEAR? Also REVEAL still the same as READ? And I don't see a revised prompt for PARROT, I see a proposal for "Repeat after me", are we set on it?

[JackMostow]

@kevindeland - Please explain where to put audio files for tutor prompts for them to get installed. @octavpo - Please address questions about RoboTutor architecture/infrastructure/installation explicitly to @kevindeland because he might not see them otherwise.

Yes, I suppose the prompt for HIDE should be the same as for HEAR, namely "Listen".

The prompt for REVEAL, if any, should be text-specific, not general.

The revised prompt for PARROT is "Rudia baada ya RoboTutor" ("Repeat after RoboTutor") rather than "Repeat after me," to avoid anthropomorphizing RoboTutor by using the first person in prompts.

[octavpo]

The PROMPT mode should be done now. I've put all changes on branch reading_modes and sent a PR. I also put the new files on RTAsset_Publisher/octav and sent a PR.