1.8.9.1 Eliminate rating menu

[JackMostow]

Rating menu wastes time. Eliminating it will please Judith ;-).

1. Already moved Exit to activity picker.

2. Move Redo to activity picker. Either: a. Disable it until kid completes an activity in the session. b. Keep track of last activity in previous session. c. Don't worry about it.

3. Change "Let RoboTutor pick." Either: a. Eliminate it, especially if seldom used. b. Change it to pick randomly rather than always pick a story. c. Change it to pick whichever content is least completed.

4. Check whether Rating menu triggers any Mentor messages. I don't think it does.

[judithodili]

Can we assign this to @octavpo to work on?

After each activity, the kids need a way to

1. redo an activity
2. go to the next activity
3. go back home/activity picker.

Can we just modify the rating menu page to have these three options? Also having a 'next activity' gives us the opportunity to engineer the variety to activities that a child sees e.g. three literacy activities followed by 2 math activities followed by 1 story etc.... Maybe this is what we change the "Let Robotutor Pick" function to become.

Perhaps the activity picker should be modified so the kids can pick between Literacy, Numeracy, Stories, and Songs ...... but the "Next" button after each activity regardless of content area cycles thru them all to ensure an even spread thru the activities.

[JackMostow]

It's premature to assign this issue to Octav or anyone else until we converge on how to resolve it. Here's the email discussion to date, in chronological order. Please forgive the length. Future updates should be much shorter, and can address posts without quoting them.

Jack, Sat 5/19/2018 1:43 AM:

Epiphany from discussion with Fortunatus: RoboTutor doesn’t need choices to engage Tanzanian kids, because they expect to be told what to do and they typically get love RoboTutor. Let’s get rid of the rating menu but also the content area menu. We just need the Back button to escape a disliked activity, timeout to exit after prolonged inactivity, and perhaps a way to redo a liked activity. RoboTutor can do, say, 3 activities in each content area before switching to the next. Maybe Back should offer options to skip this activity or redo previous activity.

Amy, Sun 5/20/2018 12:11 PM:

Hi Jack, brilliant idea to get rid of the kid ratings! Let's definitely do that.

I think I may agree with getting rid of the content menus, but we need to replace that with another form of kids choice. Fortunatus has good insight about what happens when there's a teacher in the room, but we have data that shows it's different without a teacher and it would be a mistake to get rid of all kid choice.

Happy to brainstorm about how to get a good approach to kid choice into the architecture!

Jack, Tue 5/22/2018 1:19 AM:

Amy - Credit Judith for wanting to eliminate the rating menu.

Let’s make choice optional rather than mandatory as at present. Eg make choice available via the Back button but otherwise proceed to whatever RoboTutor picks next.

Amy, Thu 5/24/2018 5:25 PM:

Great, let's do that! [Eliminate the rating menu.]

So, that's [optional choice via Back] not actually choice (if you recall my analogy of not eating liver and onions versus choosing chicken) :). Rather than do that, I vote to stick with the choice of content areas. However, I think there are MUCH better ways we can introduce simple, small numbers of choices, and I will push hard for doing that instead. Some of the other teams are making some interesting decisions around choice (some more choice and some less), and we can look at some of those, but I think we can do something even better without much work.

Jack, Fri 5/25/2018 10:20 AM:

I should have been more explicit. I mean use Back as the way to whatever menu(s) we decide to offer. The design principles here include:

1. Make it easiest to do whatever RoboTutor picks, without taking time to make choices.

2. Provide easy access to opportunities to choose.

3. Ensure that kids work on what they need to learn (especially reading), not just what they like most, like the girl we saw who picked only math. That’s easier if RoboTutor picks the content area.

4. Strongly prioritize ease of implementation in light of limited resources, the need to address gaps in coverage, and uncertainty about the relative value of choice in the Tanzanian context.

Amy, Fri 5/25/2018 2:55 PM:

Ok, I think I understand better! We can work out a choice policy that's hopefully simple to architect and for kids to use, and that gets them to appropriate content.

Jack, Sat 5/26/2018 2:40 AM at Schipol layover:

Amy et al, adding Nirmal - Can you suggest any choices understandable and meaningful to kids but orthogonal to pedagogical considerations, improve engagement (usage duration, frequency, and longevity, and task persistence), yet are easy to design, implement, kid-test, and evaluate?

Amy, Sat 5/26/2018 4:28 AM:

My first pass would be to take off from Ken's suggestion, and turn the 'bug' that we don't know perfectly what sequence of activities is most effective for learning, and we don't have perfect knowledge of students' abilities, into a feature.

That is, give them a choice between 3 activities that we think are likely to be next in the sequence for them. We can balance them across the content areas - for example, if they keep choosing reading, we can start giving them only choices near each other in the math sequence.

I don't think this would take any major architecture changes because you would keep the sequences we've already developed, but could be tracked with a few variables that persist across activities.

On Sat, May 26, 2018, 8:40 AM Nirmal Patel nirmal@playpowerlabs.com wrote:

Choices remind me of the difficulty choice experiment we did in the Battleship Numberline. It was a 2x2 with (Choice, No Choice) x (Information about content difficulty, No information about content difficulty). We found that providing a choice of content difficulty resulted in an inverted-U relationship between difficulty and engagement. We never provided the choice of content areas, since the game only covered fractions. This makes it more difficult to use the knowledge gained from our experiment, but I'll still describe the results that can be helpful for RoboTutor. They are from our CHI 2017 paper.

The experimental design was like this (we had Robo Pirates in our game... :p)

In each of these conditions, the level difficulty had different effects on player persistence:

And players preferred less to play medium levels (there could be different reasons behind this):

I am guessing that it is difficult to incorporate the choice of difficulty in RT, so I'm ignoring that part. But if you see in the no choice feedforward part of the first plot (bottom left subplot,) informing players (without giving them any choice) that their level was easy or medium resulted in more persistence than telling them that their level was very easy or hard. We did not get a clear inverted-U though, and so Derek writes in his thesis (p.73):

One explanation for the weak effect is that the act of choosing difficulty makes the difficulty more salient and memorable - in contrast, players that were merely informed of their difficulty level could have easily missed the difficulty label. In that case, providing difficulty feedforward with greater salience should result in similar results as the difficulty choice.

So coming back to RoboTutor:

If kids have a perceived difficulty associated with different content areas (which can act as a feedforward) -- like if Susan thinks that math is difficult even though the underlying content is easy -- that might affect her persistence when she chooses math. In that case, telling her that RoboTutor ensures she won't get anything too difficult might help.

Amy, Sat 5/26/2018 4:43 AM:

Interesting! But I agree that we probably don't want to give them a choice of difficulty.

Jack, Sat 5/26/2018 5:13 AM:

1. How could we make activity choices intelligible to kids without lots of design and user testing? The current RoboTutor offers a choice of content area and a choice of difficulty that kids don’t understand as we do.

2. How could we keep choices from being bottlenecks? a. Offer choices only after kid bails out.

b. Proceed to default choice if kid hesitates (3 seconds) without tapping Back.

Jack, Sat 5/26/2018 5:26 AM:

Let’s distinguish 2 cases:

1. Kid finishes an activity. Presumably engaged, so default is for RoboTutor to pick next activity.

2. Kid bails by tapping Back. Disengaged, so offer 2-3 kid-intelligible choices.

Jack, Sat 5/26/2018 5:41 AM:

XPRIZE weights gains in reading 60%, writing 10%, and math 30%. So what mix of activities should RoboTutor pick?

Amy, Sat 5/26/2018 7:36 AM:

I think kids can distinguish among pictures of activities, at the very least. And numbers versus letters.

I would go with always offering 2-3 intelligible choices. Why not?

Jack, Sun 5/27/2018 8:42 AM:

Amy - Certainly worthy of discussion!

1. First let's pin down what we mean by "pictures of activities":

a. Thumbnails of the ~7 basic activity types -- bpop, Akira, story, WRITE, math, countX, number sequence aka CountBy (I refuse to call it NumberScale). +: sufficient to distinguish basic activity types +: only requires ~7 thumbnails, plus any new activity types added +: easy for kids to distinguish +: likely to show the key feature they care about -: insufficient to distinguish different instances of the same activity -: allows kids to always avoid a particular activity, especially math

b. 1/4 scale full screenshots of the actual initial screen +: sufficient to distinguish them except for activities that differ only in whether the stimulus and/or response is spoken or not, i.e. Show vs. Show&Say, or mode, i.e. HEAR/PARROT/ECHO/READ -: requires thousands of screenshots -- one for every cell in every matrix -- and adding more whenever adding new activity types or instances +: should be feasible for kids to distinguish +: likely to show the key feature(s) they care about +: sufficient to distinguish different instances of the same activity -: allows kids to always avoid a particular activity, especially math, unless all 2-3 choices are for the same activity type

c. Other -- if so, what?

2. Time spent picking is not spent doing. But this objection is very weak, especially compared to the current content activity selection menu.

Comments? - Jack

Amy, Sun 5/27/2018 9:29 AM:

Over a 6 month period, the time spent picking between only 2 or 3 options is negligible IF they lead to greater engagement overall.

One hypothesis is that (activity type x content type) is the biggest factor in kids preferences. It would be neat to study that hypothesis as I'm sure we could come up with a few more.

If it's true, though, a solution is to modify Option A to also give an indication of content type (e.g. put a letter or a number beside the BPop icon). And then if kids are e.g. only picking math, track a variable that sets a threshold for how often any of the choices can be a math choice.

Jack, Sun 5/27/2018 1:00 PM:

1. I agree that a single short menu is bound to take less time than the current 2 menus (content area + rating).

2. In which cases should RoboTutor pick the next activity?

a. If we include a "Let RoboTutor pick" option and the kid picks it. +: Making this the first option streamlines the overall interaction for kids so inclined

b. If the kid hesitates [3] seconds without picking.

c. To provide "mandatory" instruction (e.g. addition, subtraction, place value) not provided by any other activity

d. If RoboTutor takes turns picking with the kid, as Project LISTEN's Reading Tutor did. +: allows controlled comparison to evaluate impact of learner control on persistence +: provides opportunity to include mandatory instruction (at the cost of clean experiment design) ?: How much of a RoboTutor-chosen activity must a kid complete to "earn" the next turn? Otherwise kid can game the system by Backing out of RoboTutor-chosen activities. ?: How much of a kid-chosen activity must a kid complete to count as "spending" a turn? Otherwise kid can't pick an activity, quickly regret it, and pick another instead.

e. Unless the kid mastered the activity. +: Reward mastery. -: Gives slower learners less choice. Is this bad?

f. Unless the kid "won" the activity, defined as improving over previous attempt, as Nirmal described. +: Reward improvement.
-: Gives non-improvers less choice. Is this bad?

3. How can we intuitively present the choice of activty type + content area?

a. Show content area icon + screen thumbnail. -: uck

b. Show stimulus and/or response

Gotta go... - Jack

Amy, Sun 5/27/2018 5:25 PM:

Let the kid pick and move forward if there's a.... 5 second hesitation?

Jack, Mon 5/28/2018 11:34 AM:

Amy - En route to Pittsburgh, where I can move this thread to GitHub. Will you follow it there?

Amy, Mon 5/28/2018 2:22 PM:

Probably not while I'm in Africa!

Jack, Mon 5/28/2018 7:11 PM:

If you’re subscribed (or assigned) to a GitHub issue, it will email you just the updates plus a link to the issue without quoting all prior posts, and if you reply by email I think it will post your reply to the issue.

At any rate I was going to point out that heterogeneous activities may be harder to describe understandably to kids, so we’ll have to settle for describing just the first step unless someone suggests a better way.

Besides an intuitive display, what prompt should accompany each choice? a. None, just “tap what you want to do next”. +: easiest

b. The generic task prompt for the first item, e.g. “tap on the letter” or “repeat after me”. +: makes the task explicit, which the visual-only item does not

c. A prompt crafted for the purpose, e.g. “tap here to tap letters” +: states the task, e.g. “copy words”. -: many prompts to craft!

d. Better idea?

[JackMostow]

Amy, Tuesday, May 29, 2018 3:21 PM:

I think you could go with the easiest thing - that reduces complication, it's not super important what they choose if the options are all reasonable, and they'll figure it out after a few times

Jack:
a. Single prompt “tap what you want to do next” and one screenshot for each activity type. +: easiest -: doesn't indicate content area -: menu may contain identical screenshots

e. Like a, but a distinct screenshot for each content area. +: almost as easy +: indicates content area -: menu may contain identical screenshots, though less often

[JackMostow]

Upshot of discussion at 2018-07-11 RoboTutor strategy meeting:

1. keep Redo with current icon and prompt +: retain option kids used frequently +: increase perceived learner control even if overlaps with existing option -: meaningless the first time

2. keep Stop using RoboTutor

3. drop Let RoboTutor decide +: eliminate work for an option kids chose rarely (5-10% in Dec. 2017)

4. rephrase "Tap here for " as "For tap here" +: address kids tapping without finding out what option will do

5. Refer to ___ as at present (reading & writing, stories, or numbers & math) +: avoid having to name the activities -: don't worry if they overlap

Default: current activity in each matrix

If a content area is under-represented in frequency of previous choices, replace the over-represented area with a second option in that content area.

[JackMostow]

Also: after reciting menu 3x, exit RoboTutor.

[amyogan]

Not sure what the last one means, about exiting RoboTutor, but the rest I agree with!

[JackMostow]

Exit = stop using RoboTutor.

Let's operationalize "over-represented" ("under-represented") as any content area(s) where the kid is above (or below) their average level in the other content areas. Omit the over-represented area(s) and offer more activities in the under-represented area.

Rationale:

1. Aim to complete all 3 content areas around the same time. So if a kid has completed 1/3 of reading, 1/2 of stories, and 2/3 of math, offer two reading activities and one story activity.

2. Normally include one activity from each content area. So just use the (coarse-grained) level rather than the (fine-grained) fractional position within the level.

3. Don't try to over-optimize by matching the time spent to XPRIZE's weighting of 60% reading, 10% writing, and 30% numeracy. Instead, assume the curriculum already reflects those weightings.

[kevindeland]

Resolved in a previous commit.