DISCUSS comprehension questions

[JackMostow]

We already have specific issues for cloze, picture-matching, and generic wh- questions. This issue is for discussion that cuts across them and other types of questions to add, if any.

Let's start with a reality check on comprehension questions, in terms of how and why we will use them:

1. A (perhaps the) main reason is to prepare kids for the EGRA by giving them practice in answering comprehension questions.
2. Another reason to insert comprehension checks is as a forcing function simply to get kids to pay attention to story meaning while they read.
3. A related reason is to scaffold comprehension by triggering comprehension processes such as constructing situation models, drawing inferences, and thinking about context.
4. Some questions – e.g. open-ended generic wh- questions – are to assist comprehension without assessing it.
5. The wh- questions are also useful practice for EGRA insofar as they resemble EGRA questions, e.g. “Who is this story about?”.
6. Unlike school settings, we are not generating assessments for parents, teachers, or report cards – only for tutorial decisions about what to do next.
7. Should we make kids keep rereading each story until they answer some percentage of comprehension questions?
8. Another use of comprehension assessments is to adjust the level of difficulty of the comprehension questions, e.g. by picking easier or harder distractors in cloze questions.

[JackMostow]

@kevindeland - How should we implement comprehension questions?

a. Extend READ.

+: Allow tight integration of questions with text as illustrated in the video prototype of cloze and picture matching text comprehension questions, at 2017-05-24 fpsedit-new comprehension questions prototype.mp4.

-: Risk of breaking READ.

-: Applies only to READ, not to other components.

b. Allow heterogeneous data sources so as to interleave READ for each page with a separate ASK step.

+: Avoid breaking READ.

+: General solution applies to other components too, e.g. alternating spoken, written, and/or multiple choice responses to the same question so as to combine practice on EGRA/EGMA spoken responses with other modalities that we can assess automatically.

c. "Have our cake and eat it too": Mimic option a by replicating the READ page display in a separate component that presents multiple choice questions but does not listen to or assist oral reading itself.

+: Avoid breaking READ

+: Allow tight integration of questions with text

-: Extra work to replicate READ's text display -- but very little

-: Question-asking components won't listen -- but they don't need to. (I'll post separately about free-form spoken responses at https://github.com/RoboTutorLLC/RoboTutor/issues/92.)

[judithodili]

Why did you assign me to to this?

[JackMostow]

One lesson from our discussion of what to learn from code drop 1 was to discuss issues via GitHub rather than by email, so I added this issue to continue yesterday's discussion of comprehension. I included you so that you'd see the posts, i.e. to subscribe you to the discussion, not to assign you the task of adding comprehension activities.

1. Are you still interested in the discussion?

2. Who else from the meeting should be added to this thread?

[JackMostow]

At Saturday, June 02, 2018 3:59 PM, Judith wrote: "You seemed to know exactly what listening comprehension should look like and how it should be built in stories ... Since you volunteered to find the resources for it I expected the UI/UX specs from you (or for you to work with Shiven or whoever to get that developed)."

I'm not sure what form you want them in, but here's what we have:

For documentation of comprehension questions, please see COMPREHENSION.

For a video prototype of cloze and picture matching text comprehension questions, see 2017-05-24 fpsedit-new comprehension questions prototype.mp4.

Please let me know what else you need. Thanks! - Jack

[JackMostow]

@judithodili Some thoughts from our discussion today:

1. EGRA tests kids' comprehension of their unassisted oral reading by asking questions to answer aloud. a. It would be useful to give them practice in doing so in order to prepare them for EGRA. b. Generic wh- questions (e.g. "who is this story about?") are good practice and presumably teach kids to ask those questions on their own, which improves their comprehension. c. Text-specific questions don't apply to other text, but are sometimes needed for testing inference. c. Question-answering is useful practice even if unassessed. d. Recording session audio will let us assess spoken answers off-line if warranted.

2. Oral reading and listening are inseparable from comprehension, so they must be practiced together. a. RoboTutor needs to ask comprehension questions during oral reading to or by the child. b. Comprehension questions should be interleaved with reading without jeopardizing it or each other. c. The same types of comprehension questions can be used during listening, assisted oral reading, and unassisted oral reading.

[JackMostow]

Great meeting with Ulani and Kevin today! Upshots:

1. Ulani will join Judith's dev team starting this Thursday, 6/28.

2. Implementing questions by copying and extending the animator graph for READ looks like a good bet, eliminating the main reason for interleaving. Ulani will check what's involved before we decide for sure.

3. Ulani will implement a "custom" story mode that specifies per-sentence modes, which will help in implementing questions (e.g. use Hide mode to insert generic questions) and be useful to others (e.g. to insert spoken-only prompts on the cover page as Judith requested).

[judithodili]

Ulani is done with the generic questions and currently working on the cloze questions. Her biggest issue now is that cutting off the audio on the "question" sentence is problematic because 1. the seg timings are not always accurate 2. and it is breaking other things... She is still trying to figure out a solution for this... she plans to be done with all three question types by 8/16.

[JackMostow]

1. How far off is the worst-case start time of the omitted word, and in which direction -- too early or late?
2. What else is breaking?

[JackMostow]

@uhq and @judithodili -

1. Which types of comprehension questions can you point me to demos of, and when?
2. What work to complete them remains, and what help would be most useful?

[judithodili]

I’ll let you know when demos are ready for you. --

Regards, Judith Odili Uchidiuno www.judithu.com

[JackMostow]

@judithodili, @uhq1, @ealanhill, @kevindeland, and @amyogan -

Listening and text comprehension questions are must-haves explicitly tested by EGRA. They're top priority, in contrast to the word-to-picture matching task, which as I pointed out weeks ago could be implemented as a trivial special case of matching text to picture by making the text of each page be a word to match.

It's now already a week past the 8/16 target date to finish all three question types, with no ETA announced. So let's take advantage of @ealanhill's extensive Android app development experience to speed things up.

Please point Alan (and me) to the specs, existing source code, current sticking points, and remaining goals, so that together you can figure out how to finish all three question types by August 31 if at all possible.

I'd like to see them at least in demo form when I'm back on August 29, along with a timetable to finish them.

Thanks! - Jack

[judithodili]

Jack, I told you I will send a demo to you because Ulani was traveling home and I hadn't heard from her on the progress of the tasks.... It is completely unnecessary to bring up the priority of the tasks - I would have raised an alarm if there were any true concerns.

We have target dates on all the different activities and while we shoot for them, we have to be accommodating of the students running into bugs or even travel plans ... most students travelled in the last 2 weeks to transition from the summer to the fall semester and we have to be understanding of that.

Ulani is currently working on the demos to send them ... I'll send them to you before the end of the week.

[uhq1]

Hi Jack,

I've sent you and others an email with the demos, sorry for the wait and thank you for your patience. The demos for the cloze and picture matching tasks are attached below as google drive links for anyone else who would like to see them.

https://drive.google.com/open?id=1A9baB9krT96ZvsOnyqHid-g92BAmSOEk https://drive.google.com/open?id=1tgKBhu964XnYI280nEjl3tnbH11lbmH3

[JackMostow]

Demos received, enjoyed, and responded to by email with request to post and answer my comments and questions in the GitHub issues for the 3 individual types of comprehension questions (generic, picture, cloze).

BTW, standing request: please put demos and anything else in logical locations under XPRIZE - Home so we can find them again, rather than in links to private folders.

Thanks. - Jack

[uhq1]

@JackMostow I don't have file adding permissions directly to XPRIZE - Home, but I was able to add the demo videos under XPRIZE-Home -> Team can edit -> COMP QUESTIONS DEMO

[kevindeland]

Resolved.