1.8.9.1 fix tables of difficulty levels and regenerate data sources

[JackMostow]

1.8.9.1 has a wealth of new data sources, but the harder levels should have more varied items. For instance, the blank should appear in different positions within the same level of the Missing Number task. At present, all the items at a given level have the blank at the same position. Similarly, harder levels of arithmetic problems should include both addition and subtraction within the same level, to help kids learn to choose the right operation to perform in a given problem.

The tables of difficulty levels need to be modified to follow Methodology to Generate Data Sources. Each level should specify the set of values for a difficulty factor, not just a single value.

[judithodili]

I think this is especially relevant for the last activity in each row (skill) on the matrix if we are going use that as a way to promote them to the next skill. So far as the tasks leading up to the final task in the row, mixing the task types can make analysis especially difficult because it is harder to isolate (especially given the number of data sources) why the kids are performing good/bad at the activities.

I think the middle ground for this is keep most of the activities the same, but make sure that the activities leading up to the child's promotion to the next skill is varied in problem types.

This is currently possible to do for Missing Number and ASM I think. It is not possible for something bubble pop if it requires more than one prompt per datasource.... @dptam I know you have moved to Michigan already... but do you possibly have any bandwidth to make bpop accept more than one prompt? It would be fastest for you to do it since you have worked on bpop so much but we can try to find someone else to work on it with your documentation if needed.

[JackMostow]

Can we reduce this problem to the harder one of allowing heterogeneous data sources? I.e. if a data source can specify which activity to use for an individual item, it should be able to specify the prompt too.

[judithodili]

Actually - it is very easy to make these change as long as it doesn't involve making changes to the tutor themselves.

What value does heterogenous data buy us more than selecting 20 random problems from 30, and mixing up the problem types IF the tutor allows it? Again the problem is only in a few datasources. It might even be quicker to do this manually depending on how much work it requires.

On Mon, May 28, 2018 at 9:12 PM JackMostow notifications@github.com wrote:

Can we reduce this problem to the harder one of allowing heterogeneous data sources? I.e. if a data source can specify which activity to use for an individual item, it should be able to specify the prompt too.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/RoboTutorLLC/RoboTutor/issues/306#issuecomment-392626030, or mute the thread https://github.com/notifications/unsubscribe-auth/ACiS-LtjzroMwxX3Xo-B67_XVvzr2GO9ks5t3KBggaJpZM4Tw3xC .

--

Regards, Judith Odili Uchidiuno www.judithu.com

[JackMostow]

Yes, heterogeneous data sources will require the ability to mix item types (i.e. invoke multiple components) within the same activity. As Project LISTEN's Reading Tutor illustrated, this generic capability enables much richer activities, e.g.:

1. Use BubblePop to insert cloze questions during stories with much less implementation effort than the format we prototyped and presented to the Judging Panel but have yet to implement.

2. Mix modes during stories:

   • HIDE: speak a prompt without displaying it
   • HEAR: display and speak the text
   • PARROT: display and speak the text, then listen to the kid repeat it
   • ECHO: display the text, listen to the kid read it, then play the fluent narration
   • READ: display the text, listen to the kid read it
   • REVEAL: listen to the kid read the text, displaying each word after the kid says it

3. Interleave different tasks for the same stimulus, e.g. "5":

   • Visual matching: say and show the stimulus to tap on
   • Recognition: say the stimulus to tap on, to assess whether kids know which answer is correct
   • Recall: show the stimulus for the child to say, to practice responding orally as in EGRA/EGMA
   • Write: say the stimulus for the child to write, to assess whether kids can produce the correct answer

Random selection of items from a larger set to use paired RCTs to evaluate impact of activities is a separate issue Modify Next-item method for randomized and adaptive activity and item selection #234.

Generating data sources for different problem types separately to pool and select from seems kludgey. A cleaner way is to modify the data source generator to generate the heterogeneous data source in the first place, especially for the cited example of varying the position of the blank in Missing Number items, which are really all of the same type. Generating truly heterogeneous data sources, i.e. that use different components, may require implementing separate item generators for each component, and calling them in the appropriate order to generate a heterogenous data source of the desired form.

[judithodili]

I agree that heterogenous data sources will generate richer activities. However, given our experience with trying to make seemingly simple changes to the tutors and all the trouble we faced ... I'm afraid afraid that this is another time sink that requires resources that we don't have. Making the tutors take these data sources will not close any of the learning gaps that we have and as a result, has lower priority than those activities that directly address it. If you have any resource other than Kevin and Octav (because we need them for high priority activity) ... please feel free to give it a whirl. We can have it thoroughly tested before it is integrated into the main code base.

On Tue, May 29, 2018 at 8:32 AM JackMostow notifications@github.com wrote:

Yes, heterogeneous data sources will require the ability to mix item types (i.e. invoke multiple components) within the same activity. As Project LISTEN's Reading Tutor illustrated, this generic capability enables much richer activities, e.g.:

1.

Use BubblePop to insert cloze questions during stories with much less implementation effort than the format we prototyped and presented to the Judging Panel but have yet to implement. 2.

Mix modes during stories:

• HIDE: speak a prompt without displaying it

• HEAR: display and speak the text

• PARROT: display and speak the text, then listen to the kid repeat it

• ECHO: display the text, listen to the kid read it, then play the fluent narration

• READ: display the text, listen to the kid read it

• REVEAL: listen to the kid read the text, displaying each word after the kid says it

  1. Interleave different tasks for the same stimulus, e.g. "5":

• Visual matching: say and show the stimulus to tap on

• Recognition: say the stimulus to tap on, to assess whether kids know which answer is correct

• Recall: show the stimulus for the child to say, to practice responding orally as in EGRA/EGMA

• Write: say the stimulus for the child to write, to assess whether kids can produce the correct answer

Random selection of items from a larger set to use paired RCTs to evaluate impact of activities is a separate issue Modify Next-item method for randomized and adaptive activity and item selection #234 https://github.com/RoboTutorLLC/RoboTutor/issues/234.

Generating data sources for different problem types separately to pool and select from seems kludgey. A cleaner way is to modify the data source generator to generate the heterogeneous data source in the first place, especially for the cited example of varying the position of the blank in Missing Number items, which are really all of the same type. Generating truly heterogeneous data sources, i.e. that use different components, may require implementing separate item generators for each component, and calling them in the appropriate order to generate a heterogenous data source of the desired form.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/RoboTutorLLC/RoboTutor/issues/306#issuecomment-392759232, or mute the thread https://github.com/notifications/unsubscribe-auth/ACiS-A5xmYcxvWjlItD4XFZ39yv1Y0c5ks5t3T_XgaJpZM4Tw3xC .

--

Regards, Judith Odili Uchidiuno www.judithu.com