Multiple Instructors can cause practice to appear unconfigured

[bnmnetp]

A bit of a deep dive into the practice feature due to a spate of problem reports leads to this issue.

The Problem

1. The practice feature creates a row with a null end_date as soon as the instructor brings up the practice tab. -- but has not even looked at the various things to configure.
2. The practice feature assumes there is only a single instructor per course but does nothing to enforce that assumption.
3. If number 2 is not a true assumption then there can be multiple rows in the course_practice table for a single class even if the TA does not click the save button - see number 1.
4. Even worse, there is no consistency in which of the multiple rows will be used because there is no order by clause on the query which seems silly if you assume only one instructor.

I think the single instructor assumption is there so that the same set of conditions will be chosen for an instructor across all of their courses. This makes sense when you are running an experiment but not for production.

Solution

Update the code for practice configuration to ensure there is only one row per course. Enforce that with a unique constraint on the table to prevent duplicates. Allow for multiple instructors to edit the configuration. -- Find out which are the best values to use for spacing and interleaving and use them for all courses.

@presnick @ImanYZ which settings would you recommend?

@barbarer this fully explains the crazy behavior you were seeing in practice last week.

[ImanYZ]

Hi @bnmnetp, We know the spacing incentive is very helpful based on our within-semester analysis, but we're not sure about the interleaving algorithm. Those students under the interleaving condition complained that the tool is asking them questions from basic topics that they already know. On the other hand, the literature is strongly supporting interleaving. So, the only way to make sure whether our implementation of the interleaving helps students or not is to analyze the multi-instructor dataset. I've been working on the Jupyter Notebook that you asked me to create and will send it to you ASAP. My progress is slow because I'm also working on my dissertation proposal. Sorry for that. Best regards, Iman

[bnmnetp]

Thanks @ImanYZ

A practice related question: In the self paced practice mode we only add one question from the page when a student marks the page as finished. But the page could have 10 questions on it. Shouldn't we choose more than one? Also we always use the first question in the list returned from the query wouldn't randomly selecting one (or more) be better?

[ImanYZ]

Hi @bnmnetp

These are very good questions. Because the questions on each page (topic) gradually get more difficult, I thought it may be a good idea not to randomize them. Every time the algorithm decides to ask a question from that page, it should ask another question (different from the one it asked before) until we run out of questions on that page, and restart from the beginning. In our survey responses, most students did not complain about topics with multiple questions, but they did complain about those with only one or two questions. Another issues they complained about was due to the interleaving (spaced repetition algorithm). These algorithms gradually increase the intervals between asking questions from the same topic, unless the student fails to answer questions from each topic correctly, in which case, they reduce the intervals for that topic. This way, topics that are taught earlier in the semester will be asked more times than those taught later. We received multiple complaints about this, and more students preferred not interleaving. This is not surprising because the literature shows that students don't like to interleave, but it significantly boosts their exam scores and long-term learning. The issue is that in our course, we did not observe any significant difference in scores between the group that interleaved and the other group. I think it's possible that for courses like programming or math that topics are built on top of each other, as opposed to language learning, the course content is interleaved enough that it does not require an extra algorithm for interleaving. Maybe for such courses, we only need a spacing incentive mechanism. Again, the only way to make sure whether our implementation of the interleaving helps students or not is to analyze the multi-instructor dataset. @presnick Please let us know whether you're thinking along the same lines, or I'm making a mistake in my interpretation.

@bnmnetp I have been super busy with my dissertation and keep postponing working on Runestone bug reports and the Jupyter Notebook. However, I've been working with good web developers in Pakistan for about $20/hour on our other projects. Would it be possible for you to hire one of these developers to help me with the development of the Runestone practice tool?

[github-actions[bot]]

Stale issue message

[bnmnetp]

Fixed.