An idea for a model of error-carried-forward

[christianp]

Here's an idea for how we could implement error-carried-forward marking (or "method marking") that's been bubbling in my head:

Add a tab to the part editing screen which lets you give a list of variables used in the computation of the correct answer, and replace each of them with the student's answer to another part of the question.

When the student submits the part, the following will happen:

• Their answer will be marked against the correct answer, scoring credit_original amount of credit.
• Make a new JME scope to evaluate the error-carried-forward version of the "correct" answer.
• For each variable in the list of variables to be replaced, look up the corresponding part and see if the student has answered that question (and got less than full marks?). If yes, replace the variable in the new scope with the student's answer to that part.
• Work out the alternate "correct" answer using this new scope, and mark it against the student's answer, to get credit_error_carried_forward amount of credit.
• Optionally apply a percentage penalty to credit_error_carried_forward.
• If credit_original > credit_error_carried_forward, use that and give the same feedback you'd currently get.
• On the other hand, if credit_error_carried_forward > credit_original, add a line to the feedback saying "This part has been marked using your answers to parts X,Y and Z [, and a penalty of P% has been applied]." so that the student knows they're getting error carried forward. (Should this message be customisable?)

  Example

Suppose you're asking the student to do a linear regression on some data to get y=mx+c. Part a asks for the intersect (c) and part b asks for the slope (m). Part c asks for the model's value for y at x = 5.

The correct answers for the parts are as follows:

• a - c = 1 (1 mark)
• b - m = 2 (1 mark)
• c - m*5+c = 11 (1 mark)

  Case 1 - student gets everything right

• Enters 1 for part a, gets 1 mark
• Enters 2 for part b, gets 1 mark
• Enters 11 for part c. This is marked as correct against the original correct answer. Parts a and b both have 100%, so don't bother with trying error-carried-forward. Student gets 1 mark.

Total score: 3 marks.

Case 2 - student makes a mistake in part b

• Enters 1 for part a, gets 1 mark
• Enters 3 for part b, gets 0 marks
• Enters 16 for part c. This is marked as incorrect against the original correct answer. Now look for errors to carry forward. Part a has 100%, so don't bother with it. Part b has 0%, so substitute 3 for the variable m.
• Now, recalculate a correct answer for c - m*5+c = 3*5+1 = 16. The student's answer is marked as correct against this, so they're shown the message "This part has been marked using your answer to part b" in the feedback box. They get 1 mark.

Total score: 2 marks.

Case 3 - student makes a mistake applying their incorrect value

Same as above for parts a and b, but the student swaps m and c and calculates 5*1+3 = 8 as their answer for part c. Neither the original answer nor the error-carried-forward answer match this, so they get 0 marks for c

Total score: 1 mark.

Case 4 - student is inconsistent

• Enters 2 for a, gets 0 marks
• Enters 3 for b, gets 0 marks.
• When working out their answer for part c, somehow uses the correct value for m, but their incorrect value for c, and enters 2*5+2 = 12 as their answer. This matches neither the correct answer nor the error-carried-forward answer, so the student gets 0 marks.

Total score: 0 marks

Things to think about

In case 4, a really charitable marker with access to all the student's working-out might give a mark for the single error carried forward. We've got to hope that students will be consistent.

If the correct answer is 1/x, and the student's value of x is 0, we need to catch the error and conclude that error-carried-forward isn't possible.

If the student answers part c first, using wrong values for m and c, then submits those values to parts a and b, should part c be resubmitted automatically? Should the student be invited to resubmit it? On clicking the submit part button for a, should a popup tell them something like "Your answer to this part was used when marking part c. It will be remarked when you submit this part"?

If the student gets points with error carried forward for part c, then goes back and resubmits the correct values for a, should c be resubmitted automatically? Otherwise, the feedback for part c will be inconsistent with what's shown on the rest of the page.

If we're going for warnings instead of automatic resubmission, what do we do when feedback is turned off?

Suppose the correct answer to the part is defined with just a reference to a single variable ans, which is defined to be 5*m+c in the variables screen. That implies we need to recalculate variable definitions in order to finally get to an updated value for ans. This could be quite powerful - replace some initial data, and go through a lot of calculations involving several variables to work out the correct answers to several subsequent parts. On the other hand, if one of the variable definitions along the way involves a random value, we're stuck! Maybe we should only allow direct substitutions and not recompute the dependencies.

Can you carry errors forward in JME parts? For example, suppose you want the student to differentiate a function y(x), and you want the student to use a substitution u(x). Get them to enter du/dx, and then use that in the answer for y'(x). You'd need to be able to substitute a subexpression into the expression for y'(x).

[christianp]

Just realised this can be used to allow "give an example of X" questions, where the student could give one of many correct answers to an initial part (or it could just be something like "think of a number", where anything is acceptable), and then you ask them to do something with that in the following parts.

A really cool example would be to let the student enter experimental data, then ask them to compute statistics about it.

[christianp]

Edgar Seemann talked at CAA 2015 about his system for allowing error-carried-forward (he models marking algorithms for parts as unit tests, using the student's answers as inputs - basically, verifying the student's answers are consistent). Something that came up was that the student could enter really stupid answers to an early part, so that they can very easily write down a "correct" answer to a later part, which would have been hard to compute with the correct answer. For example: the question asks (a) to compute the stationary point of f(x,y) = a*x^2 + b*xy + c*y^2 + d*x + e*y, and then (b) to evaluate the function at that point. If the question allows error-carried-forward when marking (b), and the student can't answer (a), they can just enter (0,0) as the stationary point and they know immediately that f(x,y) = 0, earning them an easy point. Is it incorrect to use error-carried-forward in this case, or should the author be able to reject trivial answers like this, and allow error-carried-forward for "legitimate" errors?

[christianp]

The algorithm described at the top first tries to mark using the correct answer, then tries to carry errors forward. Should there be an option to demand that the student is consistent with their own errors, i.e. only do the error-carried-forward marking?

[christianp]

I've been working away at this, on the error-carried-forward branch.

Here's how it goes now when the student submits an answer:

• There are two variable replacement strategies: "Try the original marking first", and "always replace variables".
• If the strategy says to, the answer will be marked against the default correct answer, scoring credit_original amount of credit.
• Make a new JME scope to evaluate the error-carried-forward version of the "correct" answer.
• For each variable in the list of variables to be replaced, look up the corresponding part and see if the student has answered that question. If yes, replace the variable in the new scope with the student's answer to that part. (There's an option to insist that the referenced part is answered first, in which case an error message is shown when you try to submit)
• Recalculate the rest of the variables in the question which depend on the replaced variables. The question author has to be careful not to introduce any randomness in the definitions of these dependent variables.
• Work out the alternate "correct" answer using this new scope, and mark it against the student's answer, to get credit_error_carried_forward amount of credit.
• If credit_original > credit_error_carried_forward, use that and give the same feedback you'd currently get.
• On the other hand, if credit_error_carried_forward > credit_original, add a line to the feedback saying "This part has been marked using your answers to previous parts." so that the student knows they're getting error carried forward. (This message is not customisable at the moment)

If part A is used in the marking of part B, and the student changes their answer to part A, they should resubmit part B. They're shown a warning next to the input for part B and it's marked as dirty so they can't leave the question until they resubmit. I decided automatically resubmitting led to too many possibilities for crazy bubbling behaviour or infinite loops.

[christianp]

Now implemented!