Correct submissions attract penalties

[alcarola]

Hi,

I just did an electronic exam with STACK version 2020022400. I am puzzled by the rather seriuos fact that it seems like correct submissions attracted penalties.

A student first answers a question only partly correctly; STACK correctly awards 4 marks. Then the student answers the question completely correctly and the PRT grades the second answer as correct. Nevertheless, STACK outputs "Your answer is partially correct", gives 0 marks and imposes a new penalty:

Marks for this submission: 0.00/20.00. Accounting for previous tries, this gives 4.00/20.00. This submission attracted a penalty of 4.00. Total penalties so far: 8.00.

Another student had a similar score bookkeeping issue, and curiously, they both had the same random instance of the question. However, the automatic question tests reveal no issue with this random variant.

Edit: I forgot to mention that the students' answers were correctly graded initially, but then when I edited another question, not related to this bug report, and did an automatic regrade (without editing this question), then the automatic grading dropped the marks for the two students.

Edit 2: I also emptied the STACK cache and tried using the optimized Maxima image directly, without the Maxima pool that I normally use, but neither action made any difference.

Thankful greetings, Mikael

[shinuito]

@alcarola Is it possible that the Stack Question itself has number other than 0 in the Question penalties box?

I'm still getting used to moodle questions and quizzes myself, but I'm imagining regrading a quiz as counting as a separate attempt for some reason.

https://github.com/maths/moodle-qtype_stack/blob/master/doc/en/Authoring/Options.md

This is the percentage of the marks deducted from each different and valid attempt which is not completely correct, when the penalty mark modification scheme is in use.

I presume it didn't affect other students with not completely correct submission? It could be something in the partial response tree either then for the inputs that the two students got incorrect?

Maybe someone who know more about this can add their input.

[alcarola]

@shinuito, thank you very much for your help!

Since the students were initially correctly graded, and then graded incorrectly without me changing the PRTs, and moreover the other students were correctly graded, I think that this is a bug. I wonder if it is related to the question state (which I unfortunately don't know anything about), as were #552 and #531 which depict slightly similar behaviour.

I find this bug very disturbing, and I would like to track it down, since we are now several teachers that use stack for electronic exams in large courses at my university, and some students could easily lose exam points without anyone noticing.

I realize that my question as posed is a bit vague, but I hope that it is enough to determine what debug information could be of use. Since it is an exam question, I do not want to post unnecessary detail here. I can e-mail additional troubleshooting detail on request.

[alcarola]

No interest in this issue? Am I the only one who finds this alarming?

Now this happened again: I had a stack question in a quiz. Students took the quiz, and after the deadline, without changing any questions, I did a regrade test. A few of the students got reduced points in the regrading, while the marks of the vast majority were unchanged. This time the students who got reduced marks did not have the same random variant of the question.

This time it was not an exam question, so I attach the zipped xml here: quiz-DiffEkv 2020-Matrix 2x2, char. poly., eigenval., eigenvect. SV-20201008-1745.xml.zip

[sangwinc]

I am interested in this issue, and your question will help us track it down. It will be difficult, as this is at the interface between the quiz and the question. STACK is basically "stateless", so we also need to look at what is going on inside the Moodle quiz I think.

[alcarola]

Thanks for the interest! I now found yet one more question, for which the same problem has occured: regrading without changing the question lowers the marks from the grading carried out before/on the deadline. I attach it below.

I investigated the cases a little closer and found that:

1. One student appears in all three incorrectly-behaving questions. The other students differ between the questions.
2. In the response history / PRT traversal, for those getting lowered marks on regrade, there appear error messages, such as ATAlgEquiv_SA_not_logic and ATAlgEquiv_SA_not_matrix. I did not spot anything suspicious about the associated answers, see the attached screenshots.

quiz-DiffEkv 2020-ODE, constant coeff, non hom. 3 (order 3) SV-20201008-2203.xml.zip

This is from the question Matrix 2x2:

This is from the question ODE, constant coeff:

[alcarola]

An update on this issue from this year's teaching:

1. I now did an automatic regrade of the question Matrix 2x2 mentioned above, in last year's course. Now the student gets 0 points, even if the answers are correct. If I copy-paste the student's answers into the question preview, they all receive full score. I did not change the grading logic. Clearing the STACK cache, increasing the STACK timeouts to 30 seconds and/or turning off the maximapool (switching to pre-optimized Maxima) had no effect. See the screenshot below.
2. I also did an automatic regrade of the same quiz in this year's version of the course. Now the question Matrix 2x2 grades all answers correctly, but another question in the same quiz gives 0 points for a correct answer, which correctly received 1.7 points before the regrade. I tried regrading also other quizzes in this year's version of the course, and also here questions that graded incorrectly last year graded correctly this year, while some questions which graded correctly last year grade incorrectly in this year's version of the course.

I still find this phenomenon seriously disturbing, but I am still clueless how to efficiently troubleshoot the issue. The version of STACK that I presently use is 2020120600. Any input?

Edit: In all cases I have seen so far, it has been only one question per quiz that grades answers incorrectly. This one question frequently grades several students incorrectly. (While the vast majority of students are graded correctly.)

Screenshot of grading of the question "Matrix 2x2..." ("Rätt svar" is Swedish for "correct answer".):

[lassikor]

We have noticed this same issue in several cases. There are some things in common in our cases when this issue appears: -Automatic regrade is used -Question behaviour is "Adaptive mode" -Answer type is Matrix -There are more attempts (Check button is used) than one.

In the following example student's answer was initially graded to 1.00 but regrade dropped score to 0.75. However, the feedback from the trees reveals that everything was fine. And as we can see from the screenshots below, there appears an error prt1: [RUNTIME_FV_ERROR] # = 0 | prt1-1-F | prt1-2-F; that wasn't there before regrade. I hope that issue will be taken seriously, because the regrade function is useless at the moment.

BR Lassi Korhonen

[alcarola]

Thank you for your support in getting rid of this problem, Lassi! I use regrade after essentially every exam, as I tune the grading logic of the questions after the deadline.

I just confirm that I also use adaptive mode in my quizzes, but I observe the problem for answers not of matrix type.

[alcarola]

I did not observe any instances of this problem in this year's version of the course. I don't know if the problem has been solved, or if it did just not show up.