Q0004_AnSwEr values disappear from form, even with evidence that the student did type in answers

[Alex-Jordan]

I'm posting this, aware that maybe it has already been addressed in develop. I think @drgrice1 will know.

There have been reports in the forum of answers disappearing. I think only with quizzes, but of course it's more likely with quizzes that someone would bother with a report. The description is something like answers were put into answer blanks, and the user hits submit. But the answers are all removed and the student gets a zero for the quiz. The student swears they had correct answers there, but they are just gone.

This happened with a server I manage, and I took a closer look at the transaction log. Below I paste the parts of the log that matter. This is one student's information (anonymized). It's a four-question quiz. They took it one time, got the zero. Later the instructor reassigned it, they retook it, and got 4/4.

The time they got 0/4, there is data like:

"Q0004_AnSwEr0001","","MaThQuIlL_Q0004_AnSwEr0001","73"

For this question, the correct answer was 73. So the MaThQuIlL_Q0004_AnSwEr0001 form field had the student's submitted answer, but the regular Q0004_AnSwEr0001 field was empty. During the second attempt, we have:

"Q0004_AnSwEr0001","73","MaThQuIlL_Q0004_AnSwEr0001","73"

The 73 is in both places, and the student got credit.

I certainly don't know how to explain this. But it makes me wonder if MathQuill failed (in the first instance) to put the student's answer into the Q0004_AnSwEr0001 input of the field when the submission happened. Could that be possible? (And if so has this already been addressed in develop?)

Here are the lines from the log:

[Thu Feb 18 02:29:47 2021] 1    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_1.pg   1   -1  3519    0   1   ["Q0001_AnSwEr0001","","MaThQuIlL_Q0001_AnSwEr0001","2","Q0001_AnSwEr0002","","MaThQuIlL_Q0001_AnSwEr0002","4","Q0001_AnSwEr0003","","MaThQuIlL_Q0001_AnSwEr0003","5","Q0001_AnSwEr0004","","MaThQuIlL_Q0001_AnSwEr0004","4"]   0   1
[Thu Feb 18 02:29:47 2021] 2    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_4.pg   1   -1  2280    0   1   ["Q0002_AnSwEr0001","B","MaThQuIlL_Q0002_AnSwEr0001","","Q0002_AnSwEr0002","I","MaThQuIlL_Q0002_AnSwEr0002","","Q0002_AnSwEr0003","N","MaThQuIlL_Q0002_AnSwEr0003",""]  0   1
[Thu Feb 18 02:29:47 2021] 3    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_6.pg   1   -1  1682    0   1   ["Q0003_AnSwEr0001","","MaThQuIlL_Q0003_AnSwEr0001","16","Q0003_AnSwEr0002","","MaThQuIlL_Q0003_AnSwEr0002","0","Q0003_AnSwEr0003","","MaThQuIlL_Q0003_AnSwEr0003","8","Q0003_AnSwEr0004","","MaThQuIlL_Q0003_AnSwEr0004","0"]  0   1
[Thu Feb 18 02:29:47 2021] 4    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_10.pg  1   -1  3901    0   1   ["Q0004_AnSwEr0001","","MaThQuIlL_Q0004_AnSwEr0001","73","Q0004_AnSwEr0002","","MaThQuIlL_Q0004_AnSwEr0002","74","Q0004_AnSwEr0003","","MaThQuIlL_Q0004_AnSwEr0003","84","Q0004_AnSwEr0004","","MaThQuIlL_Q0004_AnSwEr0004","214"]  0   1

[Thu Feb 18 16:57:59 2021] 1    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_1.pg   1   -1  3519    1   1   ["Q0001_AnSwEr0001","2","MaThQuIlL_Q0001_AnSwEr0001","2","Q0001_AnSwEr0002","4","MaThQuIlL_Q0001_AnSwEr0002","4","Q0001_AnSwEr0003","5","MaThQuIlL_Q0001_AnSwEr0003","5","Q0001_AnSwEr0004","4","MaThQuIlL_Q0001_AnSwEr0004","4"]   1   1
[Thu Feb 18 16:57:59 2021] 2    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_4.pg   1   -1  2280    1   1   ["Q0002_AnSwEr0001","I","MaThQuIlL_Q0002_AnSwEr0001","","Q0002_AnSwEr0002","I","MaThQuIlL_Q0002_AnSwEr0002","","Q0002_AnSwEr0003","N","MaThQuIlL_Q0002_AnSwEr0003",""]  1   1
[Thu Feb 18 16:57:59 2021] 3    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_6.pg   1   -1  1682    1   1   ["Q0003_AnSwEr0001","16","MaThQuIlL_Q0003_AnSwEr0001","16","Q0003_AnSwEr0002","0","MaThQuIlL_Q0003_AnSwEr0002","0","Q0003_AnSwEr0003","12","MaThQuIlL_Q0003_AnSwEr0003","12","Q0003_AnSwEr0004","0","MaThQuIlL_Q0003_AnSwEr0004","0"]   1   1
[Thu Feb 18 16:57:59 2021] 4    0_4-ReadingQuiz USERNAME    DMOI/DMOI/0-Introduction_and_Preliminaries/0_4-Functions/0_4_10.pg  1   -1  3901    1   1   ["Q0004_AnSwEr0001","73","MaThQuIlL_Q0004_AnSwEr0001","73","Q0004_AnSwEr0002","74","MaThQuIlL_Q0004_AnSwEr0002","74","Q0004_AnSwEr0003","84","MaThQuIlL_Q0004_AnSwEr0003","84","Q0004_AnSwEr0004","214","MaThQuIlL_Q0004_AnSwEr0004","214"] 1   1

[Alex-Jordan]

I was wrong to say the instructor "reassigned" the quiz. Judging by the random seeds, both instances were the same version of the quiz. The instructor has this quiz set to permit unlimited graded submissions.

But the student clearly tried to submit mostly correct answers at 2:30 in the morning and got 0/4. Then submitted mostly the same answers (with a few changes) again at 5:00 in the evening at got 4/4.

[drgrice1]

I have not seen this occur, and I don't know why it would happen. There is nothing in develop that would address this issue. Clearly something occurred that prevented MathQuill from putting the students answer into the original input, but I don't know how that would happen.

[Alex-Jordan]

This will be frustrating to understand better. The reports of this happening have never indicated that they can reproduce it. I would like to see the console log from an instance when it happened, but good luck getting that from a student, well after the zero was scored.

I confess I have been tempted to write off the reports as user error on the part of the student. But if nothing else, now we can advise people to look in the transaction logs and there is a chance to recover lost answers.

[drgrice1]

It would be useful to see what is in the console, and certainly we would need to be able to reproduce this in order to fix it.

[drgrice1]

One thing that is odd here is that both of the inputs are set at the same time in the javascript. So it really doesn't make sense that the input with id MaThQuIlL_Q0001_AnSwEr0001 would be set and the input with id Q0001_AnSwEr0001 not set.

[Alex-Jordan]

One possibility is that it does successfully set them, but then something else clears them.

I found another clue. In the login log, I find this student timed out for inactivity a few seconds before they got the 0/4 score. Then then logged with a password one second before the submission. Then made the submission:

[Thu Feb 18 02:29:41 2021] LOGIN FAILED inactivity timeout user_id=USERNAME login_type=normal credential_source=params host=136.244.148.252 port=52347 UA=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
[Thu Feb 18 02:29:46 2021] AUTH WWDB: password accepted user_id=USERNAME login_type=normal credential_source=params host=136.244.148.252 port=52355 UA=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15
[Thu Feb 18 02:29:46 2021] LOGIN OK user_id=USERNAME login_type=normal credential_source=params host=136.244.148.252 port=52355 UA=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15

(The 0/4 submission in the transaction log is at 02:29:47.)

That tells a story that they had answers in answer blanks. Then hit submit, but were rejected for an inactivity logout. Entered password to get back in. Hit submit again, and we see that one in the transaction log. But no real answers, just the MathQuill ones.

So maybe when the entered their password and the screen returned to the quiz, it populated some input forms (like MaThQuIlL_Q0001_AnSwEr answers) but not all (like Q0001_AnSwEr).

Or maybe when they enter the password, that automatically tries to complete what I described as the "first" submission, and it just got logged a second later. But something about the re-logging in dropped the Q0001_AnSwEr type input values.

This would mean it's not MathQuill to blame, but a design flaw with what happens when someone times out for inactivity during a quiz. (The instructor has no time limit on this quiz, so this is more likely to happen than with a timed quiz.)

[Alex-Jordan]

I will try to mock up this situation now to see if it is repeatable.

[Alex-Jordan]

No luck. I repeated the scenario such that I submitted answers after timing out, then had to enter my password to log back in, and then the answers were all accepted. There was no need to push submit a second time. And the logs do indicate one second in between when I logged back in and the answers were written to the transaction log. (But for me, they were all present.)

[dlglin]

Did you try with a failed login in between?

On Feb 22, 2021, at 3:42 PM, Alex Jordan notifications@github.com<mailto:notifications@github.com> wrote:

[△EXTERNAL]

No luck. I repeated the scenario such that I submitted answers after timing out, then had to enter my password to log back in, and then the answers were all accepted. There was no need to push submit a second time. And the logs do indicate one second in between when I logged back in and the answers were written to the transaction log. (But for me, they were all present.)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/openwebwork/webwork2/issues/1208#issuecomment-783729126, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AANWEGVLYNS4D3RQOKZIQOTTALMV7ANCNFSM4YBDXXJA.

[Alex-Jordan]

@dlglin I did not, but the student in the original case here did not have a failed login in between.

My test was on the PCC server, on a flavor of develop. The incident was on a 2.15 server. I'm about to try it on the same server, same quiz, same course config..(except I'll cut the inactivity time to like 30 seconds). I'll try it with a failed login too, just to see what can be learned. It would be good to know if the data persists through a failed login attempt.

There is the larger issue of holding on to quiz answers in local storage when something goes wrong. Some student may not even have passwords, and need to go in through the LMS.

[Alex-Jordan]

Still no luck. Same server, same configuration, same quiz settings. Timed out for inactivity and then submitted the quiz to grade. Had to enter password, so I did, and immediately am told I got 4/4.

Tried again, resetting the quiz. This time using an incorrect password once when I have to log back in. But then after using the correct one on the next try, I get in and am told I got 4/4.

So I'm back to having no ideas. Other than speculation that it has something to do with peculiarities of the student's web browser.

[drgrice1]

@taniwallach: Please don't just add priority 1 and requires attention to issues like this. I have been using MathQuill for a long time, and have never seen this issue. Until this can be reproduced there isn't much that can we can do.

[Alex-Jordan]

At this point I think it is more likely that something removed the input values for the AnSwEr keys during a timeout-log-in-again situation rather than MQ failed to insert them. And it just so happened that because MQ was being used, there was extra interesting stuff to look at in the transaction log. I'll rename the issue to not necessarily distract people into looking into MQ.

[somiaj]

edit My issue might be a different issue than the original one reported. I can move this to a new issue if needed.

I can confirm that I have had students when working with gateway quizzes (I too use MathQuill) have this issue, where all of their answers will go blank (I use one problem per page, unsure if that has anything to do with the issue), but I can still see their score and past answers. In my last gateway quiz I gave, I had 5/79 students answers go blank while taking the quiz. Here is what I see looking at the past answers of one of these students.

Timestamp	Scores	Answers	Comment
02/22/2021 at 09:06am MST		[newPage] No answer entered
02/22/2021 at 09:08am MST	0,0,0	[submit] No answer entered
02/22/2021 at 09:08am MST		[newPage] No answer entered
02/22/2021 at 09:09am MST		[newPage] No answer entered
02/22/2021 at 09:09am MST	0,0,0	[newPage] No answer entered
02/22/2021 at 09:09am MST		[newPage] No answer entered
02/22/2021 at 09:10am MST		[newPage] No answer entered
02/22/2021 at 09:10am MST		[newPage] No answer entered
02/22/2021 at 09:10am MST		[newPage] No answer entered
02/22/2021 at 09:14am MST		[newPage] No answer entered
02/22/2021 at 09:22am MST	0,0,0	[submit] No answer entered
02/22/2021 at 09:22am MST		[newPage] No answer entered
02/22/2021 at 09:22am MST	0,0,0	[submit] No answer entered
02/22/2021 at 09:22am MST	0,0,0	[newPage] No answer entered
02/22/2021 at 09:36am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 09:39am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 09:45am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 09:46am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 09:46am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 09:50am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 09:53am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:01am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:01am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:01am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:01am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:07am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:09am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:10am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:10am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:10am MST	1,1,1	[newPage] 2 -4 1
02/22/2021 at 10:10am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:10am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:18am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:19am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:19am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:19am MST	1,1,1	[newPage] 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST	1,1,1	[submit] 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:19am MST		[newPage] 2 -4 1 2 -4 1
02/22/2021 at 10:21am MST		[newPage] No answer entered
02/22/2021 at 10:21am MST		[newPage] No answer entered
02/22/2021 at 10:21am MST		[newPage] No answer entered
02/22/2021 at 10:21am MST	0,0,0	[newPage] No answer entered

This question has three parts, and the answers are 2, -4, and 1. I do notice one oddity I don't understand, that when moving to a [newPage], the answers are duplicated in the log, and it appears that somewhere in switching between pages, all answers were lost.

I did notice that in each case the answers were blanked in my case, it was always due to hitting the quiz time. In this case the quiz was due at 10:20, and that is the time the answers vanish. I wonder if it is some issue with the grace time or how quizzes deal with answers once the time limit is hit.

I have no clue how to reproduce the issue, I can just confirm that it happens with a very small subset of my students. I am using the develop version now, but also noticed this in with the version from the master branch before I updated last semester (so don't think it is an issue with just develop).

[Alex-Jordan]

Are your quiz questions in sequential order? As in, if you viist the quiz in the Homework Sets Editor, does it have something like 1,2,4,5 as the problem numbering?

There was a bug with quizzes where problems are not in sequential order. Work by @drgrice1 fixed that bug, but it is part of the upcoming 2.16 version of WeBWorK. If this is the issue you are experiencing, then forcing the numbering to be in order should fix things for future quiz versions. I'm not sure if there is an easy fix for quiz versions that were already started.

And of course it's possible this is not the same issue.

[maxlemieux]

Yes, the problems in the set were in sequential order.

I wasn't sure if this was even the right issue to post in (issue was disappearing answers/answers given on previous attempts being wrongly submitted instead of most recent ones) and had removed my comment. Thanks for your response. Let me know if I should open a new issue if I can reproduce it... As a new student it's not clear to me which problems are related to hosting/implementation and which are part of the core WeBWorK software, and I don't want to muddy the waters unnecessarily.

[somiaj]

My problems were in sequential order when some of my students answers were blanked at at the end. I'm unsure if my issue is related to the original issue in this thread but maybe they are related. I think my issue has to do with the grace period and the setting cap exam time at close date. Where submits that happen right after the timer ends (but the grace period is still open) cause all student answers to go blank (though they are all in history). I haven't had time to test if setting cap exam time at close date to no helps with this issue or not.

[Alex-Jordan]

@maxlemieux I see that you are a student at PCC, which not so coincidentally is where I work. Feel free to email me directly if there is an issue that you can reliably replicate. I would love to have a screen share session where I could witness this first hand to understand its cause and work on a resolution.

What I referred to is not something you would see as a student. The quiz questions would appear to you as 1,2,3,4,... but there is a place in the instructor interface where they might be nubmered differently. The quiz module just always presents them as 1,2,3,4, in part because it may be ordering them randomly.

[dlglin]

I'm seeing this behaviour on my 2.17 system. In my case it happened on a gateway quiz, with one additional observation: One question on the quiz has both MQ answer blanks and a popUp question. The popup answer is saved in Q0002_AnSwEr0003, but the numeric answers aren't saved in Q0002_AnSwEr0001 and 2:

["Q0002_AnSwEr0001","","Q0002_AnSwEr0002","","Q0002_AnSwEr0003","No, not mutually exclusive","MaThQuIlL_Q0002_AnSwEr0001","25","MaThQuIlL_Q0002_AnSwEr0002","\\frac{16}{25}","MaThQuIlL_Q0002_AnSwEr0003",""]

[drgrice1]

I suspect that there is some JavaScript error that is occurring in these situations. However, until this can be reproduced and a console log is obtained showing what is going on, I don't know what to do about it.