search

openwebwork / webwork2

Course management front end for WeBWorK
http://webwork.maa.org/wiki/Main_Page
Other
146 stars 165 forks source link

error saving score with manual grader #1311

Closed Alex-Jordan closed 3 years ago

Alex-Jordan commented 3 years ago

I was using @drgrice1's new manual grader on an essay question, preferable to me over the older essay grading device. I reached one student, and after entering scores on the problem parts (so the final score for the problem was recorded) and entering a feedback message, clicking save gives:

Error saving score. /webwork2/instructorXMLHandler response: Forbidden

(That message flashes in pink.) I revisited this page a few times and one time, it managed to save the score (as the wrong score, becasue it had reverted) but I had not entered the feedback message. Every time since I have not had success saving. It's weird that is just for this one student. There is nothing unusal about this student's account. I have cleared my cache and there is no change.

I will work around it and directly adjust their score. Does anything occur to you @drgrice1 what might be a cause I could probe for?

drgrice1 commented 3 years ago

I can't think of anything right off hand. That is odd. Clearing your cache wouldn't help in this situation. You might look in the database and see if there is anything odd in the data in the problem_user or past_answer tables for this problem.

Alex-Jordan commented 3 years ago

OK, I'll steel myself up to look into the database. I did try clearing the cache before and that didn't help.

drgrice1 commented 3 years ago

The issue is something with the xmlrpc request, so clearing your cache won't do anything useful.

I think you are saying that you tried submitting the score without the comment, correct? I was guessing that there might be something in the comment that was causing an issue (an odd character or something), but if you have the problem when just submitting the score, that is unlikely to be the problem.

taniwallach commented 3 years ago

The XML subsystem sometimes has issues detecting how to handle some conversions and this can trigger the "Forbidden" error.

Some forum threads to look at:

https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4933

https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4173

https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4795

https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4641

On Thu, Apr 8, 2021, 02:48 Alex Jordan @.***> wrote:

I was using @drgrice1 https://github.com/drgrice1's new manual grader on an essay question, preferable to me over the older essay grading device. I reached one student, and after entering scores on the problem parts (so the final score for the problem was recorded) and entering a feedback message, clicking save gives:

Error saving score. /webwork2/instructorXMLHandler response: Forbidden

(That message flashes in pink.) I revisited this page a few times and one time, it managed to save the score (as the wrong score, becasue it had reverted) but I had not entered the feedback message. Every time since I have not had success saving. It's weird that is just for this one student. There is nothing unusal about this student's account. I have cleared my cache and there is no change.

I will work around it and directly adjust their score. Does anything occur to you @drgrice1 https://github.com/drgrice1 what might be a cause I could probe for?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openwebwork/webwork2/issues/1311, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJKHHRQRQFHKHTVKHNSAPELTHTVOVANCNFSM42RX5VSA .

Alex-Jordan commented 3 years ago

I still have not peered into the database. But I want to note a few more things. All questions in this set are essay questions. There is no issue with saving score and comment for this student for question 1. Question 2 was where the previously reported issue happens, and is still the case. I just reached question 3 for this student, and it happened there again. However, I had entered a score and a comment. I got the Forbidden message. But I've navigated away and come back, and the score persisted. Now, if I submit on this page (even with the comment field blank) I get the Forbidden error, but again, the score is there in the database now. I moved on to other questions for this student and so far, no issues with submitting scores and comments.

@taniwallach I looked through those threads, thank you for gathering them. This seems like something else, although perhaps a different face of a common issue.

drgrice1 commented 3 years ago

Hmm. If you don't mind, could you send me the source file for question 3? It sounds like this might be specific to that problem.

taniwallach commented 3 years ago

Alex - Do you have a full stack-trace from the error. You might need to look in the Apache error.log file.

drgrice1 commented 3 years ago

The XML subsystem sometimes has issues detecting how to handle some conversions and this can trigger the "Forbidden" error. Some forum threads to look at: https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4933 https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4173 https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4795 https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4641

These are all unrelated to this issue entirely. All of those are using the renderProblem command of instructorXMLhandler.pm, which calls xml_filter. The problem grader does not use that command, and never calls xml_filter.

Alex-Jordan commented 3 years ago

Here are the two problems files. Number 2 is 1.2.2.pg and number 3 is 1.2.4.pg.

http://spot.pcc.edu/~ajordan/mth243-alex_jordan-202102.tgz

drgrice1 commented 3 years ago

Okay. I will see if I can reproduce it.

drgrice1 commented 3 years ago

Also, is this in a gateway quiz or regular homework?

drgrice1 commented 3 years ago

I am not able to reproduce this. Does this happen with all students on these problems, or just with one student? Perhaps there is something specific with the answer the student gave, or with the comment you are entering, etc.

Alex-Jordan commented 3 years ago

It is a regular homework, and it is only one particular student. I mentioned above there is nothing strange about this student's user ID. It's only these two questions, and the error occurs whether or not there is a comment. I'll find time to look into the database and see if that reveals anything. For me, I need to set aside time since I don't know mysql and I will be carefully studying before I run any commands. I'm solo parenting the first half of the day, then covid vaccine appointment, so I may not get to it. Thanks for looking at this, and I think waiting now until I access the database is best.

Alex-Jordan commented 3 years ago

Worth mentioning: I can use the older "Grade Problem" method for assigning scores and comments to problems with essay answers. There is no error from that avenue. So it's at least possible to write to the database for this student int he score and comment fields. But it's not as nice to use that tool than it is with the manual problem grader for a few reasons.

Alex-Jordan commented 3 years ago

Posting an update. I am still grading this set (which is a painfully tedious statistics set with lots of essay questions). I reached another student on another problem number where the issue occurs.

drgrice1 commented 3 years ago

I have been trying to reproduce this, but still can't.

I did manage to get the

Error saving score.
/webwork2/instructorXMLHandler response: Forbidden

message and I though I might have reproduced it, but it turned out it was because I had left the computer for a while and the session timed out.

I suspect, this may have to do with the actual comments that your are entering. Is there anything different about the comments that are causing the issue than from other comments?

drgrice1 commented 3 years ago

Is there anything in your apache2 error.log?

Alex-Jordan commented 3 years ago

Is there anything different about the comments that are causing the issue than from other comments?

Nothing that I can tell, and I did look at that. Once it happens, then form that point forward I cannot save even with a blank empty comment field. Next it happens, I will try to preserve the comment. And again it may become clear once I look in the database.

Note that the following has happened:

  1. I got the forbidden error for one student for one problem.
  2. I went to the older page for grading essay questions, entered a score and a comment, and saved the page successfully.
  3. I return to the manual problem grader, to that problem for that student. I see the comment that I entered in step 2. I am still unable to click save, even with an empty comment field, without getting the error.

That at least mildly suggests that it is not about the comment I left. It's already been successfully saved to the database through the other mechanism.

Here is the comment I saved in step 2. This is copy-pasted from where it appears in step 3, in the editable field:

It does not sound like age was used as a variable here. But all of the other variables you listed are good

It also appears above on display, again copy-pasting:

Instructor Comment:
It does not sound like age was used as a variable here. But all of the other variables you listed are good

The latter has a space at the end that the former does not. But I don't see any other difference or anything weird about the characters.

Alex-Jordan commented 3 years ago

Is there anything in your apache2 error.log?

I will look. I have to stop now and get ready to head out for my covid shot. I will look at the log and the database this evening.

drgrice1 commented 3 years ago

Yeah, I don't see anything that should cause problems with those comments. I think the apache2 error.log should reveal something. So I will wait to hear what you find there.

I got my first covid shot yesterday! My arm is sore still.

Alex-Jordan commented 3 years ago

OK I'm all dressed up now, just anxiously waiting on my wife to get home.

I jumped on and in the error log I see:

Error message for command: putUserProblem 
<br/>faultcode: Client 
<br/>faultstring: Wide character in subroutine entry at /usr/share/perl5/XMLRPC/Lite.pm line 181.

<br/>End error message<br/>
Use of uninitialized value $headers{"sec-ch-ua"} in join or string at /opt/webwork/webwork2/lib/Apache/WeBWorK.pm line 255.
[Fri Apr 09 11:33:52.118083 2021] [perl:error] [pid 11317] [client 10.14.0.144:60508] [5cb15bc7-d42b-5be7-bb5c-ee249e546918::21ff8fbf-9962-11eb-8222-e23612ba273d] [/webwork2/instructorXMLHandler] {"Time":"Fri Apr 09 11:33:52 2021","HTTP Headers":{"Cookie":"WeBWorKCourseAuthen.mth060-michael_mcafee-202102=alex.jordan%09xKkpBtDJtdgemWDA2dDLOIZZhFdqgqzP%091617389004; WeBWorKCourseAuthen.mth252-alex_jordan-202102-assessment=alex.jordan%09LdVLVS55Kvbq7ApMUDttseVKxtaDkryg%091617683325; WeBWorKCourseAuthen.mth251-emiliano_vega-202102=alex.jordan%09wrPPRSOU9k3UJd3rjlcLeiR9Bxo4s6qQ%091617911351; WeBWorKCourseAuthen.mth252-alex_jordan-202102=alex.jordan%09Zmc0c6Tukd1Dqb0i0potxUa8yjSvHV6Y%091617923094; WeBWorKCourseAuthen.testing=alex.jordan%095qbinxi8jWlHZTkeyysjf4Mr50cEmxy4%091617931773; WeBWorKCourseAuthen.mth243-alex_jordan-202102=alex.jordan%09CcXfUTaJfBvSTpXrP4b9dJE9gvLrEMzm%091617991528; _ga=GA1.2.835395407.1522627130; __utma=168888480.835395407.1522627130.1617080511.1617926315.77; __utmz=168888480.1617926315.77.17.utmcsr=authenticate.pcc.edu|utmccn=(referral)|utmcmd=referral|utmcct=/; mjx.menu=collapsible%3Afalse; _gid=GA1.2.51112692.1617644831; TS01a4419f=01e249fa8b3ddee0ddd51675eaf9e639ee323bb849a22a1ecd9e9f71465be1f0eaf2ca452f76ce0f961771b0dd04b669f276599a7ed251fd38515643b33b6b83362a4e5ee56f23de4ae9cbc75312383c50103dca3c; ADRUM=s=1617986475655&r=https%3A%2F%2Fonline.pcc.edu%2Fd2l%2Flp%2Fauth%2Fsession%2FExpired%3F-1761606171; __utmc=168888480; IDMSESSID=97DA117FBB72906916370DE30953DE377DBF3B438440392EE09720D6165C5F03110673CC8DAAA3E34E3A957DDFCCEB8444C32A019F28C6A5F6A00889C899EEE7","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:87.0) Gecko/20100101 Firefox/87.0","Accept-Encoding":"gzip, deflate, br","Origin":"https://webwork.pcc.edu","Referer":"https://webwork.pcc.edu/webwork2/mth243-alex_jordan-202102/Module_1_Homework/2/","Accept-Language":"en-US,en;q=0.5","Host":"webwork.pcc.edu","X-Requested-With":"XMLHttpRequest","Content-Length":"213","Content-Type":"application/x-www-form-urlencoded; charset=UTF-8","Accept":"*/*","Connection":"keep-alive"},"URI":"/webwork2/instructorXMLHandler","Error record identifier":"5cb15bc7-d42b-5be7-bb5c-ee249e546918::21ff8fbf-9962-11eb-8222-e23612ba273d","Warnings":[],"Method":"POST"} Error message for command: putUserProblem \n<br/>faultcode: Client \n<br/>faultstring: Wide character in subroutine entry at /usr/share/perl5/XMLRPC/Lite.pm line 181.\n \n<br/>End error message<br/>\n * in WeBWorK::ContentGenerator::instructorXMLHandler::content called at line 233 of /opt/webwork/webwork2/lib/WeBWorK/ContentGenerator.pm\n * in WeBWorK::ContentGenerator::go called at line 386 of /opt/webwork/webwork2/lib/WeBWorK.pm, referer: https://webwork.pcc.edu/webwork2/mth243-alex_jordan-202102/Module_1_Homework/2/

The "mth243-alex_jordan-202102" course is mine. I see "wide character" but unsure where it comes from, since I'm just typing comments with a regular keyboard. Is it doable that you could feed me the mysql command I need to look at what stuff in the table for the the student at this problem?

drgrice1 commented 3 years ago

The second message you show is not related. I put in a pull request to fix that message, but it won't fix your issue.

drgrice1 commented 3 years ago

So your message reveals where the error is occuring. Something is going wrong when the user problem is begin saved to the database. Unfortunately, the error message that is shown is not very useful as to the actual cause of the error. It is not to different than the other error message you showed. The other message is a mistake @taniwallach made in an earlier pull request (he forgot to check a hash key was defined), and this error message is my mistake. I tried to return the actual error that occurs in this situation in the text message. That message is encoded to base64, and apparently the error message contains a character that doesn't work for that. I am going to put in a pull request that fixes that. It will instead make it so that the error is output to the apache2 error.log, but send a generic message (that will be encodable) back.

drgrice1 commented 3 years ago

@Alex-Jordan: Is there any chance you could merge https://github.com/openwebwork/webwork2/pull/1314 into the branch your server is running? If you do that, then we can get to the actual error that is occuring. It will appear in your apache2 error.log with that fix, instead of this uninformative error we are seeing from XMLRPC::Lite.

drgrice1 commented 3 years ago

There is another possibility that occurs to me. If this is the case #1314 won't help (in fact I closed that pull request as I don't think it was an issue after all). I suspect this other possibility actually is the problem, based on what is happening. That is the problem score is saved, but the comment is not. The javascript first submits the score, and then when that is successful it submits the comment. What is happening here is that the first request to submit the score fails, and so you get the error message, and the request to submit the comment is never sent. The failure of the first request occurs after the score is saved to the database when it is utf8/base64 encoding the result text message. The wide character is either in the problem id (doubtful), the set id, the user id, or the course name. This may be incorrect, but it seems to be the only thing that makes sense.

If you are willing to edit lib/WebworkWebservice/ProblemActions.pm on your server and do a little testing, we might be able to find for information. The only thing in all of webwork that uses this is the new problem grader as I created that file for the problem grader. This also won't prevent others from being able to use the problem grader. Change lines 89 and 90 to 

text => encode_utf8_base64("Updated problem.")

In other words, just take all of the interpolated variables out.

If that fixes the problem then my assessment above is correct.

drgrice1 commented 3 years ago

Another thing you could try is to modify /usr/share/perl5/XMLRPC/Lite.pm by changing line 55 to base64 => [10, sub {$_[0] =~ /[^\x09\x0a\x0d\x20-\x7f]/ && !utf8::is_utf8($_[0])}, 'as_base64'], as is done in docker-config/xmlrpc-lit-utf8-fix.patch for the docker build and see if that helps.

Alex-Jordan commented 3 years ago

OK, the change to lib/WebworkWebservice/ProblemActions.pm did not help the situation, but the change to /usr/share/perl5/XMLRPC/Lite.pm does. Now I can return to the problem for that student and save.

So does this mean that some non-utf8 character snuck into my comment?

Alex-Jordan commented 3 years ago

In the three instances of this that I encountered:

  1. the student answer has an en dash
  2. the student answer has an em dash
  3. the student answer has a fancy apostrophe

This makes a little sense. You can't base64 encode those in the usual way.

drgrice1 commented 3 years ago

No. The comment doesn't go through that encoding. Furthermore, the error was actually occurring before the comment was ever sent to the server.

There is a bigger problem here. @taniwallach believes this is a bug in the XMLRPC::Lite package that is fixed via the change you made. I am not entirely certain that is correct though. I think there is an issue with trying to directly base64 encode a utf8 encoded string. I don't think that is correct, and we need to fix this. base64 expects a binary string. utf8 encoded strings are not binary. See https://stackoverflow.com/questions/30106476/using-javascripts-atob-to-decode-base64-doesnt-properly-decode-utf-8-strings for more information on this.

The student's answer is not involved in this. The student's answer never goes through the XMLRPC::List encode/decode process in these requests.

Alex-Jordan commented 3 years ago

It's an odd coincidence that the three instances happen to be the three places I find non-ASCII characters among the student answers in these essay questions. I went flipping through student answers elsewhere in the set and I don't see any. Can you reproduce it if you try submitting an essay answer with such a character, then return to the grader and try to submit a comment?

drgrice1 commented 3 years ago

Yeah, so that does cause the error. However, at no point does the problem grader encode or even deal with the last_answer string. So I am not sure why the error is occurring with the new problem grader, but works with the old.

drgrice1 commented 3 years ago

I see. So the putUserProblem method of the WebworkWebservice returns the $userProblem after saving it to the database. Then the XMLRPC::Lite package encodes that for the transfer. I missed that, and that is what causes the issue.

So for now we will probably need to add this change to /usr/share/perl5/XMLRPC/Lite.pm to the upgrade/install documentation for those not using docker. This is applied by default for the docker install. Although, this needs further investigation as I stated in my previous comment.

drgrice1 commented 3 years ago

I don't know. This probably is just a bug in XMLRPC/Lite.pm. We need to replace that package with something else as we have already said.

Alex-Jordan commented 3 years ago

If nothing else, it feels good to understand the root cause of the issue. But it's not nothing else, because you did find a temporary solution.

drgrice1 commented 3 years ago

Yeah, I see that @taniwallach already added this to the release notes. I also added a command to hit comment on how to apply the patch to fix the bug.

Alex-Jordan commented 3 years ago

Do you think we could close this issue? (a) There is the note to edit Lite.pm in the release notes. (b) Even if there should still be an issue about this for finding a better solution, if there were a new issue it saves future people who want to work on it from reading all the irrelevant things that are now in this thread.

drgrice1 commented 3 years ago

We probably should close the issue for now. It really isn't a bug in webwork, but is a bug in a library it depends on.

Alex-Jordan commented 3 years ago

Closing since this is not a WW bug.

Alex-Jordan commented 3 years ago

There is a side effect to that editing of Lite.pm.

Consider a URL like the following:

https://webwork.pcc.edu/webwork2/html2xml?courseID=orcca&amp;userID=orcca&amp;password=anonymous&amp;course_password=orcca&amp;answersSubmitted=0&amp;displayMode=MathJax&amp;outputformat=simple&amp;problemSeed=11&amp;problemSource=RE9DVU1FTlQoKTsKbG9hZE1hY3JvcygiUEdzdGFuZGFyZC5wbCIsIk1hdGhPYmplY3RzLnBsIiwiUEdNTC5wbCIsIkFuc3dlckZvcm1hdEhlbHAucGwiLCJQR2NvdXJzZS5wbCIsKTtURVhUKGJlZ2lucHJvYmxlbSgpKTsKQ29udGV4dCgnTnVtZXJpYycpOwoKQkVHSU5fUEdNTApJbiBlYWNoIGV4cHJlc3Npb24sIGhvdyBtYW55IG5lZ2F0aXZlIHNpZ25zIGFuZCBzdWJ0cmFjdGlvbiBzaWducyBhcmUgdGhlcmU%2FCgphLiAgW2AxLTlgXQoKICAgIGhhcyBbX117MH17M30gbmVnYXRpdmUgc2lnbnMgYW5kIFtfXXsxfXszfSBzdWJ0cmFjdGlvbiBzaWducy4KCgphLiAgW2AtMTIrKC01MClgXQoKICAgIGhhcyBbX117Mn17M30gbmVnYXRpdmUgc2lnbnMgYW5kIFtfXXswfXszfSBzdWJ0cmFjdGlvbiBzaWducy4KCgphLiAgW2BcZGZyYWN7LTEzLSgtMTUpLTE3fXsyMy00fWBdCgogICAgaGFzIFtfXXsyfXszfSBuZWdhdGl2ZSBzaWducyBhbmQgW19dezN9ezN9IHN1YnRyYWN0aW9uIHNpZ25zLgoKCgpFTkRfUEdNTAoKQkVHSU5fUEdNTF9TT0xVVElPTgphLiAgW2AxLTlgXSBoYXMgemVybyBuZWdhdGl2ZSBzaWducyBhbmQgb25lIHN1YnRyYWN0aW9uIHNpZ24uCgoKYS4gIFtgLTEyKygtNTApYF0gaGFzIHR3byBuZWdhdGl2ZSBzaWducyBhbmQgemVybyBzdWJ0cmFjdGlvbiBzaWducy4KCgphLiAgW2BcZGZyYWN7LTEzLSgtMTUpLTE3fXsyMy00fWBdIGhhcyB0d28gbmVnYXRpdmUgc2lnbnMgYW5kIHRocmVlIHN1YnRyYWN0aW9uIHNpZ25zLgoKCgpFTkRfUEdNTF9TT0xVVElPTgoKRU5ERE9DVU1FTlQoKTs%3D

That is a src attribute for an iframe in this PreTeXt book. With the edit to Lite.pm, this fails and cites an inability to decode base64. I have undone the edit on the PCC server, so you won't see that failure if you visit that link. But if you change the WW server and change courseID, userID, password, and course_password to something that works on that server, and also have the edit to Lite.pm, I expect that you would see the failure happen.

I discovered that if I change the &amp; in the URL preceding selectors to just plain & then the problem would render. I don't understand why that helps, but it let me understand that this is not about any particularities of that base64 string.

taniwallach commented 3 years ago

@Alex-Jordan:

  1. Do you have a stack-trace (possible from Apache's error.log file) from when the error occurred?
  2. I tried putting the part of the string after the problemSource= into https://www.base64decode.org/ and it failed to decode the base64, regardless of which of several source encodings I tried. It looks OK for the start and then looks like gibberish:
DOCUMENT();
loadMacros("PGstandard.pl","MathObjects.pl","PGML.pl","AnswerFormatHelp.pl","PGcourse.pl",);TEXT(beginproblem());
Context('Numeric');

BEGIN_PGML
In each expression, how many negative signs and subtraction signs are there

from that point on there is "mojibake".

The change to Lite.pm essentially prevents the conversion to XML from changing valid sequences of UTF-8 bytes from being changed when encoding into XML. The "old" standard (from when Lite.pm was written did not allow UTF-8 encoded text, but the later standard does). See the links in https://github.com/openwebwork/webwork2/issues/967#issuecomment-523941959 . Also of interest: https://stackoverflow.com/questions/13743250/meaning-of-xml-version-1-0-encoding-utf-8

taniwallach commented 3 years ago

This is a response to @drgrice1's comment about base64 encoding and is not necessarily related to the main theme of this issue.

I worked on the issues with btoa for the Standalone rendered in https://github.com/drdrew42/renderer/pull/30 .

base64 expects a binary string.

To be precise, a sequence of bytes.

utf8 encoded strings are not binary.

Correct it is not a sequence of "bytes", and a string in Javascript can also fail to be a sequence of bytes which btoa can handle.

See https://stackoverflow.com/questions/30106476/using-javascripts-atob-to-decode-base64-doesnt-properly-decode-utf-8-strings for more information on this.

The issue is that it is necessary to understand that btoa (base64 encoding) was designed to handle arrays of bytes, while most modern systems encode character strings as a sequence of character codes which need not be one byte each.

taniwallach commented 3 years ago

I think the problem is that the entire URL was put through a URL-encoding process which modified the base64 value, and changed the simple & which is valid in a GET URL to &amp; which is the URL-encoded value of &.

If I manually "fix" the two URL-encoded bits inside the value of problemSource=:

BEGIN_PGML In each expression, how many negative signs and subtraction signs are there?

a. [1-9]

has [_]{0}{3} negative signs and [_]{1}{3} subtraction signs.

a. [-12+(-50)]

has [_]{2}{3} negative signs and [_]{0}{3} subtraction signs.

a. [\dfrac{-13-(-15)-17}{23-4}]

has [_]{2}{3} negative signs and [_]{3}{3} subtraction signs.

END_PGML

BEGIN_PGML_SOLUTION a. [1-9] has zero negative signs and one subtraction sign.

a. [-12+(-50)] has two negative signs and zero subtraction signs.

a. [\dfrac{-13-(-15)-17}{23-4}] has two negative signs and three subtraction signs.

END_PGML_SOLUTION

ENDDOCUMENT();



I have not considered why this worked before the change to `Lite.pm`. Possibly something was "fixing" the URL-encoding inside the `base64` value early enough in the processing of the code before the change, and now this is not done - so the "broken" base64 is what something is attempting to decode and that fails.
Alex-Jordan commented 3 years ago

Do you have a stack-trace (possible from Apache's error.log file) from when the error occurred?

I'll see what I can find.

I tried putting the part of the string after the...

You have to URL decode it first. There can be %2B in that string which needs to become + before a standard base64 decoder goes to work. Similarly with = and /. The mojibake probably starts where the first + occurs.

taniwallach commented 3 years ago

My best guess at present is that for some reason with the modified Lite.pm the base-64 decode is being called on the string when it still has the %2F instead of the / and the %3D instead of = (only when the URL as &amp;), so that the decoded PG code is invalid.

I'll try to look at this more tomorrow.

Alex-Jordan commented 3 years ago

I think this detail is important.

That was my experience.

Alex-Jordan commented 3 years ago

BTW I would never have discovered that, except that my school has a copy of the referenced PreTeXt book relying on my school's WW server, and another school has a copy of the referenced PreTeXt book also relying on my school's WW server. Our version of the book was working fine, theirs was not. The only difference was the use of &amp; versus & (which was a consequene of producing the HTML under different versions of PreTeXt.)

taniwallach commented 3 years ago

I have been wondering since last night whether that change ('&' instead of '&' in the HTML file) allows the browser to decode the two percent-URL encodeded characters in the problemSource before sending the call to the server or even just the first one. This earlier decode could also potentially be happening inside Apache before the request parameter values get to the Perl code. I'm skeptical about it being something done by WW Perl code.

One of those seems to me to be a potential explanation for the observed behaviour.

One temporary test we could try is URL decoding the value received for problemSource (before the base64 decode) if a percent appears to test the hypothesis that the problem is related to that.

Just printing the data at various stages of the xml2html pineline would also help. (The commented out debugging code @drgrice1 intends to remove in a recent PR has been used by @mgage and myself to understand what is happening inside the xml2html pipeline, which is why I am opposed to removing it.)

I don't think that base64 uses the '%' character in valid base64 data (2 characters other than A-Za-z are needed from what I recall) in which case such a change could be harmless. However if some dialect of base64 does use percent it would mangle other things and be a real bad mistake.

On Tue, Apr 13, 2021, 01:00 Alex Jordan @.***> wrote:

I think this detail is important.

  • Modify Lite.pm as we are discussing.
  • Follow a template link like the one that I posted but of course change the WW server to your own and the credentials too.
  • It won't work.
  • Now only change the & in the URL to &. Don't worry about the %2B etc. And now it will work.

That was my experience.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openwebwork/webwork2/issues/1311#issuecomment-818270513, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJKHHRVEGLIG57UYCB7R6VTTINUOFANCNFSM42RX5VSA .

drgrice1 commented 3 years ago

(The commented out debugging code @drgrice1 intends to remove in a recent PR has been used by @mgage and myself to understand what is happening inside the xml2html pipeline, which is why I am opposed to removing it.) I don't think that base64 uses the '%' character in valid base64 data (2 characters other than A-Za-z are needed from what I recall)

@taniwallach: You are welcome to implement proper debugging code that is not just a commented out warn using webwork's debug methods. Use good programming practices and it benefits everyone. If every developer added commented out debugging to every pull request then the code would become so cluttered with crap that no one would be able to decipher it. Most of the things that you called "debugging" code in my pull request were exactly this sort of crap that shouldn't be there.

taniwallach commented 3 years ago

Glenn - much if not most of the commented out debugging code predates my involvement with WW and was useful enough for it to be carried along. I have certainly used bits of it when trying to understand and fix some of the strange problems which occurred in the subsystem once UTF-8 was permitted in PG code.

I suspect that browsers could submit Unicode characters in UTF-8 inside the form fields long before WW could handle it. Things just broke in various places where invalid string cropped up.

In this case, I think removing this "bad code" without putting in a better "solution" is making more work for anyone who tries to work on this messy subsystem. Viewing the data along the path helps determine what is going wrong.

Is an if test on every run really better than commented out code to be enabled when needed? Is the time and effort in fixing it a good investment of someone's time when compared to the potential need to recomment something left uncommented in a code change.

Sorry for the long rant.

On Tue, Apr 13, 2021, 06:33 Glenn Rice @.***> wrote:

(The commented out debugging code @drgrice1 https://github.com/drgrice1 intends to remove in a recent PR has been used by @mgage https://github.com/mgage and myself to understand what is happening inside the xml2html pipeline, which is why I am opposed to removing it.) I don't think that base64 uses the '%' character in valid base64 data (2 characters other than A-Za-z are needed from what I recall)

@taniwallach https://github.com/taniwallach: You are welcome to implement proper debugging code that is not just a commented out warn using webwork's debug methods. Use good programming practices and it benefits everyone. If every developer added commented out debugging to every pull request then the code would become so cluttered with crap that no one would be able to decipher it. Most of the things that you called "debugging" code in my pull request were exactly this sort of crap that shouldn't be there.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openwebwork/webwork2/issues/1311#issuecomment-818405848, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJKHHRWLFOZHTB26BPTT373TIO3SLANCNFSM42RX5VSA .

taniwallach commented 3 years ago

Upon further thought, I think we need to consider how WW3 and all the backend modules should handle debugging levels, how and to where they should send their debugging output, and how it can best be collected.

Things will only get more complicated as more backend parts and subsystems become decoupled services/APIs.

The existing system has many different approaches and hacks as not all parts of the system can access the "nicer" debug functions which work in just some places.

This is really an issue for careful consideration, and redesign.

On Tue, Apr 13, 2021, 06:33 Glenn Rice @.***> wrote:

(The commented out debugging code @drgrice1 https://github.com/drgrice1 intends to remove in a recent PR has been used by @mgage https://github.com/mgage and myself to understand what is happening inside the xml2html pipeline, which is why I am opposed to removing it.) I don't think that base64 uses the '%' character in valid base64 data (2 characters other than A-Za-z are needed from what I recall)

@taniwallach https://github.com/taniwallach: You are welcome to implement proper debugging code that is not just a commented out warn using webwork's debug methods. Use good programming practices and it benefits everyone. If every developer added commented out debugging to every pull request then the code would become so cluttered with crap that no one would be able to decipher it. Most of the things that you called "debugging" code in my pull request were exactly this sort of crap that shouldn't be there.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openwebwork/webwork2/issues/1311#issuecomment-818405848, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJKHHRWLFOZHTB26BPTT373TIO3SLANCNFSM42RX5VSA .