taniwallach
commented
4 years ago Summary
I did some pretty careful (but not exhaustive) testing.
I think the only issue causing the errors you encountered are related to having an unpatched (standard) version of Lite.pm in the two places it is used from in these tests using sendXMLRPC.pl:
- The version inside your docker container - used by the server processing the call, which is the cause of the trouble in the "first" rendering when answers are submitted.
- The version on your local hard disk. When you run
sendXMLRPC.pl -Bfrom the local command line - it will make use of the local version of Lite.pm to create the XML structure being send to the server for processing. When there is Hebrew in the answers thr-Bswitch forces the code to process - the bug hits.
As discussed in
https://github.com/openwebwork/webwork2/issues/967#issuecomment-523941959 the solution to these issues is to locally patch Lite.pm so it treats UTF-8 encoded text as text (and not as binary data to be base64 encoded). Unfortunately, the underlying Perl XMLRPC packages is not being maintained (for quite a long time) so this patch is apparently not going to get into the default package on OSes or CPAN in the foreseeable future.
Conclusions
Based on the results of the tests below, I think that at present, we do not need to change any WW code. What is needed is for suitable instructions:
- The WW install instructions should instruct admins to patch
Lite.pmon their servers (or inside their Docker image or mount the fixed file into their Docker container) - The documentation about using
sendXMLRPC.pland about setting up thedeamon_courseshould explain that local use ofsendXMLRPC.plrequires patching the local copy ofLite.pm.
We also should probably provide the patched Lite.pm inside webwork2/docker-config/ and mount it automatically using the standard docker-compose.yml provided. I had suggested that already in https://github.com/openwebwork/webwork2/issues/967#issuecomment-523942985 .
Test with patched Lite.pm only on the server side (test 1)
I already had the modified version of Lite.pm mounted via docker-compose.yml as suggested in https://github.com/openwebwork/webwork2/issues/967#issuecomment-523941959 so
I first tested running the first part of the experiment using just the patched Lite.pm inside Docker and while the PC version (/usr/share/perl5/XMLRPC/Lite.pm) is not patched:
Note: Since https://github.com/openwebwork/webwork2/issues/967 is an issue and not a pull request, making the patch needed to Lite.pm is something to be done by hand.
experiment 1
Using develop branch on origin
- sha: a4f0a6a just after merge of new spanish .po. May 7, 2020 at 9:09 AM
- Includes PR #967 which modified Soaplite to not encode into base64 (line 55 of XMLRPC/Lite.pm
- The test file I use is this one (
less t/xmlrpc_utf8_fix.pg):with the command
perl sendXMLRPC -bB t/xmlrpc_utf8_fix.pgyou get two reponses in the browser window -- the first is the rendering of the original problem and the second is the problem with answers supplied by mining the 'correct answers' returned by the original rendering.
the first rendering appears fine.
- submitting the first rendering (with one of the hebrew answers)using the browser "show correct answer" button returns
Wide character in subroutine entry at /Volumes/WW_test/opt/local/lib/perl5/vendor_perl/5.12.3/XMLRPC/Lite.pm line 181.
so I ran
perl clients/sendXMLRPC.pl -b t/xmlrpc_utf8_fix.pgfrom my webwork2/clients directory. There is a pink warning sections shown but no real warnings are there.
I then submitted an answer, and did not get any error about a Wide character in subroutine.
This seems to indicate that patching the version of Lite.pm in the Docker container should prevent the error you saw when submitting an answer in the -b first rendering. Moreover, using the unpatched version of Lite.pm instead was tested and shown to trigger that warning message.
@mgage - on your PC the file used inside Docker seems to be at /Volumes/WW_test/opt/local/lib/perl5/vendor_perl/5.12.3/XMLRPC/Lite.pm
Test with patched Lite.pm only on the server side (test 2)
Next I ran the second part (alone) still when the local PC version of Lite.pm is unpatched.
On the command line, I get an error response:
perl sendXMLRPC.pl -B t/xmlrpc_utf8_fix.pg
home directory .
There were a lot of errors
Errors:
Wide character in subroutine entry at /usr/share/perl5/XMLRPC/Lite.pm line 181.
End Errors
<h2>WebworkClient Errors</h2><p>Errors: <br /> <blockquote style="color:red"><code>Wide character in subroutine entry at /usr/share/perl5/XMLRPC/Lite.pm line 181.
xmlrpcCall to renderProblem returned no result for /data2/webwork_docker_clean_for_PR/webwork2/clients/t/xmlrpc_utf8_fix.pg
and the browser did not display the problem text and showed
Unable to decode problem text
xmlrpcCall to renderProblem returned no result for /data2/webwork_docker_clean_for_PR/webwork2/clients/t/xmlrpc_utf8_fix.pg which is the same error you reported.
Test with patched Lite.pm on both the server side and the client side (test 2 only)
Next I patch the version of Lite.pm on my local PC to match what is being used in Docker (the fix from https://github.com/openwebwork/webwork2/issues/967#issuecomment-523941959)
Then I ran the second part (alone) with the patched local PC version of Lite.pm (still with the patched version in use inside Docker):
perl sendXMLRPC.pl -B t/xmlrpc_utf8_fix.pg Results:
- On the command line, there is no longer an error message.
- The problem and the results table rendered properly in the browser.
Test with unpatched Lite.pm on the server side (test 1)
Finally, I temporarily dropped the use of the patched version of Lite.pm in the Docker container, and ran the "first" -b test again. As expected, when a Hebrew answer was submitted, the browser failed to display any lines in the results table and failed to show the problem text, and instead showed:
WebworkClient Errors
Errors:
Wide character in subroutine entry at /usr/share/perl5/XMLRPC/Lite.pm line 181.
End Errors
WeBWorK using host: http://localhost:8080, course: daemon_course format: standard
Results for this submission
# | Entered | Answer Preview | Result
-- | -- | -- | --and the below the table
All of the answers above are correct.
Unable to decode problem text
xmlrpcCall to renderProblem returned no result for /data2/webwork_docker_clean_for_PR/webwork2/clients/t/xmlrpc_utf8_fix.pg Aside: The pink warnings sections seems to be due to the the code roughly at https://github.com/openwebwork/webwork2/blob/develop/lib/WebworkWebservice/RenderProblem.pm#L482
which creates a non-empty value for WARNINGS regardless of whether or not there is anything to be reported. However, it seems that the empty warnings section can be ignored.
mgage
drgrice1
some of the changes which encode utf8 strings as bytes (and then decode them) seem to have left sendXMLRPC behind. Here is the issue:
I believe the basic issue is that in the upgrade that handled wide characters the corresponding fixes to sendXMLRPC.pl were not made. I’m having trouble getting the right fix (just the right amount of encoding/decoding). There is a slightly larger issue that if I’m having trouble keeping things balanced perhaps the basic scheme can be rethought so that it is less fragile. Let’s put that thought on hold for now and just deal with the immediate problem. Here it is:
experiment 1
less t/xmlrpc_utf8_fix.pg):perl sendXMLRPC -bB t/xmlrpc_utf8_fix.pgyou get two reponses in the browser window -- the first is the rendering of the original problem and the second is the problem with answers supplied by mining the 'correct answers' returned by the original rendering.Wide character in subroutine entry at /Volumes/WW_test/opt/local/lib/perl5/vendor_perl/5.12.3/XMLRPC/Lite.pm line 181.xmlrpcCall())shows:————————
can you verify this with this version of the repos (or any other)?