Some new testing experiences

[alcarola]

Hi!

I have now been testing out the latest developments of STACK a little bit. It mainly looks very good, with in particular instant validation being a great addition! I found the script to change the dollar signs into ( and [ too. This is great since, translating Moodle gives problems if $ $ is always rendered as latex.

I have the following questions:

1. I have a working STACK 3.0 beta installation and am worried about performance when my course starts. It has only about 20 users. Should I use the Maxima pool? (I run a Linux server.)
2. Half a year ago or so, the interactive question behaviour allowed the student to try solving one particular question in a test multiple times before handing in the test. I found this very handy, but it seems to be gone now. Is this intentional or am I overlooking something? (I am mainly going to use STACK for providing the students with opportunities to practice problem solving and trying again upon failure would seem handy to me.) I tried turning on the option "Each attempt builds on the last" in the quiz settings but that did not solve my problem. I read the moodle docs for the quiz activity, and there it seemed like it's possible to specify the number of times each question can be attempted, but I nowhere find such a setting. I am running Moodle 2.3+, but plan to upgrade to 2.4 soon, in case that matters.
3. The general feedback (i.e. the worked solution) now seems to work according to the settings for the test, i.e., if I uncheck the box to show the general feedback, then no worked solution is showed. Good. But the feedback on correctness does not seem to do the same. I uncheck the box to provide correctness feedback, but when the student feeds an incorrect answer, he nevertheless gets an answer in red saying that it's incorrect. Since we have included the feedback on correctness in the PRTs, I would like to disable the redundant automatic feedback on correctness. Is this a STACK issue or a Moodle issue? (I reported something similar to this half a year ago in this issue: https://github.com/maths/moodle-qtype_stack/issues/31 )
4. Regarding translation. Should I now contribute my translations via github or amos?

Any help would be greatly appreciated!

Mikael

[sangwinc]

1. I recommend you use either the optimized linux version (which I'm using here in Birmingham) or the maxima pool which I believe Tim will use at the OU.
2. The whole point of STACK is the interactive behaviour, and that is what I'm using. You set this in the quiz. Are you sure you have the right settings?
3. I'll check this out.
4. Please use AMOS for translations.

Thanks for your interest, and please let me know how you get on.

Chris

[alcarola]

Thanks for your quick response!

Sorry for the terminology confusion above, please replace all instances of "test" by "quiz" in what I wrote.

1. I think I run an optimized linux binary already, but I'll test the maxima pool while I still have time. :)
2. In the "Edit Quiz" window I find no settings connected to this. There you only add questions, page breaking, setting question scores etc. When I click on "Edit settings", still in the quiz, I have set the following options: https://dl.dropbox.com/u/11924798/Editing%20Quiz.pdf (My setting in "Each attempt builds on the last" does not seem to matter for my problem.) Now when I solve an exercise incorrectly, I get no option to try it again (or I can't find it) without handing in the whole quiz. See here: https://dl.dropbox.com/u/11924798/Quiz%20Attempt.png -- the "Check" button is greyed out and the only thing I can click is "Next".
3. Thanks! You can also see in my attached quiz settings, that I've tried turning this off, but still it appars in the quiz (as you can see in the quiz attempt attachment). The behaviour is by the way the same when I preview a question in the question bank, not only inside of the quiz.
4. Good, that makes life easier.

Greetings, Mikael

[sangwinc]

Mikael,

1. There really is an option, it is set when you first set up the quiz, not when you add questions.

When you first create a new quiz you go to the page "Adding a new Quiz".

Half way down is a box "Question behaviour". You need to set it here. Does this make sense?

Chris

[alcarola]

Hi again,

Thanks for the reply! Strange that the dropbox links above don't work. Here are my quiz settings in another location: http://users.abo.fi/mkurula/saker/EditingQuiz.pdf and here is me filling in the wrong answer to a question: http://users.abo.fi/mkurula/saker/QuizAttempt.png

2-3. Does the screenshot not contradict the quiz settings both regarding the missing "Try again" option and the display of correctness feedback? I can further add that I had precisely the same problems with Moodle 2.4 also.

1. Also, about another old thing I reported in https://github.com/maths/moodle-qtype_stack/issues/16 I still have the problem that the health check script overwrites maximalocal.mac, even though that issue is closed? Also the same problem persists, that if I set the maximalocal.mac file to read-only, then the healthcheck script doesn't finish running. I ran into this because I'm now trying to install the maxima pool servlet.

Thanks and have a nice weekend! :)

Mikael

[alcarola]

Ok, now I found out about the healthcheck script and indeed it solves the problem I originally had.

A minor observation. The following string showed up when I ran the healthcheck script in optimised mode: "The version of the STACK-Maxima libraries being used (stackmaximaversion) does not match what is expected (2012122800) by this version the the STACK question type. Please rebuild your optimised Maxima executable." It has two mistakes: there are two "the" after each other and the link at the end of the string does not work for me, it only points to http://localhost/moodle/question/type/stack/%7B$a%7D

Still getting a few random errors from the maxima pool servlet, but I don't know how to debug them, because I get no entries in the tomcat log. I'll try again later with a dedicated tomcat server for the pool servlet.

Friendly greetings, Mikael

[alcarola]

Another two-part update:

I. I now installed maxima pool on my production server. It seems to work well so far, but I'll test it more next week. I still ran into problems with the healthcheck script. What I would need is to use the maxima pool with an optimized maxima, but there is no such option. In the maxima pool mode, the healtcheck script still overwrites the maximalocal.mac. This in turn breaks my maxima pool. Would it be possible to set the optimized setting separately from which mode (server, pool, linux, etc.) stack runs in, e.g. with a check box?

II. Another question, which is not really a stack issue, but perhaps you know? On my test server the sbcl-optimized binary takes 12 MB of RAM, whereas on my production server the sbcl-optimized binary it takes 55 MB. It is not the same binary obviously, the test server is a 32-bin ubuntu 12.04 and the production server is a 64-bin centos 6.

Fg, Mikael

[aharjula]

As to maximalocal.mac being overwritten, when using MaximaPool or Server you can just copy and modify your maximalocal.mac where ever you want and configure the server to use that. That way the healthcheck script can't overwrite it. But when you do that you have to make sure you remember to upgrade it when you upgrade your stack, on the other hand you'll need to upgrade the optimised maxima memory-image also...

[alcarola]

Ah indeed, I have not ever upgraded my maxima. Maybe this is the explanation for me getting sporadic errors with maxima or the maxima pool? At the same time as I upgrade the maxima image, I should probably try to reduce the size of it. Do you happen to know why the production image is so much larger? The version numbers are

Production server (set to use Maxima 5.23.1, don't remember why I didn't manage to install 5.23.1): -bash-4.1$ /var/www/moodledata/stack/maxima-sbcl -version Maxima 5.23.2 -bash-4.1$ /var/www/moodledata/stack/maxima-sbcl --version SBCL 1.0.38-3.el6

Test server (set to use Maxima 5.24.0): mkurula@virtuālā_desa:~$ /moodle2data/stack/maxima-sbcl -version Maxima 5.24.0 mkurula@virtuālā_desa:~$ /moodle2data/stack/maxima-sbcl --version SBCL 1.0.50.0.debian

Also: Is there a way to make the maxima pool output some debug info into the tomcat log? I'd like to try to track these problems down and eliminate them.

[aharjula]

Older version probably optimises less or maybe the 64bit sbcl is somehow wasteful? With maxima 5.28.0 and CLISP 2.49 my testing servers basic image-file is 14MB and that is on a 64bit Ubuntu 12.04.

As to logging to tomcat-log, I never needed that so it does not exist yet. The idea is to get the basic tests running on the servlets status page and once they run you can use the maxima debug feature on STACKs side. You might want to check catalina.out as it might have exceptions listed if the Pool has some serious problems, but for now the Pool has no logging features at all.

[alcarola]

So... I now tried to compile Maxima 5.28.0 with both clisp (test server) and sbcl (production server). The results were:

The CLISP version dies immediately with the message "module 'syscalls' requires package OS." I used clisp 2.49, which seems to have been the latest version for years now. In this thread it is discussed: http://web.archiveorange.com/archive/v/nsNbdc66dCHkCg8lKyXg How did you manage to optimize this version of maxima with this version of clisp?

The sbcl version still takes 50 MB RAM but now it typically takes around 10 seconds to start, whereas the previous version usually started in much less than a second. So I guess I'll have to stick to my old versions, but at least I'll reoptimise them.

Ok about the logging, thanks for letting me know!

[alcarola]

By the way, you don't need to answer the clisp question, I will probably not use that anyway. Now I can further report that 5.24.0 with sbcl compiled on the 64-bit centos 6 server also takes 55 MB RAM. On the 32-bit ubuntu it only took around 14. :/

[alcarola]

Now I have upgraded my maxima image on the test server, but now I get no graphs in the stack healthcheck script. Have I still configured the maxima pool incorrectly somehow or is this a maxima/stack error? The derivative of the rational function earlier in the script is calculated fine and the maxima pool health check provides no info that anything would be wrong (but I'm a bit surprised it still takes around half a second to start up a maxima instance). My settings below.

This is what I get instead of the graphs:

Errors
The CAS returned the following error(s):plot([sin(x),x,x^2,x^3],[x,-3,3],[y,-3,3]) caused the following error: plot2d: some values were clipped.plot2d: some values were clipped.
Debug info
Maxima command
Errors
The CAS returned the following error(s):plot([sin(x),x,x^2,x^3],[x,-3,3],[y,-3,3]) caused the following error: plot2d: some values were clipped.plot2d: some values were clipped.
Debug info
Maxima command not found in the cache. Using the raw connection.
Context used
Platform: tomcat
Maxima shell command: http://localhost:8080/MaximaPool/MaximaPool
Maxima initial command: load("/moodle2data/stack/maximalocal.mac");
Maxima timeout: 5
Maxima command
cab:block([ RANDOM_SEED, OPT_NoFloats, sqrtdispflag, simp, assume_pos, caschat0, caschat1], stack_randseed(0), make_multsgn(dot), make_complexJ(i), OPT_NoFloats:true, sqrtdispflag:true, simp:true, assume_pos:false, print("[TimeStamp= [ 0 ], Locals= [ ") , print("-1=[ error= ["), cte("__stackmaximaversion",errcatch(__stackmaximaversion:stackmaximaversion)) , print("0=[ error= ["), cte("caschat0",errcatch(caschat0:plot([x^4/(1+x^4),diff(x^4/(1+x^4),x)],[x,-3,3]))) , print("1=[ error= ["), cte("caschat1",errcatch(caschat1:plot([sin(x),x,x^2,x^3],[x,-3,3],[y,-3,3]))) , print("] ]") , return(true) ); 

Timings
Start: 1357468762.0799, End: 1357468763.1294, Taken = 1.0494740009308
CAS result
Maxima restarted.
(%i2) 
(%o2) "/var/lib/tomcat6/webapps/MaximaPool/WEB-INF/classes/maximalocal.mac"
(%i3) 
[TimeStamp= [ 0 ], Locals= [  
-1=[ error= [ 
], key= [ 
__stackmaximaversion 
] 
, value= [ 
2012122800 
], display= [ 
2012122800 
] 
],  
0=[ error= [ 
], key= [ 
caschat0 
] 
, value=[], display=[] 
],  
1=[ error= [ 
plot2d: some values were clipped.
plot2d: some values were clipped.
], key= [ 
caschat1 
] 
, value=[], display=[] 
],  
] ] 
(%o3) true
(%o4) 
Unpacked result as
Array
(
    [-1] => Array
        (
            [key] => __stackmaximaversion
            [value] => 2012122800
            [display] => 2012122800
            [error] => 
        )
    [0] => Array
        (
            [key] => caschat0
            [value] => 
            [display] => 
            [error] => 
        )
    [1] => Array
        (
            [error] => plot2d: some values were clipped.plot2d: some values were clipped.
            [key] => caschat1
            [value] => 
            [display] => 
        )
)

And these are my maxima pool settings:

# Config for maxima pool
### Maxima related commands and parameters
maxima.commandline = /moodle2data/stack/maxima-5.28.0-sbcl
# a file to load to maxima using load(""); set to false or empty if not needed
maxima.load = /var/lib/tomcat6/webapps/MaximaPool/WEB-INF/classes/maximalocal.mac
# working directory for the process
maxima.cwd = .
# to identify the state of the process we check the output for specific strings, if you change the loading process change these strings to match
# Input the load command after this, if defined...
maxima.ready.for.load = (%i2)
# The process has loaded when this appears.
#maxima.ready.for.use = Stack-Maxima started
maxima.ready.for.use = (%o2)
### File handling related parameters, these make the difference between shared filesystem installations and truly separate servers
# Do we handle files?
file.handling = false
# The command template to pass the current working and output directories to the process, use placeholders to mark the paths, depending on your system we may escape \-chars in the path.
## Usable placeholders:
##  %WORK-DIR% the directory that won't be transferred to the client side
##  %WORK-DIR-NE% use this if you do not want \-chars escaped
##  %OUTPUT-DIR% the directory that will be transferred to the client side
##  %OUTPUT-DIR-NE% use this if you do not want \-chars escaped
maxima.path.command = TMP_IMG_DIR: "%WORK-DIR%/"; IMG_DIR: "%OUTPUT-DIR%/"; URL_BASE: "%PLOT-URL-BASE%";
### Pool related parameters, times in milliseconds
pool.size.min = 5
pool.size.max = 20
# This is the limit of simultaneously starting processes this combined to the update frequency defines the maximum load
pool.start.limit = 3
# Pool update cycle (ms between updates)
pool.update.cycle = 500
# How long does the startup of a process take? We adapt to external load by keeping track of this
pool.adaptation.startuptime.initial.estimate = 2000
# How often do we get requests? 1/ms, kHz
pool.adaptation.demand.initial.estimate = 0.001
# How big a data-set should be kept for estimates, do not make this too big if the usage is not nearly constant.
pool.adaptation.averages.length = 5
# Pool size depends on the demand and startuptimes the system tries to maintain the minimum size but as demand may vary one should use a multiplier to play it safe.
pool.adaptation.safety.multiplier = 3.0
# This is the time that a process is allowed to take when starting up.
pool.startup.time.limit = 10000
# This is the time added to the lifetime of a process when it is taken to use so that it wont be killed while in use
pool.execution.time.limit = 30000
# This is the lifetime given to a process
pool.process.lifetime = 600000
### Alternative configuration file
# If you give the name of another configuration file here, it will be read after
# this file. That lets you set the properties in a place that is not complied
# into the servlet, so you can change the configuration without re-compiling.
#extra.config = /moodle2data/stack/maximapool-step2.conf

[aharjula]

file.handling is false so you are using the MaximaPool mode. Check the following things:

• in your maximalocal.mac you define the path to save the images to it should match the path used in normal non pooled maximalocal.mac configuration so that STACK can find the pics
• make sure that the tomcat user running the pool and the maximas in it can write tho plots to that directory
• if the MaximaPool is running on another server, does that server have gnuplot to actually generate those plots

But as you get that "plot2d: some values were clipped." I would think that you need to test plotting in that maxima first.

[alcarola]

Thanks for your help again! I didn't manage to figure out the problem of missing plots, but now the plots are there and all seems to work fine, so I'll just pretend I never saw that problem. :)

Regarding huge maxima images. I think it is wasteful lisp that is the problem, more precisely that the centos 6 sbcl binary is compiled with a huge dynamic space. I downloaded a newer sbcl from the sbcl home page and compiled maxima with that sbcl. Then the maxima image size shrunk to 27 MB, which I guess will have to be considered acceptable for now. (The 64-bit SBCL has a larger dynamic space than the 32-bit version, whatever that means, and I think this is the reason for the extra 10 MB in the image size.)

This solves question 1 in the original post, but questions 2 and 3 remain. :)

[alcarola]

Issue 2 in the original post is resolved now. I see that I confused the adaptive behaviour with the interactive behaviour. I believe that half a year ago, before hints were implemented in STACK (?), the interactive behaviour behaved like the adaptive behaviour without penalties does today? I simply solved this problem by changing from interactive behaviour to adaptive without penalties.

So now only the question about undesired display of the general feedback remains.

[sangwinc]

Yes Mikael.

We really need to document these behaviours better. This is new in STACK 3, and the documentation structure is largely taken from the old system with updates. Some pages on the quiz and what options you set there are needed, and I'll work on this.

I'll also look into the display of general feedback.

[timhunt]

This issue has got rather long and hard to understand, also I think almost everything been sorted out, so I am going to close this. If Chris finds a genuinue issue with General feedback then we can open a new issue specifically for that.

To explain the maxima libraries version thing: When you build a maxima-optimised with a particular version of STACK, you get that version of STACK's Maxima code. A later version of STACK may make some changes to the Maxima code, for example to add a new feature, or to fix a bug. When that happens, you have to re-build maxima-optmised as part of the upgrade. The notice on the healthcheck page is to remind you to do that.