search

openwebwork / webwork2

Course management front end for WeBWorK
http://webwork.maa.org/wiki/Main_Page
Other
146 stars 165 forks source link

Feature requests for Gateway quizzes #1156

Open taniwallach opened 4 years ago

taniwallach commented 4 years ago

[initially collected from participant comments during the "Lessons From the Major Online Systems" session at https://rederly.com/finals-conference.]

Alex-Jordan commented 3 years ago

create a method for a PG problem to be aware it is in GW vs homework usage mode [Edward Sternin]

I have a hack for this. Not that I want to promote hacks, but until the feature is built, it has worked for me. You can look at the environment variables and see if the GW suffix is defined. As in if($envir{QUIZ_PREFIX}) {do stuff};.

taniwallach commented 3 years ago

As a reminder to myself and others: Tom (?) from Rederly mentioned using some sort of timer on changes to input boxes to trigger their auto-save of answers.

esternin commented 3 years ago

Actually the more relevant approach would be to determine whether the hints are enabled or not. One of the differences between using the same problem in homework and in a gateway/quiz is to enable/disable hints, so displaying all text that is referring to hints ("You can get a hint if you do not get the answer in three tries" or similar) should be conditional on whether hints are enabled. And, this will take care of those cases where hints are actually allowed during GW as well.

But I am not clear: is there such an envir variable already, or is this a "hack" in the sense that you need to add the variable somewhere yourself? Surely, the renderer knows whether hints are enabled or not, but how?

Alex-Jordan commented 3 years ago

Regarding the hints issue.

WW has a cascading sequence of config files: defaults.conf, site.conf, localOverrides.conf, course.conf, simple.conf. (I may have the first two out of order). What is missing (imho) is a configuration for a problem set. It would come last in the load sequence. With hints (as well as MathQuill and other things) they could be turned on/off for a problem set. GW could be built to show hints in general, but turn it on/off depending on the configuration. I imagine all these configuration settings being stored in the database (and in set...def files), not actually in a .conf file. I might even advocate to take it so far as having config options per question (in the context of a problem set, not the code for the problem).

The leg work for this would be in building the interface for the instructor to configure a problem set.

@esternin My opinion is that it's a mistake to code any meta-information about hints into a problem in the first place. I mean, yes, code the hint itself. But don't refer to it in problem text. If hints are enabled and a hint exists for the problem, the system should just always automatically tell students that a hint is available in some reasonable, accessible way. I also advocate against setting a value for $showHints in problem code, since it is an instructor choice that depends on the context of how the problem is being used. But this again brings me back to wanting/needing problem set level configuration options.

esternin commented 3 years ago

@Alex-Jordan Possibly it is a mistake to "code any meta-information about hints into a problem", but this has been the default way of coding here for a long time. And just to make it clear, I want to read, not set the $showHints value, as in if if($showHints) {do stuff};

Alex-Jordan commented 3 years ago

@esternin Yes, I do understand all that. I feel that a big part of our problems in this area are legacy design issues and I'd love to see us tear those out by the roots instead of patching things at the surface level. But I know it will take a lot more work.

esternin commented 3 years ago

@taniwallach: Nathan, do you know of a way to know whether hints are enabled from inside the pg, or is this definitely a new feature request?

taniwallach commented 3 years ago

@taniwallach: Nathan, do you know of a way to know whether hints are enabled from inside the pg, or is this definitely a new feature request?

@esternin I took the bait and took a look. From what I think I understood, I don't think one can currently determine whether hints are enabled from inside a problem (from PG code). From what I seem to understand from looking at code in pg and webwork2, the webwork2 layer only passes to PG a setting called displayHintsQ (whether the hints should be shown or not) but none of the "reasons" why it was set.

awmorp commented 3 years ago

While we're at it:

drgrice1 commented 3 years ago

@awmorp: Note that your second suggestion is already implemented (at least if I understand what you are suggesting). In the gateway quiz settings select "Yes" for both "Show Scores on Finished Assignments?" and "Show Problems on Finished Tests".

awmorp commented 3 years ago

@awmorp: Note that your second suggestion is already implemented (at least if I understand what you are suggesting). In the gateway quiz settings select "Yes" for both "Show Scores on Finished Assignments?" and "Show Problems on Finished Tests".

So it is, thank you!

esternin commented 3 years ago

@esternin I took the bait and took a look. From what I think I understood, I don't think one can currently determine whether hints are enabled from inside a problem (from PG code). From what I seem to understand from looking at code in pg and webwork2, the webwork2 layer only passes to PG a setting called displayHintsQ (whether the hints should be shown or not) but none of the "reasons" why it was set. ...

  • displayHintsQ is set in by the defineProblemEnvir routine of lib/WeBWorK/PG.pm from the value of $translationOptions->{showHints} and the value of numOfAttempts is passed to the PG side of things also in the same manner.
  • Warning: The $showHints variable in PG has a totally different meaning, related to the attempts before hints are shown, and there is also a "flag" called showHints for MathObjects.

It took me a while to digest this, @taniwallach. Thanks very much for trying, and I think your conclusion is: one cannot.

The reason I want it is to make certain text conditional. If hints are disabled, do not show this: "If you do not answer this question correctly in $showHint attempts, you can get a hint".

$translationOptions->{showHints} returns nothing. I also tried $showHints, but I think you just misspelled $showHint.

However, $displayHintsQ returns 0 on the same problem, if it is a part of a gateway quiz where hints are disabled, and 1 if it is in the homework where hints are enabled. The library browser also sets it to 0, so you do not see the text in the list of rendered problems in the preview, but behaves differently when you "Try it", depending on what is the target set defined in the top entry field of the Library browser. 

if ($displayHintsQ == 1) {
  $hintString =  "<b>If you do not answer this question correctly in $showHint attempts, you can get a hint.</b>";
  }
Context() -> texStrings;
BEGIN_TEXT
$hintString
$PAR
...

Is this a valid solution or a fluke?

drlvk commented 3 years ago

Specifically, the Manual Grader page does not appear to be available when an essay problem is used within a Gateway Quiz. Originally brought up in a forum

taniwallach commented 3 years ago

Farming out rendering to back-end standalone renderers to allow better "front end" scale to allow more students to make requests in parallel to a WW server. If rendering was handled by a "render cluster" (of containers) whose size could either be manually or automatically scaled - it is likely that a "regular" webwork server could support larger quizes. This is probably a major redesign task - but would hopefully provide for far better capabilities of the WW system to support load-spikes from quizzes and exams.

See discussion at: https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4961

I'm copying my reply from that thread below:

Given that problems are typically customized per student with random parameters - and that the "seed" range is pretty large, I suspect pre-computing and storing cached "rendered" problems would be both CPU and storage intensive, and it would not help once answers are being submitted. I suspect that load peaks occur both at the start of a quiz (many students opening the quiz in a short time period) and at the end (many submissions very close to to the deadline).

I think that the issue is the current "single build in" PG renderer per Apache process, as rendering each PG file takes time, and causes the memory ballooning issue which the current generation of the WW server software experiences.

I think that the goal should be to improve WW so that it can easily provide more or less "power" without requiring provisioning a fixed large amount of resources based on expected peak loads. Anyone running in the cloud or with access to a Kubernetes cluster or the like would hopefully be able to then provision for "typical" demand as a standard, but either schedule or "auto-scale" for higher demand when necessary.

I think that the best long-term direction will be to separate the "PG renderer" out of the Apache process, using the new "standalone render" or something similar. Hopefully that will allow somewhat parallel processing of multi-problem rendering requests - or at least provide a means to better control the memory issues which probably is the bigger problem than the CPU resource issue. (Less memory ballooning would allow more Apache processes to run, so many more students could be served in parallel.)

This is something which is not likely to be ready for WW 2.16, but could be a major goal for development in the near future.

taniwallach commented 3 years ago

From a discussion in the forums:

Ideas from Larry Riddle, @esternin, @dlglin

awmorp commented 3 years ago

There's some discussion on this here . Might be easier to implement for a gateway quiz than a homework set?

dlglin commented 3 years ago

From https://webwork.maa.org/moodle/mod/forum/discuss.php?d=5571 (Jaimos Skriletz)

I have been working with WeBWorK Gateway Quizzes (as have many due to remote teaching), and think they could use some improvement to make it clear when students start a quiz by clicking on the assignment name. I think that just clicking on the quiz name and having a new version created is a bit unintuitive (I have had students for a timed quiz on their own time click and start the timed version before they were ready). I have also had to use question 1 to be a set of quiz information/rules for students since I can't display a quiz header. I think it would be beneficial to have GatewayQuizzes (both normal and proctored) have a launch/info page that provides info about the quiz, including the setHeader that could be used to provide information/instructions about the quiz. My basic thought is when a quiz is clicked on a page comes up that contains the following information: • Quiz name. • setHeader under the quiz name in the main body of the page (this could be used for quiz instructions/information/etc). • Course Information to the right (currently the LoginProctor shows site_info and think course info is more appropriate). • A list of the quiz conditions, such as time limit (if timed), due date, open date (if it isn't opened), how often new versions can be taken and so on. • A list of current versions (if any) that have been taken, so students could just go back to a current version instead of starting a new version (if it is opened), or see grades/scores (This would be basically like the list of versions from the homework sets page). • A start new version button (if they are able to start a new version -- also maybe a comment that this starts a new version, to continue a previous version click the appropriate version above). • Login/Proctor information for a proctored quiz, similar to LoginProctor for proctored quizzes. I have looked at LoginProctor.pm as maybe a starting point to see if I could help, but quickly found myself over my head. Anyways, I think I've seen mention of such a thing elsewhere, but can't seem to find it, and though I would add my thoughts here. It would be nice to have something like this in 2.16, but I doubt there is time for that, but it would be something I would like to use on my server to provide more information about quizzes, and make it clearer about when students open/start a new version. I think this may also help deal with students accidentally starting another version if they get confused which link to click on the the homework list page.

As an aside, it would also be nice to have an easier way to edit specific version information for a student (this can be done but it is a bit hard to get to the specific version compared to just the quiz), and delete versions (that get accidentally created).

awmorp commented 2 years ago

Another feature request for gateway quizzes:

In Webwork 2.16, there is nothing to stop a student accidentally closing or navigating away from an active gateway quiz, which can result in data loss. A simple way to address this would be to use a javascript onbeforeunload event handler to trigger a warning alert before closing/reloading/navigating away from the page.

However, the event handler would need some logic to detect when the student clicked Grade Test or navigated between test pages, which should not trigger a warning.

ofloveandhate commented 1 year ago

I want to follow on to @esternin 's post above, working towards text that appears conditionally on whether hints are shown. My experience teaching with Webwork has led me to believe that I should tell students that there's a hint; a hint magically showing up after a number of attempts without telling them that it will leads to students entering garbage N times until they give up or a hint appears. So I just tell them that I'm giving them a hint. But, once the hint shows up, they no longer need the text telling them a hint will appears after N attempts. So, here's what I've arrived at:

# I put this in a macro file, included via pgCourse.pl
sub theresahint {
 my $hintafter = $showHint+1;

my $hintReminder = '';
if ($numOfAttempts<$hintafter && $displayHintsQ == 1){
    $hintReminder = '(This problem has a hint that shows after ' . $hintafter . ' attempts.)';}

$hintReminder;
};

And then I include this in the problem text via

\{ theresahint \}

or if in PGML, I use

[@ theresahint @]*

Depending on how an instructor views the problem, it appears differently. But it appears to work correctly for students, so I'm going with it.

drgrice1 commented 1 year ago

@ofloveandhate: Your code will cease to work with WeBWorK/PG 2.18. The $showHint variable will no longer be defined, and won't do anything. If and when hints are shown will be controlled by a webwork2 setting and will be configurable per problem by the instructor.

Also, your post is off topic for this conversation about gateway quizzes. Hints are never shown in gateway quizzes. They aren't even implemented for gateway quizzes. So you should open a separate issue for this.

ofloveandhate commented 1 year ago

@drgrice1 Thanks for the followup, good to know that $showHint will be removed in 2.18. Would you please add this to the 2.18 release notes?

Alex-Jordan commented 1 year ago

One suggestion mentioned here is about the set header in a test (aka quiz). In 2.18, when I first go to a quiz and see the "Start New Test" button, I see the set header in the usual place off to the right in a panel.

Do we currently have a way to show the set header on all pages while actually doing the quiz? If not, are we open to that? I'm not sure how people use set headers on tests, but I would typically want to put some kind of instructions or reference information that I'd want students to have constant access to. Locally, people are putting that kind of stuff in "question 1" instead. But for that use case it would be nice either have it off to the right on a wide screen, or maybe up at the top. And I guess up a the top on small screen.