Replace points with percentage in rubrics

[jonatanschroeder]

Extracted from #8858.

@mwest1066: The main downside of using points for all three things above is that it's very non-orthogonal. That is, everything ends up tangled together which makes it hard to just drop-in questions into assessments or to reallocate weights between questions. For example, if we have a question that is worth 12 points that requires three key steps to solve, we might have a rubric which allocates 4 points to each step. If we now decide that we want to re-weight this question to only be worth 10 points on the assessment, we have to go back and edit the rubric to change the points for each item.

I think we could live with this situation on a single assessment, given that it's no worse than what people always did for pen-and-paper exams, but this is especially bad because it hurts question re-use and sharing. At some point soon we want to allow rubric sharing. At the moment this will be a big problem because an instructor might use a shared question but assign it a very different number of points on the assessment. If they copy the rubric with the shared question then they'd have to adjust the points for all the rubric items, or we could do it automatically and end up with items work 3.3252 points and things like that.

[solution] Have all rubric items use percentages only. An alternative would be to allow arbitrary weights to be used and to take the total weight as the sum of the weights for the (positive) rubric items. Or we could allow instructors to explicitly set the total weight.

[issues] 4. How problematic will it be for instructors to set rubric credit values in percentages rather than in points? For positive rubric items I think it's actually easier -- you just think about the breakdown in terms of 100 percentage points always, but folks who are very used to points may feel differently here. [issues] 5. How complicated would the backwards compatibility story be for this? I think that for rubric items it's not too bad, especially because that's still in closed beta. We could just calculate percentages for rubric items as part of the transition because everything's in the DB and we don't have to maintain compatibility with JSON config files. For manualPerc I think it's also fairly easy -- we could allow autoPoints and manualPoints just as we do now for backwards compatibility, while using points and manualPerc going forward.

@jonatanschroeder: I'd prefer the latter (the explicit total weight), defaulting to the current question's points value (manual or total depending on if the rubric settings override auto points or not). Percentages can be clunky and hard to think about, particularly for negative rubric items, though I can be convinced otherwise. In either case, since there is a clear total of weights (either the explicit value or 100 for percentages), this would make the rubric easily transferable.

[backwards compatibility] For rubrics, we can add a maximum weight field that for now defaults to the assessment question's points if null, then have a batched migration that updates it with the assessment question's points, and after this is completed and deployed, change the interface to allow custom total weights.

@mwest1066: I think the advantage of percentages is that you don't need to reference a total to understand what's going on, because it's always the same scale. The disadvantage, as you point out, is that the numbers are bigger and so calculations with them can be more confusing. For example, if I have a question with three key steps in it, in a pen-and-paper exam I might choose to make it worth 12 points, with 4 points per step, which has nice easy math and also allows partial credit per step. With percentages I'd probably use something like 30/30/40 or 33/33/34, which is (slightly) less convenient to think about.

I'm not sure I understand the issue with percentages and negative rubric items? To me it seems equally simple to say that it's -2 points out of 12 for an error, or -15 points out of 100? Of course we still have the issue that the numbers are bigger for percentages, but I tend to work around this by using multiples of 10 and 5.

The two key questions we need to address here are:

1. Should the total weight be configurable by the instructor, and, if so:
2. What should the default total weight be?

The three obvious choices seem to me to be:

1. Configurable total weight, default to question point value.
2. Configurable total weight, default to 100.
3. Non-configurable total set to 100.

My current preference is for option (3), although I could get on board with (2) if people think that there is a convincing reason that instructors want to be able to change the total weight. I dislike option (1) because I fear that instructors will mentally associate the total weight with the question points and then be surprised if they change one of them and the other one doesn't change. I think that it's bad UI to have two value that appear to be the same thing but are in fact separate. I think that option (2) is less sensitive to this because it would be clear to the instructor that they are manually making two things the same and so it's on them to keep them the same if they change either value.

One nice property of (2) is that it allows instructors to mimic pen-and-paper point setups (where the question weight in the assessment is the same as the total weight inside the question). However, I actually think that this might be a disadvantage of (2) because it allows people to do things the "wrong way" that they are used to from pen-and-paper exams. I'm torn here between giving people flexibility versus trying to push them to do things a better way.

@jonatanschroeder: I'm ok with option 2. Option 3 can be an issue in examples like you mentioned, where you want the weight to be a multiple of 3 or 7. We can make the custom weight an option that is "hidden" behind a radio ("percentage" vs "custom weight") to discourage it.

[jonatanschroeder]

There is some consideration to apply here that I'd love to hear some thoughts, particularly from @mwest1066 and @nwalters512. In essence, the question is, when the rubric total weight is 100 (to simplify the discussion), what does a total of 100 represent? Here are the scenarios:

Scenario	Manual points	Auto points	Rubric applies to	Rubric reaches 100 means
1.	10	0	N/A	10 points
2.	10	20	Total (replace auto points)	30 points
3.	10	20	Manual (keep auto points)	10 points
4.	0	20	Total (replace auto points)	20 points
5.	0	20	Manual (keep auto points)	20 points (???)
6.	0	0	N/A	???

My question is mostly scenarios 5 and 6 above. Scenario 5 can be used to apply a rubric that applies additional penalties (or bonus points) to an auto-graded question that add up to the result of the auto-grader itself. In this case, does the rubric value apply to the total, even though the auto-grader points are retained?

Scenario 6 is a question with no assigned points. In this case, is it safe to assume that a rubric cannot have any points (not even penalties or bonuses that apply to the assessment as a whole)?

[jonatanschroeder]

Another question: what should be shown to students in the feedback panel for the points associated to a rubric item? Points as assigned by the rubric, points as applied to the question/assessment, or percentages?