pbrosnan
commented
9 months ago
### Summary:
Students with higher scores on assignments in an assignment group can get lower assignment group scores.
### Steps to reproduce:
1. Make assignment group called Quiz with rules to drop the lowest 2 and highest 1 assignments in the group.
2. Add 4 assignments to the group named Q1, Q2, Q3, and Q4 worth 100, 91, 55 and 38 points respectively.
3. Enter grades as in the assignment entries in the following table (the 5 leftmost columns). The entries in the rightmost column are the assignment group grades, which will be calculated by Canvas LMS.
| Student Name | Q1/100 | Q2/91 | Q3/55 | Q4/38 | Quiz |
|--------------|--------|-------|-------|-------|-------|
| Albert | 100 | 42 | 14 | 1 | 46.2% |
| Bob | 100 | 42 | 14 | 2 | 46.2% |
| Carl | 100 | 42 | 14 | 3 | 7.9% |
| David | 100 | 42 | 14 | 4 | 10.5% |
### Expected behavior:
The students with higher scores on the assignments should not get lower scores on the assignment group. In particular, Carl's Quiz score should be at least as high as Albert and Bob's.
### Actual behavior:
When the instructor fills out the above Quiz grades for Q1-4 in the Canvas gradebook, Canvas computes the Quiz averages in the right-hand column.
### Additional notes:
The root of the problem is the algorithm used to compute the assignment group grade in [AssignmentGroupGradeCalculator.ts](https://github.com/instructure/canvas-lms/blob/master/ui/shared/grading/AssignmentGroupGradeCalculator.ts). This algorithm, which was adapted from the paper [Dropping Lowest Grades](https://cseweb.ucsd.edu/~dakane/droplowest.pdf) by Kane and Kane, works as expected when the instructor is either dropping only lowest grades or only highest grades. It also works as expected when the assignment weights are all equal.[^1]
[^1]: See also the earlier paper [Choosing Subsets with Maximum Weighted Average](https://doi.org/10.1006/jagm.1996.0849) by Eppstein and Hirschberg, which gives a similar algorithm along with several other apparently more efficient ones.
#### Undesirable dependence on order of dropping grades:
The issue above arises when an instructor is dropping the L lowest grades and H highest grades, where L and H are both positive integers and the assignment weights are not all equal. In this case, which was not treated in the article by Kane and Kane cited above, the program in [AssignmentGroupGradeCalculator.ts](https://github.com/instructure/canvas-lms/blob/master/ui/shared/grading/AssignmentGroupGradeCalculator.ts) first drops the L lowest grades and then the H highest grades.[^2]
[^2]: See the function dropPointed on line 164 of the file in commit:26f4dabb03e9e5d695cda85553ff9083f33ae562
Dropping the grades in this order can produce a different result from dropping the highest grades before the lowest grades. For example, in the case of Carl's grades above, Carl's Quiz score would be a 42/91, or approximately 46%, if Canvas dropped the highest grade before dropping the lowest grades.[^3]
[^3]:The values for Carl's grades were taken from Table 3 in the paper by Kane and Kane.
This behavior where the result of dropping the grades depends on the order is problematic by itself, but it is also related to the more problematic behavior of the group grade decreasing when the student's actual scores increase illustrated by the table above.
### Recommendation:
Exactly how to best fix this issue is a subjective question, but my suggestion is that Canvas take both of the following steps together.
1. Canvas LMS should give an option that allows for instructors to compute assignment group grades by percentages, i.e., for assignment grades to be be computed with assignments equally weighted even if the assignments differ in their maximal points possible. This would add new functionality to Canvas LMS and be a valuable change in and of itself.
2. When an instructor drops only highest grades or only lowest grades in a group, the instructor should be free to calculate assignment group grades by weights (as it is done now) or by percentages. However, when an instructor chooses to drop both highest and lowest grades in a group, calculating group grades by percentages should be the only option.
Calculating group grades by percentages would eliminate the problem illustrated by the example above. For example, in the table above, Carl's Percentages are as follows.
| Quiz | Q1 | Q2 | Q3 | Q4 |
|------------|-----|-----|----|----|
| Percentage | 100 | 46 | 25 | 8 |
If we were to calculate Carl's assignment group grade by percentages, dropping the lowest 2 percentage grades and the highest 1, his score would be the score on Q2, 46%. And, naturally, you would get the same result regardless of whether you drop the highest grade first or the lowest two grades first. I think this is what an instructor would intuitively think Canvas should do. (If we were to drop no grades for Carl, his score would be the average of the 4 percentages, or 71%.)
One might argue that an instructor could simply give the assignments equal weights by making them have the same possible number of points if he or she wants the group grade to be calculated by percentages. However, in some situations, the weights of the assignments are forced on the instructor. (This has been the case for me with assignments created by external tools.) In other cases, having assignments with different maximal scores may be convenient for numerical reasons. For example an assignment with 6 questions might be graded out of 90 points because 90 is divisible by 6, while an assignment with 5 questions might be graded out of 100 points. But an instructor might still want to count these two assignments with equal weights in the group grade.
From my discussion on order [above](#order), it might seem possible to fix the issue by simply switching the order in which Canvas drops grades, dropping the highest scores before dropping the lowest ones instead of the other way around. As you can see from the discussion above, this would fix the problem with Carl's grades. But, while I have not tested this method, I suspect that it would have many of the same undesirable issues discussed here.
Of course, another possible way to fix the above problem would be to just stop allowing instructors to drop highest grades. Certainly, use of this option must be exceedingly rare. Still, if there are instructors using the option now, it is hard to imagine discontinuing it.
### Conclusion
The way assignment group grades are computed now in Canvas is incorrect because it can result in a situation where students who get better (or equal) grades on all assignments wind up with lower asssignment group grades. While choosing a way to fix this issue is subjective, I have presented a suggestion which fixes the problem and also implements a new desirable functionality to Canvas LMS.