search

CodeGrade / bottlenose

CS assignment / grade mangement system
GNU Affero General Public License v3.0
38 stars 21 forks source link

Option to anonymize grading #92

Open blerner opened 6 years ago

blerner commented 6 years ago

Courses should have an option (settable by the professor) that when grading homeworks, graders should not be shown the names of the students being graded.

NatTuck commented 6 years ago

It looked like grader autoassignments are deterministic, so this feature would depend on randomizing assignments.

blerner commented 6 years ago

Hmm? Line 76 of the grader allocations controller takes all the submissions that don't yet have a human grader allocated to them, randomizes the last, and then allocates submissions to human graders. So it's not deterministic...

NatTuck commented 6 years ago

I got anecdotal reports but never looked into it.

blerner commented 6 years ago

I'll double check on some random students' grades, to see how much of a spread they had among who graded them. But I'm at least somewhat willing to bet there's a pigeonhole principle in effect here (500+ students, 25 graders, 1-2 dozen assignments ==> definitely likely to be repeats of who grades whom). In any case, the whole point of anonymizing the display of a homework to the person grading it, is that they cannot know who they're grading, so that any unconscious bias of "darn, not this student again!" goes away... 😇

This is definitely tricky to get right, since there are at least a few places where graders can see who they're grading. (Offhand: the course index page that shows their grading assignments, links from a code review to the underlying submission, and TAs can also see the grading allocations page itself. Possibly some more.)

kylesferrazza commented 5 years ago

Some places where students' names appear while grading (note to self):

blerner commented 5 years ago

Thinking aloud: we need some way of producing a stable but anonymous identifier per student per assignment, so that graders can see that they need to grade "Team: Student 5 and Student 42", but not know what names those numbers match up to. We would need to hash the assignment and students together, but to prevent a brute-force attack to invert the hash, it would need to be using some detail that graders can't access. I suggest hashing together the assignment creation-time timestamp with the student ids, and modding down to some smallish range of numbers. The creation time is guaranteed to be immutable per assignment, and is never exposed in the UI anywhere.

Self-evals make things slightly tricky, since the hashing will be different per assignment and so the names won't quite line up...