Allow to re-open reviews / submissions for individuals

[justinpombrio]

FYI, this is a major pain point in Brown cs 173.

Justin (grad TA)

On Tue, Sep 9, 2014 at 6:31 AM, Joseph Collard notifications@github.com wrote:

— Reply to this email directly or view it on GitHub https://github.com/brownplt/admiral-edu/issues/26.

[jcollard]

Added ability to re-open reviews here: https://github.com/brownplt/admiral-edu/commit/b78bf59278553173c1f846ff971914e619125d33

[jcollard]

@justinpombrio @jpolitz @arjunguha @shriram @kfisler

Reopening submissions for individuals who have not received any reviews is easy. However, if a student has already received feedback this becomes tricky as the inline comments may not line up anymore. Also, there is a question on what to do with the students previous submission. I would like to get input on what you as instructors would like.

Here are two options I've considered:

Overwrite Existing Submission

A student is allowed to resubmit work and it simply overwrites the existing submission. This may cause existing reviews to be inconsistent depending on the extent of the changes; a message on the feedback page explains this.

Pros:

• Assigned reviews that have not been completed will see the new submission.
• Exporting an assignment remains the same (each student has a single submission for each step).

Cons:

• We lose the original student submission
• Previously completed reviews may be inconsistent

Submission Versions

In this scenario, we have a submission version number that increments each time a student resubmits their work.

Reviews are associated with a particular version number and the feedback remains consistent.

Pros:

• We keep all of the submission data
• Previous reviews remain consistent

Cons:

• Exporting an assignment may include multiple submissions for each student.
• It is not clear how re-submissions should be assigned reviews.

[jpolitz]

I think versions is a good idea. For our courses, we usually only want resubmission on the final step; we don't want students doing multiple initial submissions. If they do need to resubmit an earlier step, then I think we'd be happy with always assigning the first version to review, and just storing the updated submissions for download. That way, if a student needs to go back and fix something for a technicality, they can, but it doesn't affect the review process at all.

[kfisler]

I would prefer submission versions, because I want to be able to grade reviews (and maintaining the line numbers would help a lot there).

I would not send resubmissions out for review. I see resubmission mainly as a vehicle to handle students who forget to include a file in their upload, students who need to fix syntax errors, and other mistakes like this. I would imagine asking course staff to scan the resubmission to make sure it was within an acceptable delta to the original.

For purposes of writing grading scripts, however, I'd like any edits to filenames to occur on the previous versions. Leave the filenames intact on the most recent submissions. Moving previous submissions to a subdirectory would be fine with me (not sure what interactions you need to consider with the scripts that send materials out for review here).

This feature would be fantastic to have, btw -- it's been a little painful handling this manually this term.

Kathi

On Thu, Oct 2, 2014 at 11:26 AM, Joseph Collard notifications@github.com wrote:

@justinpombrio https://github.com/justinpombrio @jpolitz https://github.com/jpolitz @arjunguha https://github.com/arjunguha @shriram https://github.com/shriram @kfisler https://github.com/kfisler

Reopening submissions for individuals who have not received any reviews is easy. However, if a student has already received feedback this becomes tricky as the inline comments may not line up anymore. Also, there is a question on what to do with the students previous submission. I would like to get input on what you as instructors would like.

Here are two options I've considered: Overwrite Existing Submission

A student is allowed to resubmit work and it simply overwrites the existing submission. This may cause existing reviews to be inconsistent depending on the extent of the changes; a message on the feedback page explains this.

Pros:

• Assigned reviews that have not been completed will see the new submission.
• Exporting an assignment remains the same (each student has a single submission for each step).

Cons:

• We lose the original student submission
• Previously completed reviews may be inconsistent

Submission Versions

In this scenario, we have a submission version number that increments each time a student resubmits their work.

Reviews are associated with a particular version number and the feedback remains consistent.

Pros:

• We keep all of the submission data
• Previous reviews remain consistent

Cons:

• Exporting an assignment may include multiple submissions for each student.
• It is uncertain how re-submissions should be assigned reviews.

— Reply to this email directly or view it on GitHub https://github.com/brownplt/admiral-edu/issues/26#issuecomment-57647123.

[blerner]

FWIW, WebCAT handles multiple versions in a reasonably useful way: there's a UI to select from a list of previously submitted versions. (It then proceeds to gank up the UI with poorly handled state transitions, but that's separate...) The nice part about this is it lets graders compare older versions if they want to see progress over the course of the assignment, and it lets students re-submit after getting feedback, but still see the feedback attached to the relevant version of the code.

One minor refinement I'd like to see in the data model: for various reasons students may be late to an assignment. It would be nice to be able to re-open an assignment for a student and mark the subsequent submission as "treat this as on-time". (I don't mean this in some treat-Admiral-Edu-as-an-autograder sense, but just as a way to record relevant metadata about that student's submissions, that are of use later when actually grading the assignment and you've forgotten which special exemptions you've granted...)

[jcollard]

@blerner

A tagging feature where you could add arbitrary tags and search / sort by them would probably accomplish what you're describing.

[blerner]

@jcollard, that could indeed model the "treat as on-time" scenario. As a UI matter, I view it as part of version management and assignment re-opening, though. 1) I'd like that kind of tag to apply only to a specific revision (so I could tell a student, submit what you have and I'll count it as on time, but submit it again after that and I'll mark it late) 2) I'd like to have a UI to say "apply this #tag to this student's next submission", rather than have to babysit that submission and then remember to apply that tag. Or relatedly, 3) I'd like to re-open an assignment for a student and give them a dont'-count-as-late extension of X amount of time. (That in particular has been a pain point for me in Fundies 2 this semester several times...) Or something along these lines.

Feel free to triage these ideas and/or split them off into other feature requests as convenient for you.

[jcollard]

Some thought will need to go into that idea. Earlier we had decided that CT would be mostly ignorant of the class: doesn't know about deadlines, doesn't know about the thing being submitted, etc. Our intent is not to replace class room management tools like moodle but to provide a tool to facilitate a review process.

However, I think it is important to be flexible and a discussion of these features is important.