Timed assignments

[magsol]

I'd like to explore the possibility of adding timed assignments in nbgrader. Specifically, within a certain timeframe--

• Students would fetch the assignment, starting an internal timer
• Each assignment would have an "expiration" time attached to it, e.g. 3 hours from fetch
• After this expiration time, nbgrader will no longer allow the students to perform new submissions

This could be used in a "take-home exam" format, or even along the lines of a Google Codejam-style coding competition, where once a student actually begins the assignment they have only a short window in which to finish it before their ability to submit that assignment is disabled.

A quick-and-dirty way of implementing this would be to freeze the exchange directory (even revoke write permissions) for certain users after a certain time frame, but that wouldn't scale too well and would require resetting the permissions for the next assignment. Something more robust would likely require an optional argument for the assignments in nbgrader_config to specify their expiration time limit, and some kind of logging within nbgrader to enforce the time limit when a student attempts to submit.

[jhamrick]

Hmm. I can see the use case, but I'm not sure if it is within the scope of nbgrader to handle something like this.

I think probably the easiest way to do it would be to have some way to mark the time when an assignment is fetched within the notebook metadata, and then rather than preventing submissions, apply late penalties based on that recorded timestamp. But, implementing that in a way that prevents students from modifying the timestamp would be tricky -- currently the way these types of things are prevented is through storing the relevant information into a database when the instructor runs nbgrader assign, but nbgrader fetch doesn't have access to the database.

In any case, if you want to propose something more specifically about how you would envision it working, I'm open to persuasion, but I do think this is probably a feature that is going to be quite difficult to implement robustly, which I am wary of.

[magsol]

That's perfectly fair. I can jury-rig an extension of some sort to record the time when students fetch the assignments, though if the fetch subcommand doesn't have access to the database then that potentially makes things difficult.

On the other hand, I just discovered the timestamp.txt file included with collected assignments--does this record the last submission from the student? If so, I could potentially do something similar with fetch time--just create a text file in the exchange folder with the fetch timestamp and perhaps have a listener script on the instructor end siphon them up as they appear.

Would that be a viable approach?

[jhamrick]

Yes, I think something like that could work.

I would recommend probably implementing this as a separate extension/wrapper around nbgrader first and see how it goes, and if it works well and there is demand for it we can talk about integrating it into the core 😄

[magsol]

Just to gauge things: how much of a brutal hack would it be to simply add a couple of lines of file I/O logic to the copy_files method of fetchapp.py that will record the timestamp and write it to a file in the exchange folder?

I'd have a script (cron? something) running on the instructor end, listening for any timestamped files that show up in the exchange directory, to ward off the possibility of multiple fetches clobbering timestamps that were already written.

[jhamrick]

Yeah, I think that would be fine. You could have it save multiple timestamps too so they don't get clobbered by multiple fetches.

[angadhn]

@magsol if you successfully implemented this, could you share instructions with the rest of us about this? Would love to implement this for our large classes (350 students).

[magsol]

@angadhn I put this repo together a few years back with the scripts / changes you're looking for: https://github.com/eds-uga/nbgrader-scripts

Keep in mind it's two years old at this point so may be slightly out of date. It's also not super robust--I only ever used it for classes with 20-30 students, so e.g. the tamper-"proofing" of the timestamps was never actually triggered, and so I can't say for sure that it works or even works well--so YMMV.

Still, I'm happy to help out with it if needed! PRs / tickets are welcome :)

[perllaghu]

This may be solvable using a bespoke external exchange plugin: you subclass the exchangefetchassignment class (see https://nbgrader.readthedocs.io/en/stable/exchange/exchange_api.html#exchangefetchassignment) and replace the copy_files method.

In there, you can add logic to note who fetches what, when.

You could also subclass ExchangeSubmit, and replace the copy_files method to block submissions past a certain time.

Alternatively, or in parallel, you could subclass ExchangeCollect.copy_files() to not collect late submissions.