Make nbgrader use symlinks for everything except student submission

UBC-DSCI / rudaux

LMS + External Tool Workflow Orchestration

https://ubc-dsci.github.io/rudaux

MIT License

7 stars 5 forks source link

Make nbgrader use symlinks for everything except student submission #109

Open trevorcampbell opened 3 years ago

trevorcampbell commented 3 years ago

We often run into a problem where a problem set has a big data file associated with it (e.g. 10MB). When rudaux collects assignments it just collects notebooks; but when nbgrader runs it makes duplicates of entire assignment folders. So if you have 300 students, that 10MB becomes 3GB (for each of autograded and feedback folders).

We should prevent nbgrader from copying that stuff over and over again. When rudaux clones the repo, the first thing it should do is move all the associated data out of the folder, and replace every file with a symlink. I think that when nbgrader runs, it'll copy symlinks over, which will be super cheap.

trevorcampbell commented 3 years ago

NB: we should probably do this when we're working on #95

trevorcampbell commented 3 years ago

Maybe we can get this to happen in nbgrader itself? It may actually not be a massive change...

See for the autograde step: https://github.com/jupyter/nbgrader/blob/f5729fb2b6638ea2c6f9b0f9226a0dd7c8cb2ad5/nbgrader/converters/autograde.py#L120

And for the feedback step: the nbgrader converter: https://github.com/jupyter/nbgrader/blob/f5729fb2b6638ea2c6f9b0f9226a0dd7c8cb2ad5/nbgrader/converters/generate_feedback.py

but that doesn't actually copy anything, that's handled in the base class:

https://github.com/jupyter/nbgrader/blob/f5729fb2b6638ea2c6f9b0f9226a0dd7c8cb2ad5/nbgrader/converters/base.py#L229

trevorcampbell commented 3 years ago

Can use this to create links: https://www.geeksforgeeks.org/python-os-symlink-method/

trevorcampbell commented 3 years ago

An issue on the main nbgrader thread about this:

https://github.com/jupyter/nbgrader/issues/1414

trevorcampbell commented 3 years ago

Should have rudaux do two things:

when creating the grader accounts, set everything in the git repo to "read only" (except the nbgrader DB / any other important files that the graders need to change)
for any redundant/duplicate files created during autograding, have rudaux replace them with symlinks.

I'll still leave this for beyond V1.0 since it's a potentially tricky change that isn't on our critical path.