search

jupyter / nbgrader

A system for assigning and grading notebooks
https://nbgrader.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.29k stars 317 forks source link

Previewing answers when manually grading #1155

Open psychemedia opened 5 years ago

psychemedia commented 5 years ago

When creating an assignment, a sample answer can be provided within a predefined comment block:

### BEGIN SOLUTION
...
### END SOLUTION

When the assignment is prepared for release to students, the answer is stripped out.

The example solution is useful for checking the answerability of the stated question, and generating and checking the autograder tests, but appears not to be used at any other time.

For manual grading, it would be useful if:

In terms of managing and making use of canned example solutions, would it make more sense if they was a cell type defined for "example solution" that could be more easily removed from and reinjected into a notebook?

I've also started wondering whether it would be easy to work with nbgrader if notebooks were split into and recreated from the database.

(This is how the original "read only" behaviour appears to have been implemented?)

A couple of possible related items in this respect:

Sefriol commented 5 years ago

Just to comment about this idea:

Elaborating more on the first point: I think the most useful feature for manual grading would be an easy way to launch a temporary copy of the instructor and/or student's notebook (i.e. it would be destroyed after execution has stopped). This way you could test and compare students solution towards the instructor version in cases where just looking the code is not enough.

Though, I would say that most of the UI related issues could be solved by moving nbgrader into JupyterLab. And that will most likely create many other problems.

jhamrick commented 5 years ago

Just a quick comment on this:

an easy way to launch a temporary copy of the instructor and/or student's notebook

You can launch a read-only version of the student's notebook in the actual notebook during manual grading by clicking the hyperlink with the name of the notebook at the top of the formgrader.

It probably wouldn't be too hard to have a similar link to open the instructor version of the assignment.

On Sun, Jun 30, 2019 at 11:31 PM Joakim notifications@github.com wrote:

Just to comment about this idea:

  • Usually even the original model solution is not enough. Some solutions are visual or hard to compare when just looking into what student has written. Therefore, I would say that providing a hyperlink / a way to easily access the instructor notebook is the best way to solve this issue.
  • In addition, usually instructor versions are necessarily not meant to be example solutions. They might be done to have better optimization for the calculation and at the same time harder to read for those who not fully understand more advanced python/library concepts.

Elaborating more on the first point: I think the most useful feature for manual grading would be an easy way to launch a temporary copy of the instructor and/or student's notebook (i.e. it would be destroyed after execution has stopped). This way you could test and compare students solution towards the instructor version in cases where just looking the code is not enough.

Though, I would say that most of the UI related issues could be solved by moving nbgrader into JupyterLab. And that will most likely create many other problems.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbgrader/issues/1155?email_source=notifications&email_token=AAAUL5ATIUOMXTYIXUK56ZDP5EX37A5CNFSM4HXB2ZG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY4VB3A#issuecomment-507072748, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAUL5CSDKB6UQIQS7X4SW3P5EX37ANCNFSM4HXB2ZGQ .

Sefriol commented 5 years ago

Yep, but usually the read-only version won't be that much of a difference. It's enough for simple exercises, but more complicated ones, you usually need to test part of the student code and maybe put prints in between to check whether student has had the right idea.

Sometimes it also happens that student implementations pass preliminary test but cause a failure later on. Usually then, you need to copy the implementation done later on and put it on the instructor notebook to see whether it also fails there with correct preliminary implementations.

Of course some of these can be fixed by doing better tests, but sometimes you just cannot imagine how students will solve certain exercises and your tests will just fail you.

What I tend to do currently, is just use local git copy of the instructor notebook, copy paste certain cells / parts of the code done by student and just git stash when I'm done.

jhamrick commented 5 years ago

Yep, but usually the read-only version won't be that much of a difference. It's enough for simple exercises, but more complicated ones, you usually need to test part of the student code and maybe put prints in between to check whether student has had the right idea.

Yes, you can do this in what I mentioned---it's only "read-only" in that you can't save the changes to disk but you can still edit it and execute the changes (i.e. it's a temporary notebook, as you were requesting before).

What I tend to do currently, is just use local git copy of the instructor notebook, copy paste certain cells / parts of the code done by student and just git stash when I'm done.

It would indeed be interesting to be able to launch a live, temporary version of the instructor notebook---i.e. have it be copied to a temporary location and then opened in the notebook. I am not sure what the best way is to ensure that it gets cleaned up, though, as you wouldn't be able to put it directly in /tmp as the notebook server doesn't have access to that directory. I suppose it could be in a hidden directory somewhere in the nbgrader directory that periodically gets cleaned out or something.

On Mon, Jul 1, 2019 at 4:28 PM Joakim notifications@github.com wrote:

Yep, but usually the read-only version won't be that much of a difference. It's enough for simple exercises, but more complicated ones, you usually need to test part of the student code and maybe put prints in between to check whether student has had the right idea.

Sometimes it also happens that student implementations pass preliminary test but cause a failure later on. Usually then, you need to copy the implementation done later on and put it on the instructor notebook to see whether it also fails there with correct preliminary implementations.

Of course some of these can be fixed by doing better tests, but sometimes you just cannot imagine how students will solve certain exercises and your tests will just fail you.

What I tend to do currently, is just use local git copy of the instructor notebook, copy paste certain cells / parts of the code done by student and just git stash when I'm done.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbgrader/issues/1155?email_source=notifications&email_token=AAAUL5G77HD3MNKEOBRRR23P5IPCTA5CNFSM4HXB2ZG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY6PXRA#issuecomment-507313092, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAUL5AJ6YZXQSZV3ZN4X73P5IPCTANCNFSM4HXB2ZGQ .

psychemedia commented 5 years ago

Trying to think around the problem more generally in case it suggests a solution by roundabout means: what if the instructor notebook was launched into a temporary Jupyter notebook server? It doesn't matter what you do with it then? But it would require a temporary notebook server.

psychemedia commented 5 years ago

There are a couple of extensions that might be interesting when it comes to finding ways of comparing specimen and student answers:

The workflow I imagine is annotating student notebooks with marker rubrics, specimen answers, etc, so that a marker can look up marking guide info within the student notebook context without having to open another notebook etc.