Manual grading in live notebooks

[ellisonbg]

After doing lots of manual grading of notebooks, I have found that doing so in the static HTML nbgrader web app doesn't work. A few issues I ran into:

• I found that I often wanted a new comment space in a location where the nbgrader web app didn't put one.
• Many comment areas in the nbgrader web app went unused, making it difficult to navigate with all of that extra space.
• I also found myself adding code comments in student code cells so I could be very specific like # <---- this line has an index error.
• I ended up rerunning student code most of the time. It was very unusual to manually grade a notebook were I didn't rerun something.

Because of this I ended up doing all of my manual grading in the live notebook with a modified nbgrader JS extension that allowed me to simple set the manual grade score (also showed the autograde score). This worked very well.

[ellisonbg]

I should mention that we do have plans to improve the performance of loading large notebooks (should be much better already in master) and to add per cell comments. This should further alleviate the need to maintain a separate web app for nbgrader.

[jhamrick]

I found that I often wanted a new comment space in a location where the nbgrader web app didn't put one. Many comment areas in the nbgrader web app went unused, making it difficult to navigate with all of that extra space.

I would really like to have the ability to just insert comments wherever in between cells... I think this probably wouldn't be too hard to add to the current HTML grader, though is definitely more predisposed to the live notebook.

I also found myself adding code comments in student code cells so I could be very specific like # <---- this line has an index error.

Hmmm, I hadn't thought of that use case before. That would definitely be nice to have, though.

I should mention that we do have plans to improve the performance of loading large notebooks (should be much better already in master) and to add per cell comments.

Having that would be great, but at least right now that's not feasible for a class size of 200. Currently it takes about 6-7 seconds to load the notebooks for my class locally (plus 1-2 seconds to load the kernel, and an additional 1-2 seconds if it's remote), which would mean at least 30 minutes of extra grading time per problem just waiting for the notebooks to load (and that's assuming you grade the whole problem at once, which usually isn't true for us, because it's easier to grade all of one part of the problem, then all of the next part, etc.)

Because of this I ended up doing all of my manual grading in the live notebook with a modified nbgrader JS extension that allowed me to simple set the manual grade score (also showed the autograde score). This worked very well.

Could you make a PR with your extension? I don't think we'll be able to use it for my class this semester unless the notebook load time is significantly faster, but I'd really like to try it out!

[ellisonbg]

Yeah, the load time is pretty painful for a class of 200 students. I do think that until we have those times down much lower it is very reasonable to maintain the nbgrader HTML web app. Especially if you are grading the notebooks in sections like that.

I will submit a PR with the nbgrader extension changes.

[jhamrick]

I just had a thought about the notebook loading times. I think if there were two things possible:

• if the notebook could be loaded cell-by-cell, rather than all at once, and
• if there could be a config option (or maybe just the extension could do this) to not automatically start kernels -- just start it on demand

Then this would significantly decrease load times and definitely make it possible to do grading in the live notebook. Obviously, these aren't changes (especially the first one) that are feasible to do right now, but I just wanted to write my thoughts down somewhere.

Also, when you were doing grading in the notebook, was it just saving into the notebook metadata, or was it also communicating with the database?

[ellisonbg]

We have definitely been thinking about these two optimizations for performance (lazy loading of cells and delayed kernel starting) and it would help a huge amount. These will almost surely be required when we start to work more on the real time collaboration as well.

When I was doing manual grading, I did everything through the modified grading cell toolbar that I wrote. I just saved the metadata in the notebook and a later "nbgrader record" run would put that into the database.

On Thu, Jan 22, 2015 at 5:19 PM, Jessica B. Hamrick < notifications@github.com> wrote:

I just had a thought about the notebook loading times. I think if there were two things possible:

• if the notebook could be loaded cell-by-cell, rather than all at once, and
• if there could be a config option (or maybe just the extension could do this) to not automatically start kernels -- just start it on demand

Then this would significantly decrease load times and definitely make it possible to do grading in the live notebook. Obviously, these aren't changes (especially the first one) that are feasible to do right now, but I just wanted to write my thoughts down somewhere.

Also, when you were doing grading in the notebook, was it just saving into the notebook metadata, or was it also communicating with the database?

— Reply to this email directly or view it on GitHub https://github.com/jupyter/nbgrader/issues/67#issuecomment-71132263.

Brian E. Granger Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[jhamrick]

@ellisonbg commented in #604:

The core client UI of formgrade could be replace in many situation by a live notebook with the following:

• An nbextension that allow human grading and commenting and stores the results in nbgrader metadata.
• A new nbgrader command line phase that takes that metadata and stores it in the db. We already have nbconvert preprocessors that essentially do this, so this would be super simple.
• Some sort of additional frontend UI that makes it easy to navigate through notebooks for grading.

In JupyterLab, offering additional UIs is relatively easy. I don't have a ton of time right now to work on this stuff, but I may pick away at it.

[jhamrick]

While I think it would be really nice to do grading in the live notebook, I'm still not convinced it is something that can really replace formgrader right now.

Currently, the formgrader stores the results in the database (through the use of a REST api) rather than in the notebook itself. This is really a critical feature for several reasons:

• The database is critical because it allows instructors to easily get out grades in, for example, a CSV format. To do this takes only a few seconds under the current implementation, if you had to go through all the notebooks and extract the metadata from them this could easily take 10 minutes if you have a large class.
• If you keep the database but also want to store metadata in the notebooks, then (as you mentioned) this introduces an additional step of having to run something that goes through all the notebooks, after formgrading, and pulls out the grades and saves them in the database. Instructors then have to remember to run this step every time they make any changes when grading.
• Not having information saved directly in the database (i.e., while grading) makes it really difficult to determine what notebooks have or have not been graded. A very commonly used feature of the formgrader is to go through and grade all of one part of a notebook, then the next part of the same notebook, and so on. This means that you have a bunch of notebooks that are all in a partial state of grading and the formgrader makes it easy to identify where you need pick up where you left off. This would be difficult to present if all of the information is only stored in the metadata, again because you'd have to iterate through all the notebooks and pull out the metadata every time you want to show that information.
• The formgrader presents a lot of really useful information that will pretty much be impossible to display without a database being updated in realtime. For example, you can see what the average grade on each problem is, how many students failed the test cases, whether any assignments have been "flagged" (e.g. for cheating or other reasons), etc. These could still be displayed in some UI drawing from the database, but again if you have to run a separate command to update the database, then this information will frequently be out-of-date. I forsee a lot of instructors forgetting to run the command to refresh the database.

Those critiques aside, I'm happy to explore the idea of doing live grading, and maybe we could even offer a "lightweight" version of the grading interface that works in the live notebook. But, I don't think we can retire the formgrader entirely.

[ellisonbg]

Good points:

• Fully agree with having the grade information in the db. I have notebooks that compute final grades from the gradebook. Wouldn't want to give that up.
• For me my grading process is already strongly driven by the command line, so having another command wouldn't really bother me (but see my next pont)
• Yes, it really helps to see what notebooks have been graded. For that I was thinking about a serverextension for the single user notebook that provides the REST API for form grading logic. That would also allow live grading to be written directly to the db without a separate command line pass.
• Any additional frontend UI for formgrading could also be provided as additional pages to the single user server or juptyerlab. It could integrate with or sit along side the live notebooks.

The point is that if you can work within the (extended) bounds of the single user notebook server (or jupyterlab) all of the deployment problems vanish. But all of this is predicated on being able to host single user servers that have group access permissions.

[jhamrick]

For me my grading process is already strongly driven by the command line, so having another command wouldn't really bother me (but see my next pont)

Some feedback that I've gotten about nbgrader from other instructors is that there is too much going on with the command line already, so I'm hesitant to introduce more commands that are required (I'm ok with adding additional commands that are optional, e.g. nbgrader export)

For that I was thinking about a serverextension for the single user notebook that provides the REST API for form grading logic. That would also allow live grading to be written directly to the db without a separate command line pass.

Ah, yes, if we could still have that, then my concerns would be alleviated. It would still require re-writing a lot of the UI functionality to work as a plugin with the notebook, but if we can keep the REST API for the grading stuff in the backend then that will at least keep the core functionality the same.

[ellisonbg]

Classic notebook serverextensions can offer additional HTML pages at URLs the notebook isn't using. Couldn't we move the existing formgrade UI to a URL prefix like /grading on the single user server and have it serve up the REST API and existing formgrade pages? Then the classic notebook and formgrade pages would be on a single server and having formgrade pages link out to the live notebooks would be trivial. That single serverextension could also provided the needed REST endpoints for the assignment-list nbextension.

Then creating a formgrade instance would simply be:

• Create a shared single notebook server for each course as a JupyterHub service (@minrk is working on that). In cases where there is a single instructor/grader person all rolled into one, this step is even optional.
• Installing nbgrader, enabling its extensions and writing an nbgrader config file for that server.

[ellisonbg]

From the implementation perspective that would require:

• Removing entirely the auth handling in nbgrader/formgrade. All of the auth would be handled by the existing auth of the single notebook server.
• Bundling the existing formgrade/assignment handlers into a single user serverextension.

[jhamrick]

Classic notebook serverextensions can offer additional HTML pages at URLs the notebook isn't using. Couldn't we move the existing formgrade UI to a URL prefix like /grading on the single user server and have it serve up the REST API and existing formgrade pages?

Yes, I actually have something like that already to make it work with the binder demo:

https://github.com/jhamrick/nbgrader-demo/blob/master/formgrade_extension.py

Bundling the existing formgrade/assignment handlers into a single user serverextension.

I am still a little concerned about the loading times that we discussed when this issue was originally opened. Do you know if that's been addressed at all?

[ellisonbg]

Yes and no. In the JupyterLab beta we are doing a lot to make the rendering of notebooks faster. But the solution I am starting to think about wouldn't face that slowdown. I am imagining not using actual live kernel backed notebooks for grading, but rather just host the existing formgrade app as a serverextension. No notebook loading times would be relevant for now as it would still use the formgrade UI/backend.

[ellisonbg]

But it sounds like this type of implementation might simplify things in a huge way if you already have much of it working.

[jhamrick]

Ah, yes, I see. In that case then yeah, I have implemented almost everything we'd need to do already. The only thing is that there are no links between the notebook interface and the formgrader, so people have to know what URL to include explicitly.

This is the auth that I wrote for the notebook extension:

https://github.com/jhamrick/nbgrader-demo/blob/master/notebookauth.py

So if we can get this setup to work with JupyterHub as well then we could just get rid of all the auth stuff, and put the notebook auth as being the one (and only) way of linking the formgrader up with the live notebook server.

[ellisonbg]

When you say "no links between the notebook interface and the formgrader" what do you mean? The formgrade UI could link directly to live notebook. Or are you thinking about links from the main notebook UI over to the formgrade pages?

On Thu, Jan 5, 2017 at 4:31 PM, Jessica B. Hamrick <notifications@github.com

wrote:

Ah, yes, I see. In that case then yeah, I have implemented almost everything we'd need to do already. The only thing is that there are no links between the notebook interface and the formgrader, so people have to know what URL to include explicitly.

This is the auth that I wrote for the notebook extension:

https://github.com/jhamrick/nbgrader-demo/blob/master/notebookauth.py

So if we can get this setup to work with JupyterHub as well then we could just get rid of all the auth stuff, and put the notebook auth as being the one (and only) way of linking the formgrader up with the live notebook server.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbgrader/issues/67#issuecomment-270789050, or mute the thread https://github.com/notifications/unsubscribe-auth/AABr0GjXr1ZIFu4yjAeLt3AMt1w9Ztz7ks5rPX1mgaJpZM4DU-bl .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[jhamrick]

Sorry, I just mean there is nothing currently in the UI (the way I've implemented it) that links to the formgrader. But yes, that is certainly something that could be (and probably should be) added.