search

data-8 / notebook-ok-extension

Jupyter notebook extension for simple mode and optional ok integration
Apache License 2.0
1 stars 1 forks source link

Running tests takes a long time when a notebook is compute-intensive #13

Closed henryem closed 8 years ago

henryem commented 8 years ago

Currently the way we write labs and projects is: sprinkle questions throughout the assignment for students to fill in; put a hidden test cell after each question; students run all the tests by clicking a button (or use a trick devised by Sam to run individual tests).

The issue with this is that each test cell must run the entire notebook before running its tests. Some of our notebooks are unavoidably compute-intensive (especially the projects, which use large datasets), so each individual test set can take minutes to run. This is very frustrating to students. It has been a major pain point for several labs and both projects this semester. I can't imagine troubleshooting issues related to this for the 400 less-tech-savvy students we anticipate having next semester.

I understand that there isn't an easy way to fix this, but I think it's worth redoing the way we run tests to get this right. Here are the four things that I think are technically possible:

  1. Have the "run OK tests" button run a single test process and then run all of the tests. Instead of having a shell command in the notebook's OK test cells, we could just have a magic string that indicates to the test framework that certain test files are associated with that cell.
  2. Associate with each question cell in the notebook a button that runs only the tests for that cell. 2'. Instead, the button could run only the code up to that point, which would also fix #11. 2''. Even more ideally, the button would not create a new process at all, but rather run the associated test code inside the notebook's running kernel, which would be much faster. It would also be less confusing for students, since there would be no discrepancy between test results and their own kernel's state. This would not populate a submission file. We could require students to click "run OK tests" when they are done with the lab, and that would generate their final submission file as it does now.
papajohn commented 8 years ago

Plan is to have ok tests run in the current environment without reloading everything.