Add notebook testing setup

[huard]

Description

In Raven we're using notebooks as both documentation material and integration tests.

After unit tests are run, the notebooks are executed and compared with their expected output using the pytest plugin nbval.

These tests are run on Travis-CI to make sure notebooks used as documentation reflect the current software state.

Here are some make commands that we've created in Raven (room for improvement):

test_nb:
    @echo "Running notebook-based tests"
    @bash -c "source $(ANACONDA_HOME)/bin/activate $(CONDA_ENV);env RAVEN_WPS_URL=$(RAVEN_WPS_URL) FLYINGPIGEON_WPS_URL=$(FLYINGPIGEON_WPS_URL) pytest --nbval $(CURDIR)/docs/source/notebooks/ --sanitize-with $(CURDIR)/docs/source/output_sanitize.cfg --ignore $(CURDIR)/docs/source/notebooks/.ipynb_checkpoints"

notebook:
    @echo "Running notebook server"
    @bash -c "source $(ANACONDA_HOME)/bin/activate $(CONDA_ENV);env RAVEN_WPS_URL=$(RAVEN_WPS_URL) FLYINGPIGEON_WPS_URL=$(FLYINGPIGEON_WPS_URL) jupyter notebook $(CURDIR)/docs/source/notebooks/"

Another useful command would be to run and save all notebooks to refresh the expected output.

I think this model would be useful to extend to other birds.

[huard]

An nice improvement would be a mechanism to bulk update the notebooks output. When packages are updated, sometimes the outputs change, but it's not a bug. So we usually want to update the notebooks to reflect their expected output in a given environment. At the moment I do this manually:

• fire a jupyter notebook server in the target environment
• run the cells from all the notebooks and save the output

Using jupyter nbconvert this could be automated. I could imagine for example a make command refresh_notebook that opens a conda client environment and update the notebook. The testing using nbval would be done in the same environment.

[Zeitsperre]

@huard This looks doable. Should I look into a way of performing this in a bird-agnostic way? or should there be some recipes that are only valid for some birds / services (thinking of Raven)?

[huard]

Yes, you'll see examples of this in existing birds. The objective would be to standardize this through the cookie-cutter template. I think these basic services are generic:

• test notebooks
• refresh notebooks
• launch local notebook server (with environment variables)