Nbtesting

[darribas]

Re-arranging the notebook testing framework and including all of the relevant files.

[darribas]

Latest commit allows all the notebooks to be run and tested but it only collects the filename of the notebook that fails, not where or how. The type of output that we get now is:

(pydata_exp)kokopelli:nbtest dani$ nosetests nbtest.py 
E
======================================================================
ERROR: Runs a series of test notebooks and compares them to their actual output
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dani/code/pysal_notebooks/nbtest/nbtest.py", line 113, in test_run_notebooks
    raise Exception('\n\nThe following notebook comparisons failed:\n\n{}\n\n'.format(fail_list))
Exception: 

The following notebook comparisons failed:

/Users/dani/code/pysal_notebooks/nbtest/../notebooks/intro_scicomp_python.ipynb
/Users/dani/code/pysal_notebooks/nbtest/../notebooks/PySAL_esda.ipynb
/Users/dani/code/pysal_notebooks/nbtest/../notebooks/PySAL_modules.ipynb
/Users/dani/code/pysal_notebooks/nbtest/../notebooks/PySAL_weights.ipynb

-------------------- >> begin captured logging << --------------------
root: INFO: Executing notebook with kernel: python2
root: INFO: Executing notebook with kernel: python2
root: INFO: Executing notebook with kernel: python2
root: INFO: Executing notebook with kernel: python2
root: INFO: Executing notebook with kernel: python2
root: INFO: Executing notebook with kernel: python2
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 31.724s

FAILED (errors=1)
(pydata_exp)kokopelli:nbtest dani$ 

[sjsrey]

@darribas was this meant for nbtesting or master?

[darribas]

@sjsrey It was really still experimenting. It's at a stage now where it'll run all the tests the notebooks folder (only on the current branch) as a single unittest. If all tests pass, it'll pass. Otherwise it'll fail. Those notebooks that fail will be listed, but it'll still not show up the error that made them fail.

I think the next steps would be:

1. Make it print the error by catching the traceback message when it fails. Currently in only picks up the path to the notebook failing.
2. Hook it up to Travis so any time a commit is made to master, everything gets tested.
3. Extend to all branches.

I hadn't thought about it but, with the branch approach we have to include new material, we might have to include the tester in each branch so they're tested. Any more practical approaches? @carsonfarmer @ljwolf ?