Integration tests using ipython notebooks

[jhprinz]

Hi,

I have added a python module that can run ipython notebooks in travis and make some rather nice output. The code is found here: https://github.com/jhprinz/msm-tis/blob/ipythontesting/examples/ipython/ipnbdoctest.py The main features are:

• run a complete ipynb and check if all cell run without error.
• If there is an error the traceback is written to the travis output.
• There is also an additional timeout per cell.
• requires conda install ipython and conda install ipython-notebook where I think the latter is dependent on the first
• the output is somehow similar to what nose tests do
• automatic travis code folding of error tracebacks to clean up the output
• optionally (not yet activated) one can check if the output generated is identical to the one already present in the notebook (including images!)
• UUIDs and other runtime dependent outputs are ignored

• Finally Cell can terminate with

  1. ok : passed and the outputs are identical
  2. DIFF: passed but the output is different (treated as okay)
  3. TIME: timeout occurred (right now 9min < 10min of travis, also treated as okay)
  4. FAIL : cell could be executed in ipython but python reported an error
  5. ERROR : cell could not even be executed (e.g. Timeout or kernal failure)

  if FAIL or ERROR occure the test exits with an error

We can also put code folds into .travis.yml scripts simply using

echo travis_fold:start:{name}
...
echo travis_fold:end:{name}

although name MUST be unique in the whole output!

The code is based on some gist (see the code) and I tried to quickly patch something, so the code needs some cleanup, but you might get the idea. If there is interest in this I will make this more useable with seom commandline options to be used in travis.

[Lnaden]

Having more helpful travis output would be good. Probably won't get into 1.0. Moving to future release