Add a test suite

[deeplook]

As ipyleaflet evolves it seems more and more important to add a proper test suite to prevent nasty errors. Of course, this will be non-trivial for testing output created with JS. But I've done some experiments using Selenium and Voilà, and it seems at least worth trying.

[martinRenou]

We could at least check that all Notebooks are running properly with nbval, that might be a good first step

[deeplook]

I've looked briefly at nbval, which gives me something like this:

$ pytest -v --nbval A*.ipynb
============================== test session starts ==============================
[...]
plugins: mypy-0.7.0, flake8-1.0.6, rerunfailures-9.1, nbval-0.9.6, cov-2.10.1
collected 15 items

AntPath.ipynb::Cell 0 PASSED                                                                           [  6%]
AntPath.ipynb::Cell 1 PASSED                                                                           [ 13%]
AntPath.ipynb::Cell 2 PASSED                                                                           [ 20%]
AntPath.ipynb::Cell 3 PASSED                                                                           [ 26%]
AntPath.ipynb::Cell 4 PASSED                                                                           [ 33%]
AntPath.ipynb::Cell 5 PASSED                                                                           [ 40%]
AntPath.ipynb::Cell 6 PASSED                                                                           [ 46%]
AntPath.ipynb::Cell 7 PASSED                                                                           [ 53%]
AntPath.ipynb::Cell 8 PASSED                                                                           [ 60%]
AwesomeIcons.ipynb::Cell 0 PASSED                                                                      [ 66%]
AwesomeIcons.ipynb::Cell 1 PASSED                                                                      [ 73%]
AwesomeIcons.ipynb::Cell 2 PASSED                                                                      [ 80%]
AwesomeIcons.ipynb::Cell 3 PASSED                                                                      [ 86%]
AwesomeIcons.ipynb::Cell 4 PASSED                                                                      [ 93%]
AwesomeIcons.ipynb::Cell 5 PASSED                                                                      [100%]

If I run it on all example notebooks it barfs with: py.error.EMFILE: [Too many open files]: getcwd(), though.

[martinRenou]

Is there a way for nbval to not consider each cell to be a test but the entire Notebook? Maybe it would not open so many files in that case.

[deeplook]

I think the purpose is precisely to consider each cell a test. According to https://github.com/computationalmodelling/nbval/issues/154, not yet.

According to https://pypi.org/project/nbval/ there is some parallel mode together with pytest-xdist. But you can always run it over subsets of all examples (now 40). I can run [A-L]*.ipynb and [M-Z]*.ipynb separately without the error locally.

In any case quite a few notebooks would need to have cells marked to be executed or not. But we could of course start with a subset known to work.