Doing integration (front and backend) tests using the notebook

[maartenbreddels]

I was thinking how to do proper testing for ipyvolume, but the same holds for ipywidgets, so I thought of bringing the discussion here. I think unittests in javascript/typescript are not an issue, Python the same. But at some moment, you want to check if things really work as expected from the notebook, hence an integration test. I asked @blink1073 how they've done it for jupyterlab, and they use karma+mocha, which uses a real browser. This is perfect for me, since for ipyvolume I need to have a WebGL context, which I believe is not supported in something like PhantomJS.

I had the following idea in mind. You write your tests in a notebook, once you save it, karma (or webpack) will detect a file change, and will do the webpack/karma dance, including the json from the notebook (possibly minus the outputs) and include it in a test. Then karma will execute the tests and our custom code (mostly coming from the jupyterlab project) will attempt to executing all the code coming from the notebook. You could possibly get code coverage from JS and Python in the same run as well.

This will require 'stealing' some code from the jupyterlab project (actually, on the long run it will probably need to go in a seperate package). For instance run-test.py seems useful.

I don't have the time or knowledge to set this all up myself (I don't have any experience with the Jupyterlab TS code). However, @jasongrout, do you think this is something you have time for before (and for) the release of ipywidgets 7.0? I'm happy to contribute where I can, and after 7.0 we can see how we can polish it, and put it maybe in a separate package.

[jasongrout]

Let's look at doing something with the testing infrastructure after we do a beta (hopefully this week or next). I'm putting this in the Future milestone for right now since it seems quite ambitious.

[StefanBrand]

I have a simple use-case at hand. In an integration test with papermill, how can I figure out from the backend whether an IntSlider will be displayed as widget? E.g. I'd like to verify whether my environment can display the slider like this:

The plain text representation would cause the test to fail:

(There is an Exception opening new comm in Javascript:)

[jtpio]

FYI this PR adds visual regression testing to ipywidgets using galata: https://github.com/jupyter-widgets/ipywidgets/pull/3172