Test visual output - grab browser screenshot (also on CI)

[almarklein]

We should have tests to validate visual output of widgets and apps. These should be runnable both locally and on Travis. We'd want to be able to run the tests for each browser that we support. On CI it might be ok to do only FF, but would be nice to do all.

General options:

• Selenium webdriver is sort of what flexx.webruntime does, but with more control. First time I tried it did not catch FF. Needs exes to be on PATH. Easy support for grabbing screenshots. For chrome it needs a chromedriver.
• PhantomJS is a webkit-based headless browser.
• I can make a webkit browser with high control via PyQt.
• On Linux (and maybe also OSX) I can grab a screenshot from any process that I have the pid of using imagemagix. This could be a good quick way to allow testing on browsers locally.
• On Windows I can do the same (via ctypes).

CI options:

• Travis comes with FF installed and xvfb.
• Appveyor probably has IE.
• Chrome?
• Saucelabs is a service where you can test all OS/browser combos (also mobile). Free for OSS. Can be linked to Travis CI. (pip install from repo to get py3k compat). Uses Selenium to remotely run the VM (run from Travis).
• There is also browserstack.com (free for OSS)
• applitools allows visual testing (semi-automated)

Eventually we'd want to test for several browsers in CI using saucelabs. For the short term, we should aim for a solution that can be run locally and also on Travis.

Approach:

• In the main repo we have a directory with .py files that each describe a static app
• These apps are exported to HTML and published via a git repo. The commit is tagged to make it corresponding to the commit of the main repo.
• We run browser tests on these apps, and compare the results with images in yet another repo. If not on Travis, the system will prompt the user if a new app is present or if a comparison fails. The user can then mark the result as good or bad. Once a good image is present for each app, the images-repo is committed, and tagged with the same commit-id.

[almarklein]

Note that we added some for of live test in #94. No screen captures, but still an actual connection with a browser, and some communication between the two.

[almarklein]

I am feeling less interested in applying many visual tests, for two main reasons:

• You'd be testing how a certain app looks. So if Phosphor changes its CSS a bit, or if we do in Flexx, we need to fix the test, update screenshots etc. In other words, these tests are likely a pain to maintain.
• You'd only be testing how a certain app looks statically; the test does not even say that much.

Therefore, I think that the visual tests should be a small test suite, which draws a few layouts with widgets in different colors, so that we can verify on a range of browsers whether the basic layout elements (HBox, HPanel, Splitter, DockPanel) work. Also no screenshots, just test that the amount of e.g. red in the image makes sense.