Testing flexx.ui

[almarklein]

Related to #12 (testing visual output).

We need tests to verify that our widgets work correctly. But testing this kind of stuff is hard and can lead to tests that are painful to maintain.

What we want from these tests:

• A: To verify that Flexx ui works on the browsers that we promise to support
• B: To verify that Flexx more or less looks as we intended
• C: To verify that the widgets behave as intended
• D: To verify that a combination of widgets interact in the way that we intended

I think we might need a mix of automated and manual tests. Automated tests are great, because they prevent regressions and take no human time to run. They (can) suck because they have to be maintained. Manual tests can suffice for some cases, but people get lazy, and its unpractical to do manual tests on all browsers.

Some ideas:

• There are several small apps in the docs. We can collect these in a single gallery-like page, that we can look over (and interact with) in a few different browsers to have a quick overview whether at least the basics work. This is easy to do, so we should definitely do it. This covers B, and to lesser extent A, C, D.
• We can write mini-apps with a short "task" (<5 s) that the user has to perform to "finish". These mini-apps can be loaded one-by-one in a tester app. That way you have visual and functional tests that are manual, but are quite fast, yet force the user to do well. This covers B, C, D (maybe A if other people help out). The individual tests should definitely be dead-easy to write and maintain.
• Mmm, maybe a similar thing that shows the doc-examples one by one.
• Test visual output for layouts using CI on a range of browsers (as mentioned in #12). I think this would be very useful to get A (although it would not test all widgets).
• We have quite complete tests in flexx.event and flexx.app that cover D to some degree.
• Selenium tests to test the working of each widget. Would test B, and also A if we run it on Saucelabs. Would this effort be worth it if we have the task-like-apps-test?

(Marking for v0.6. I want to get some of this in v0.6, but some stuff might be for v0.7).

[Korijn]

Would it be helpful if I contribute our LiveServerThread code?

[almarklein]

Sure! :)