Wish list for regression testing

[kaa-ching]

The current regression framework (after issue #225) still leaves a lot to be desired. I'd like to see the following options:

• [ ] allow skipping of levels
• [ ] allow super-fast-forward to reduce testing time
• [ ] take everything out of the Makefile: automatically run on all levels in levels.xml
• [ ] keep going to test other levels when regression fails
• [ ] show statistics whilst running
• [ ] make it more user friendly (not using asserts to convey messages)

Feel free to suggest more things, then we'll prioritize.

[Wuzzy2]

• [ ] More straight-forward testing: Place objects, start simulation, wait, go on with next test. All automated.

[Wuzzy2]

take everything out of the Makefile: automatically run on all levels in levels.xml

I would change that to “take everything out of the Makefile: automatically run on all levels in finished”, when hints have become a QA requirement. Playable levels and levels with hints are different things.

Also, we might add some test levels which are solely designed for regression testing (since the normal levels might not cover all corner cases) and put them into an extra directory regression. Then all levels in that directory are automatically included as well.

More wishes:

• [ ] Support setting phone number (very important!)
• [ ] Special treatment of levels without anything in the toolbox and without detonator boxes: Here it only does one try and succeeds if won
• [ ] If level has been won / failed, continue with next test immediately instead of waiting until the timer runs out

Note: The second wish fits on the first level which currently would be a false-negative in a regression test because you win without placing anything. The assumption is that levels like this must be “demonstration” levels of some sort, not to solve some particular puzzle. Also, it is very stupid we can't regression test our first level, imagine a TBE release in which the very first level is broken. :D