Improve test suite runner to fail on errors and track test execution

nicksay commented 8 years ago

This change updates the test suite runner (scripts/crunner.py):

Output test results similarly to python -m unittest discover, tracking the number of tests executed and their time
Buffers test failures during execution and outputs traceback information for each at the end to make finding errors faster
Replaces the -t option with --skip-test, making running the test the default behavior
Replaces the -q option with --verbose and --debug for increasing levels of information, making the flags consistent with the compiler options
Improves the --debug option to also show template output in addition to source tree information
Provides defaults for --test-input and --test-output and makes the values relative the to the working directory to allow running tests in other locations in the future

Closes #37

awbraunstein commented 8 years ago

Lol. The tests don't pass so this can't be merged..

lgtm, but we probably need to either fix or disable those tests.

nicksay commented 8 years ago

So strange. This passes on Mac Python 2.7.10 and not Linux Python 2.7.6:

Mac:

$ uname -sr; python -V; make tests &> /dev/null && echo pass || echo fail
Darwin 15.2.0
Python 2.7.10
pass

Linux:

$ uname -sr; python -V; make tests &> /dev/null && echo pass || echo fail
Linux 3.13.0-67-generic
Python 2.7.6
fail

However, it looks to me like the test suite is failing on master as well. I think #39 will fix this.

nicksay commented 8 years ago

I've rebased onto #39 and tests pass now. Once #39 is in, I'll rebase back on master to remove the extra commit.

awbraunstein commented 8 years ago

lgtm

youtube / spitfire