Try to achieve better coverage

[gergelypolonkai]

This is a WIP, but before I continue I’d like to hear your opinion.

Test coverage level is very low right now, and I was about to increase it a bit (I managed to up it by around 40% by now). This messed up everything at one level, though, and I’m currently working on some tox black magic to make it pass even though the Django test suite will obviously fail. Do you think it worths it? Don’t hesitate to say “no” if you feel so, it’s only cca. 50 minutes work up until now ;)

[Geekfish]

Hey @gergelypolonkai I think there is real value to the coverage goal, plus the current tests don't really test the runner explicitly!

I'm also wondering what the most efficient way to go would be.

Even with extended test cases, we basically just check if the test suite blows up or not, which is useful, but doesn't tell us if the output and behaviour of the runner is really what we expect.

We could possibly have 2 test suites, one that is the actual tests for the runner, plus the one we already have as part of a sample app. So basically something like what django does for testing their test DiscoverRunner - but we could also add output capture (or something, to check the counter / ordering / output etc works). It feels like a less-magical way to do this.

[Geekfish]

^ it would mean that we don't need magic tox: we would run the tests and then coverage once (for each django version etc). The runner test suite would call the runner in different configurations from within the tests.

[gergelypolonkai]

Thanks for the idea on Django’s own internal tests, I didn’t think about that one.

Right now I simply added a bunch of test cases to the test app, and for thorough testing, I think they should remain. The problem here is that TextTestRunner.run() returns non-zero because there was an error, which coverage (and because of that, tox) will report as an InvocationError.