A way to timeout tests without failing.

Consider an algorithm with good average case complexity, but bad worst-case one. (QuickSort with its O (n · log n) runtime on average but worst-case O (n²) would be an example; in the actual application I'm working on right now, it's actually more like quadratic vs. exponential!)

So, if we QuickCheck this it'll usually work just fine and, well, quickly, even for long lists – but sometimes QuickCheck will generate a sorted or mostly-sorted one, which dramatically slows down the runtime. This means the test suite has a very nondeterministic runtime – in my case, the test in question usually does 500 runs in 0.1 seconds, but then sometimes it takes 10 seconds for just a single run, and gigabytes of memory. This is not all that bad while I'm working on my own machine, but for CI it does pose a problem when there are sporadic failures due to timeout or memory quota.

I don't really want to manually restrict the inputs to avoid triggering this behaviour in the first place (it's actually tricky to see what inputs are problematic, unlike in the QuickSort example).

What I can of course do is adding a timeout in QuickCheck itself. Currently, within is exported for that purpose – but again, it'll mean there will be sporadic failures of the test suite, although the tested code doesn't have a bug but just follows expected behaviour of the algorithm.

The rejectOnTimeout I'm proposing does instead only reject such a test case, so it is possible to ignore these very long-running instances without changing anything else about the test coverage.

nick8325 / quickcheck

A way to timeout tests without failing. #332