c-cube / qcheck

QuickCheck inspired property-based testing for OCaml.
https://c-cube.github.io/qcheck/
BSD 2-Clause "Simplified" License
347 stars 36 forks source link

Interrupt long shrinking #37

Open Gbury opened 7 years ago

Gbury commented 7 years ago

Sometimes shrinking can take a long time, so a way to stop shrinking after some time and/or a number of failed tests would be nice to have. The new events added in my last PR should help do that in QCheck_runner without having to modify the code in QCheck.

c-cube commented 7 years ago

I think it's necessary to modify a bit the code in qcheck, because if we interrupt shrinking after some total amount of time / if some step takes too long, we still want to keep the partially shrunk value. A custom exception for interrupting shrinking should do.

Gbury commented 7 years ago

Well, ideally, giving the event hook a meaningful return value would probably be cleaner, but probably a lot more complex (since the return value would depend on the event, it would probably require a gadt), so I agree an exception to stop shrinking is a good idea. (I'm actually worried about having to setup an exception handler for each potential counter-example tested, but it probably doesn't matter very much).