custom stopping criteria for tests

[ghilesZ]

Hi,

I can't find a way in the documentation to stop a test when a given criterion is met. Is there an way to do that using the current API?

Ideally i would like to have an additional parameter to the function Test.make*, let say a predicate ?until: () -> bool, that would be called at each iteration, an if it returns true, stop the testing. This could be useful in many cases, eg, if the function under test has a behavior that depends on an external environment, or if we want to test a function until its coverage is more than a threshold etc.

The Test.make function would have as a signature something like:

val make : 
  ?if_assumptions_fail:([ `Fatal | `Warning ] * float) ->
  ?count:int ->
  ?long_factor:int ->
  ?max_gen:int ->
  ?max_fail:int ->
  ?small:( 'a -> int ) ->
  ?retries:int ->
  ?name:string ->
  ?until:(unit -> bool)
  'a arbitrary ->
  ( 'a -> bool ) ->
  t

I am willing to participate to the development of such a feature if you think it's a good idea.

[jmid]

Here's a couple of half-baked ideas to achieve something like that:

• By raising a suitable exception upon meeting the criteria and wrapping a try-catch of around Test.check_* to catch the test error
• By expressing the criteria as a precondition with assume (or ==> - but beware it evaluates both sides eagerly in CBV OCaml). This approach will have the drawback of repeatedly trying to satisfy the criteria before finally giving up.

Interface-wise, rather than an impure function unit -> bool I suppose one could also extend QCheck with a dedicated exception Stop_criteria_met and a handler and message for that particular exception in the test driver loop. That should be mostly free for QCheck users who don't use the feature.

From 10.000 ft, If I'm interested in maximizing coverage, rather than extending a blackbox PBT framework, I would probably recommend a "graybox" PBT framework such as Crowbar: https://github.com/stedolan/crowbar. It has a similar QuickCheck-like interface, but underneath the hood the AFL fuzz engine takes coverage into account when generating input test data. As such, I would prefer to better understand a concrete QCheck use case first :thinking: