Integration with Scalacheck.

[sageserpent-open]

Write an implementation of a Scalacheck Prop that takes a Trials[X] instance and an X => Result, paralleling how Scalacheck integrates its own Gen[X] instances into the framework. This would allow a simple cutover from Gen[X] with its attendant shrinkage problems to Trials[X] where shrinkage comes for free.

[sageserpent-open]

Looking through Scalacheck, there is Test.check that embodies a loop potentially replicated in several threads that applies a Prop iteratively to instances taken from a sequence of Gen.Parameters whose seed attribute changes on each call.

This provides an outer loop of test case generation - the generation of a test case and its use in a test are encapsulated in a Prop implementation. Typically, this will be furnished by Prop.forAllShrink, which builds a Prop instance from a test lambda, a Gen that can create a test case given a seed, and some shrinkage function that reduces an existing test case to something simpler.

The shrinkage function is put to use within the application of Prop - so shrinkage takes places as recursion below the level of Test.check, which regards the application is atomic - it provides a Gen.Parameters and it gets a Result which contains either a positive outcome or a minimised test case as a result of the internal shrinkage.

So a way into Scalacheck for a Trials instance is to write another Prop factory method as a syntax enhancement to Prop; this would take a test case consumer lambda, just as Trials.withLimits().supplyTo does, and one or more Trials instances, thus paralleling the existing forAll overloads in the Prop companion object.

An alternative would be to make this a syntax enhancement in Trials, and thus reuse the existing way of joining Trials instances together with and / or. In effect, Trials.forAll(<consumer>)...

[sageserpent-open]

Let's try a spike - cook up a Trials with an interesting invariant and some difficult shrinkage, write some syntax enhancement, and try writing a Scalacheck test with the corresponding Gen and then cut it over to Trials. What do they yield? (Assuming this to be feasible).

The interpreter buried in the cases method in the anonymous implementation of SupplyToSyntax needs to be extracted so that it can be called from the Prop implementation.

Likewise, the shrinkage pseudo-recursion also needs to be extracted so that the inner shrinkage loop in classic Scalacheck can be realised by it.

[sageserpent-open]

There is a problem - Scalacheck keeps counts of the number of successful and discarded test separately in the implementation of Test.check from anything that happens in Prop.apply. So how do we communicate the cases limit - which would seem to naturally derived from Gen.Parameters.minSuccessfulTests?

[sageserpent-open]

One rather ungainly solution would be to add an extra parameter to the Prop factory that specifies a cases limit applicable only to shrinking.

Another, horrible, approach would be to cache the number of cases examined while Scalacheck is performing its outer discovery loop, then to use this while shrinking.

So either the Prop implementation holds the cache - and therefore becomes secretly mutable - and thus we hope that there is a way of telling when a new loop cycle has been started - maybe via examination of the seed values, or this gets pushed down into TrialsImplementation (again with the same issues about mutability).

I'm not keen on butchering TrialsImplementation for the sake of conforming to Scalacheck's implementation constraints, so any cache will have to go into the Prop implementation.

Read 'em and weep....

[sageserpent-open]

Given #42, it may be acceptable - if somewhat hokey - to go with Scalacheck parameters for the outer loop that discovers a failure, and to supply a strategy for use only by the shrinkage. Providing a default would be handy - could simply pick up Test.Parameters.default.minSuccessfulTests?

Another idea is to devise a strategy that moonlights as a test callback in Test.Parameters.testCallback - this could pick up the number of property evaluations up the first failing case and use that as a limit when shrinking.

[sageserpent-open]

This ticket is up for grabs...