sageserpent-open / americium

Generation of test case data for Scala and Java, in the spirit of QuickCheck. When your test fails, it gives you a minimised failing test case and a way of reproducing the failure immediately.
MIT License
15 stars 1 forks source link

Allow client code to specify seed to influence supply of test cases. #51

Closed sageserpent-open closed 1 year ago

sageserpent-open commented 1 year ago

Americium diligently makes its test cases reproducible - so provided the prescription of a given Trials hasn't changes, then the test cases in a cycle produced by a deterministic strategy should be the same, also taking into consideration whether and how a test is failing.

CI builds might want to vary the seed used to prime the randomisation, so that over time, a wider spectrum of test cases can be covered via multiple low-coverage cycles, as opposed to doing carpet-bombing of the SUT in every cycle.

Some folks may wish to control this explicitly in test code, others may wish to tweak this via a JVM property, so let's support both:

sageserpent-open commented 1 year ago

Some initial work on modifying OptionalLimits / withStrategy / withLimits reveals a very messy API that would force client test code to be recompiled. The profusion of overloads in both the Scala and Java forms of TrialsApi are a concern too.

Trying a different approach, which is to move additional tweaks into SupplyToSyntax and accept that some of these tweaks might be irrelevant on conjunction with withRecipe...

sageserpent-open commented 1 year ago

... which leads to the possibility of deprecating withRecipe and using yet another JVM property, trials.recipe. Let's see how things work out and come back to this...

sageserpent-open commented 1 year ago

Reworked the Java and Scala APIs so that the seed - and the existing configuration tweaks that lived in overloads of .withLimits, .withStrategy and OptionalLimits - are now in the SupplySyntax fluent decoration interface / trait. The end ways of configuration are still there, but are deprecated now.

sageserpent-open commented 1 year ago

The price to pay for the rework of the API is that .withRecipe can be followed by a host of configuration tweaks that will have no effect whatsoever - doesn't cause any harm, but is misleading. Now if a new JVM property trials.recipe were to be added, then we could axe .withRecipe altogether and close that loophole. Given that hacking test code to make the call to withRecipe to reproduce a bug is clunky anyway, this is food for thought...

sageserpent-open commented 1 year ago

Also did a host of cleanup refactoring of the TrialsImplementation, unbundling a whole heap of implementation detail. While there is some slight improvement in terms of pulling out GenerationOperation and Decision, the extraction of the mammoth SupplyToSyntaxSkeletalImplementation is open to debate - but at least IntelliJ is less likely to mangle the sources when it auto-reformats or refactors now - that was a constant problem with it dropping imports and sometimes whole swathes of code, as TrialsImplementation was simply too large and complex for it to cope.

sageserpent-open commented 1 year ago

Delivered in commit 5b2c55e755b52ee8fea4f73b074de6d86897e2e2 .