Closed sageserpent-open closed 1 year ago
Some initial work on modifying OptionalLimits
/ withStrategy
/ withLimits
reveals a very messy API that would force client test code to be recompiled. The profusion of overloads in both the Scala and Java forms of TrialsApi
are a concern too.
Trying a different approach, which is to move additional tweaks into SupplyToSyntax
and accept that some of these tweaks might be irrelevant on conjunction with withRecipe
...
... which leads to the possibility of deprecating withRecipe
and using yet another JVM property, trials.recipe
. Let's see how things work out and come back to this...
Reworked the Java and Scala APIs so that the seed - and the existing configuration tweaks that lived in overloads of .withLimits
, .withStrategy
and OptionalLimits
- are now in the SupplySyntax
fluent decoration interface / trait. The end ways of configuration are still there, but are deprecated now.
The price to pay for the rework of the API is that .withRecipe
can be followed by a host of configuration tweaks that will have no effect whatsoever - doesn't cause any harm, but is misleading. Now if a new JVM property trials.recipe
were to be added, then we could axe .withRecipe
altogether and close that loophole. Given that hacking test code to make the call to withRecipe
to reproduce a bug is clunky anyway, this is food for thought...
Also did a host of cleanup refactoring of the TrialsImplementation
, unbundling a whole heap of implementation detail. While there is some slight improvement in terms of pulling out GenerationOperation
and Decision
, the extraction of the mammoth SupplyToSyntaxSkeletalImplementation
is open to debate - but at least IntelliJ is less likely to mangle the sources when it auto-reformats or refactors now - that was a constant problem with it dropping imports and sometimes whole swathes of code, as TrialsImplementation
was simply too large and complex for it to cope.
Delivered in commit 5b2c55e755b52ee8fea4f73b074de6d86897e2e2 .
Americium diligently makes its test cases reproducible - so provided the prescription of a given
Trials
hasn't changes, then the test cases in a cycle produced by a deterministic strategy should be the same, also taking into consideration whether and how a test is failing.CI builds might want to vary the seed used to prime the randomisation, so that over time, a wider spectrum of test cases can be covered via multiple low-coverage cycles, as opposed to doing carpet-bombing of the SUT in every cycle.
Some folks may wish to control this explicitly in test code, others may wish to tweak this via a JVM property, so let's support both:
OptionalLimits
for the Java API and towithStrategy
/withLimits
so that the default provides the same repeatable behaviour that is currently burnt intoTrialsImplementation
, but allows client test code to exercise it own choice of deterministic test case generation, possibly using a non-deterministic choice such as a random number generator.trials.nondeterministic
which when set to true chooses some random seed that overrides either the choice given explicitly via client test code or implicitly via the default.