Configure test assiduousness with a strategy.

[sageserpent-open]

By default, the random behaviour that drives the generation of cases in TrialsImplementation is based on a fixed seed.

This is to support repeatable test runs, which seems to be a desirable outcome at first sight. However, the mechanism for reproduction of a failing test case doesn't require the history of the test cases that led up to failing test case (with the exception of the JUnit5 integration via @TrialsTest), so it would be feasible for the most part to allow some scope for variation as to what cases are generated from one run to the next.

This would open up the possibility of a CI build driving the system under test either a lot harder than a regular manually-executed run, or would allow multiple CI builds to explore more cases, rather than simple repeating the same old sequence.

[sageserpent-open]

The question arises - do these tweaks apply across the board to all tests? Surely we want to focus on just specific problematic SUTs, especially if a time budget approach is adopted.

Possibilities:

• Add another configuration parameter to .withLimits that allows one or more tweaks to be picked up by the test case generation.
• Add another configuration parameter to @TrialsTest that allows one or more tweaks to be picked up by the test case generation.

So individual tests can opt-in to being tweaked according to some global policy configured by a JVM property.

We might have trials.tweak.timeLimit that when set in isolation, effectively overrides the casesLimit configuration parameter in the code and causes cases to be generated until the given time limit has expired (or a failing case has been found, in which case shrinkage takes place with the same time limit being applied at each shrinkage attempt).

Another setting would be trials.tweak.expansion, which modifies the effect of the casesLimit configuration parameter in the code by multiplying the limit by a positive value. This would apply both for the generation of the initial successful cases up to any failure and to the subsequent shrinkage.

Specifying both of the above tweaks together would be permitted and would simply cause the first encountered limit to apply.

One could also apply these tweaks to just the initial phase of finding a failing test case, or to the subsequent shrinkage phases:

• trials.tweak.initial.timeLimit, trials.tweak.initial.expansion,
• trials.tweak.shrink.timeLimit, trials.tweak.shrink.expansion.

Naturally, these can be used to override the more general settings where appropriate.

[sageserpent-open]

Let us not forget trials.tweak.nonDeterministic, which would allow the seed for pseudo-random behaviour to be picked up from some random environmental factor, such as time, a UUID or whatever.

[sageserpent-open]

... and now for something completely different. Instead of faffing around with lots of confusing JVM properties, how about simply marking some tests as tweakable and letting the tests find out for themselves what level of assiduousness / choice of seed they should use? We have already have a RocksDB store associated with the implementation, perhaps the tests should vary their own configuration, make note in the store of the outcome and focus accordingly when failures are revealed?

[sageserpent-open]

Changed the tile of this ticket to reflect a new idea - instead of loading in various bells and whistles to keep all potential customers happy, why not generalise the existing notion of a cases limit to being a strategy that can be provided in client code according to taste. This means adding in yet more overloads of withLimit(s), and cutover of the existing ones to use a built-in strategy that replicates existing behaviour.

This strategy will be consulted whenever a new case is about to be generated, and also informed of inline case filtration rejections.

[sageserpent-open]

Trials now sports overloads of withStrategy that take a factory/supplier of CasesLimitStrategy to provide custom control of case emission. The strategy instances are created anew for each call to supplyTo during the initial failure discovery cycle and are recreated for each shrinkage cycle.

There is currently no way of distinguishing between the initial failure discovery cycle or any subsequent shrinkage cycles.

Nor does @TrialsTest know about strategies.

[sageserpent-open]

This has gone out in version 1.6.0, Git commit: f297ce801983c70ab1166893f3ea06282cc6ba87 .

[sageserpent-open]

Evidence made using JShell running against Americium version 1.6.0 follows...

Code:

import com.sageserpent.americium.java.CasesLimitStrategy;
import com.sageserpent.americium.java.Trials;
import com.sageserpent.americium.java.TrialsApi;

import java.util.function.Supplier;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

import static com.sageserpent.americium.java.TrialsScaffolding.OptionalLimits.defaults;

final TrialsApi api = Trials.api();

final Supplier<CasesLimitStrategy> supplier =
            () -> new CasesLimitStrategy() {
                boolean starvationHasTakenPlace = false;

                @Override
                public boolean moreToDo() {
                    // Give up immediately as soon as starvation occurs.
                    return !starvationHasTakenPlace;
                }

                @Override
                public void noteRejectionOfCase() {
                    System.out.format("Rejection\n");
                }

                @Override
                public void noteEmissionOfCase() {
                    System.out.format("Emission\n");
                }

                @Override
                public void noteStarvation() {
                    System.out.format("Starvation\n");

                    starvationHasTakenPlace = true;
                }
            };

final Trials<Integer> cases = api.choose(IntStream.range(0, 10).boxed().collect(Collectors.toList()));

System.out.println("---------- All emitted cases are valid -----------");

cases.withStrategy(supplier, defaults).supplyTo(System.out::println);

System.out.println("---------- Using a filter in the DSL -----------");

cases.filter(value -> 5 < value).withStrategy(supplier, defaults).supplyTo(System.out::println);

System.out.println("---------- Some emitted cases are rejected in the test -----------");

cases.withStrategy(supplier, defaults).supplyTo(caze -> {
    Trials.whenever(1 == caze % 2, () -> System.out.println(caze));
});

Output:

import com.sageserpent.americium.java.CasesLimitStrategy
import com.sageserpent.americium.java.Trials
import com.sageserpent.americium.java.TrialsApi
import java.util.function.Supplier
import java.util.stream.Collectors
import java.util.stream.IntStream
static import com.sageserpent.americium.java.TrialsScaffolding.OptionalLimits.defaults

field TrialsApi api = com.sageserpent.americium.TrialsApis$$anon$1@47089e5f

field Supplier<CasesLimitStrategy> supplier = $Lambda$256/0x0000000800d75840@37b70343

field Trials<Integer> cases = TrialsImplementation(Free(...))

System.out.println("---------- All emitted cases are valid -----------")
---------- All emitted cases are valid -----------

cases.withStrategy(supplier, defaults).supplyTo(System.out::println)
Emission
8
Emission
9
Emission
0
Emission
4
Emission
1
Emission
7
Emission
6
Emission
2
Emission
3
Emission
5
Starvation

System.out.println("---------- Using a filter in the DSL -----------")
---------- Using a filter in the DSL -----------

cases.filter(value -> 5 < value).withStrategy(supplier, defaults).supplyTo(System.out::println)
Emission
8
Emission
9
Starvation

System.out.println("---------- Some emitted cases are rejected in the test -----------")
---------- Some emitted cases are rejected in the test -----------

cases.withStrategy(supplier, defaults).supplyTo(caze -> {
    Trials.whenever(1 == caze % 2, () -> System.out.println(caze));
})
Emission
Rejection
Emission
9
Emission
Rejection
Emission
Rejection
Emission
1
Emission
7
Emission
Rejection
Emission
Rejection
Emission
3
Emission
5
Starvation