sageserpent-open / americium

Generation of test case data for Scala and Java, in the spirit of QuickCheck. When your test fails, it gives you a minimised failing test case and a way of reproducing the failure immediately.
MIT License
15 stars 1 forks source link

Incorrect case cycle counts. #57

Closed sageserpent-open closed 1 year ago

sageserpent-open commented 1 year ago

Running this code in JShell:

import com.sageserpent.americium.java.Builder;
import com.sageserpent.americium.java.CasesLimitStrategy;
import java.time.Duration;
import static com.sageserpent.americium.java.Trials.api;
import static org.hamcrest.CoreMatchers.*;
import static org.hamcrest.MatcherAssert.assertThat;

    final String suffix = "are";

    final int suffixLength = suffix.length();

    api()
            .characters('a', 'z')
            .collections(Builder::stringBuilder)
            .filter(caze -> caze.length() >
                            suffixLength)
            .withStrategy(cycle -> {
                System.out.format("Number of previous cycles:  %d\n",
                                  cycle.numberOfPreviousCycles());

                return CasesLimitStrategy.timed(
                        Duration.ofSeconds(5));
            })
            .supplyTo(input -> {
                assertThat(input, not(containsString(suffix)));
            });

Yields an incorrect number of cases cycles for the final invocation of the callback supplied to .withStrategy:

Number of previous cycles:  0
Number of previous cycles:  1
Number of previous cycles:  2
Number of previous cycles:  3
Number of previous cycles:  3

Note the count is repeated. Possibly this is something to do with the shrinkage switching between ordinary shrinkage and panic mode shrinkage? A cursory inspection of shrink buried deep within SupplyToSyntaxSkeletalImplementation suggests this is the case.

sageserpent-open commented 1 year ago

Hmm - I'm not sure whether .numberOfPreviousCycles is simply a misnomer. It was named .numberOfPreviousShrinkages, which also misleading - perhaps what it should signify is the depth of shrinkage, so both an ordinary and a following panic mode shrinkage should get the same value. If so, then presumably we should reflect the difference in modes as an additional property in CaseSupplyCycle?

sageserpent-open commented 1 year ago

The name is a misnomer - what .numberOfPreviousCycles really yields is the number of nested shrinkage attempts, each of which resulted in a failing trial. So the older name was a little more accurate, but certainly not intuitive. What happens in each shrinkage attempt is that a cycle is run for normal shrinkage, and if that doesn't encounter a failing trial, an additional cycle for panic mode shrinkage is run.

So a better name would be .numberOfPreviousFailures - this captures what is going on, but isn't too tied in with the shrinkage implementation details.

For the sake of not breaking API compatibility, maybe the existing .numberOfPreviousCycles should be left in, but cutover to actually count the number of cycles?

sageserpent-open commented 1 year ago

Fixed in commit SHA: 0a64297fa0930275fcfaedac844759e4287afaef , went out in release 1.10.1