sageserpent-open / americium

Generation of test case data for Scala and Java, in the spirit of QuickCheck. When your test fails, it gives you a minimised failing test case and a way of reproducing the failure immediately.
MIT License
15 stars 1 forks source link

Extend `CaseSupplyCycle` to yield statistics from the previous cycle. #50

Closed sageserpent-open closed 1 year ago

sageserpent-open commented 1 year ago

It could be useful - especially for a Scalacheck integration - to furnish statistics from the previous cycle (or maybe across all of previous cycles) to allow customisation of the CaseLimitStrategy for the next shrinkage cycle. This would allow adaptive strategies to be formulated, which in the case of Scalacheck would fit the notion of an initial failure discovery cycle based on Scalacheck's configuration, followed by dedicated shrinkage cycles that are tailored by previous cycles and not by the Scalacheck configuration.

It is worth doing a spike with a quick-and-dirty stateful factory for CasesLimitStrategy - not strictly what we're supposed to pass to the likes of Trials.withStrategy, but it would allow some experiments to be performed to see if it's worthwhile....

sageserpent-open commented 1 year ago

The spike was very disappointing - replaced the use of timed strategies in the Americium section of the repository https://github.com/jlink/shrinking-challenge with a custom strategy factory:

    static Function<CaseSupplyCycle, CasesLimitStrategy> adaptiveShrinkingAfter(
            CasesLimitStrategy discoveryCasesLimitStrategy) {
        // NASTY HACK: park some local state here that records the results
        // from successive cycles. This is a hack because it assumes client
        // code doesn't hold on to the lambda produced by this call and
        // re-use it across multiple calls to `Trails.withStrategy`. It's a
        // spike, after all...

        final AtomicInteger emissionCountInCycle = new AtomicInteger();
        final AtomicInteger starvationCountInCycle = new AtomicInteger();

        return caseSupplyCycle -> {
            final CasesLimitStrategy cycleSpecificCasesLimitFactory;

            if (caseSupplyCycle.isInitial())
                cycleSpecificCasesLimitFactory = discoveryCasesLimitStrategy;
            else {
                final int emissionCountInPreviousCycle =
                        emissionCountInCycle.get();
                final int starvationCountInPreviousCycle =
                        starvationCountInCycle.get();
                cycleSpecificCasesLimitFactory = counted(
                        emissionCountInPreviousCycle,
                        (double) starvationCountInPreviousCycle /
                        emissionCountInPreviousCycle);
            }

            emissionCountInCycle.set(0);
            starvationCountInCycle.set(0);

            return new CasesLimitStrategy() {
                @Override
                public boolean moreToDo() {
                    return cycleSpecificCasesLimitFactory.moreToDo();
                }

                @Override
                public void noteRejectionOfCase() {
                    emissionCountInCycle.decrementAndGet();
                    starvationCountInCycle.incrementAndGet();

                    cycleSpecificCasesLimitFactory.noteRejectionOfCase();
                }

                @Override
                public void noteEmissionOfCase() {
                    emissionCountInCycle.incrementAndGet();

                    cycleSpecificCasesLimitFactory.noteEmissionOfCase();
                }

                @Override
                public void noteStarvation() {
                    starvationCountInCycle.incrementAndGet();

                    cycleSpecificCasesLimitFactory.noteStarvation();
                }
            };
        };
    }

This failed miserably in minimising the failing test cases for the Bound5Spec, CouplingTest and CalculatorSpec examples, which prior to the change would reliably yield optimal shrinkages - the approach taken was to feed the original timed strategy to adaptiveShrinkingAfter.

Marking this as a won't-fix...

sageserpent-open commented 1 year ago

Closing as this isn't a good idea, based on the spike outcome.