sageserpent-open / americium

Generation of test case data for Scala and Java, in the spirit of QuickCheck. When your test fails, it gives you a minimised failing test case and a way of reproducing the failure immediately.
MIT License
15 stars 1 forks source link

Expected number of cases are not being supplied. #41

Closed sageserpent-open closed 2 years ago

sageserpent-open commented 2 years ago

Reproduction via JShell:

import com.sageserpent.americium.java.Trials;
import com.sageserpent.americium.java.TrialsApi;

import java.util.concurrent.atomic.AtomicInteger;

    final TrialsApi api = Trials.api();

    final Trials<Integer> integers = api.integers();

    final AtomicInteger counter = new AtomicInteger(0);

    integers.withLimit(100000).supplyTo(caze -> {
        counter.incrementAndGet();
    });

    System.out.println(counter.get());

The expected count of invocations should be 100000 (or possibly a value close to that, as there might be the odd collision between cases, given that they are randomly generated).

What is printed as per Git commit: f8c9f6a1ed04e45620e2f4464dcda852c5767907 (release 1.4.0) is 14102.

What is stopping at least roughly the required number of cases from being generated?

The same lack of supplied cases is also exhibited if api.doubles() is used to supply the cases - again, only 14102 cases are supplied.

Writing a custom case factory like this:


api.stream(new CaseFactory<Long>() {
        @Override
        public Long apply(long input) {
            return 22L;
        }

        @Override
        public long lowerBoundInput() {
            return Long.MIN_VALUE;
        }

        @Override
        public long upperBoundInput() {
            return Long.MAX_VALUE;
        }

        @Override
        public long maximallyShrunkInput() {
            return 0L;
        }
    });

Still results in 14102 cases being supplied.

Maybe whatever generates the inputs that drive the case factories is cycling through a limited set of long integer values?

sageserpent-open commented 2 years ago

This was due to an integer overflow in the calculation of the updated starvation countdown value when streaming cases in TrialsImplementation - had written a naive Math.sqrt(limit * remainingGap), being blissfully ignorant of the possibility of integer overflow prior to the promotion to a double value in the call to Mathsqrt. Rookie mistake.

What is surprising was that no exception was thrown - the NaN yielded by Math.sqrt is subsequently quietly converted to a zero - this was what caused the abrupt cessation of new cases.

Have added a test to reproduce this problem, this failed, have put in a bug fix and it now passes.

sageserpent-open commented 2 years ago

This has gone out in release 1.4.1, Git commit SHA c6573a9 .

sageserpent-open commented 2 years ago

Retried the JShell reproduction above, now prints the correct value of 100000 when using release 1.4.1 of Americium.