Internal assertion failure when trying to emulate a Jqwik test.

[sageserpent-open]

Reading this blog post about Jqwik: Property-based Testing in Java: The Importance of Being Shrunk prompted me to write this test:

private static final Trials<String> first =
        stringsOfSize(1, 10, charactersInRange('a', 'z'));

private static final Trials<String> second =
        stringsOfSize(0, 10, charactersInRange('0', '9'));

@Test
void thisBreaksTrialsWithAnInternalAssertionFailure() {
    first.and(second)
         .withLimit(50)
         .supplyTo((String first, String second) -> {
             final String concatenation = first + second;
             assertThat("Strings aren't allowed to be of length 4" +
                        " or 5 characters" + " in this test.",
                        4 > concatenation.length() ||
                        5 < concatenation.length());
         });
}

private static Trials<String> stringsOfSize(int minimumSize,
                                            int maximumSize,
                                            Trials<Character> characters) {
    return api
            .choose(Range.range(minimumSize, maximumSize))
            .flatMap(size -> characters.collectionsOfSize(size,
                                                          () -> new Builder<Character, String>() {
                                                              final StringBuffer
                                                                      buffer =
                                                                      new StringBuffer();

                                                              @Override
                                                              public void add(
                                                                      Character caze) {
                                                                  buffer.append(
                                                                          caze);
                                                              }

                                                              @Override
                                                              public String build() {
                                                                  return buffer.toString();
                                                              }
                                                          }));
}

private static Trials<Character> charactersInRange(char from, char to) {
    return api.integers(from, to).map(index -> (char) (int) index);
}

Running that test results in an internal assertion failure:

at scala.Predef$.assert(Predef.scala:264)
at com.sageserpent.americium.java.TrialsImplementation$$anon$18.$anonfun$supplyTo$2(TrialsImplementation.scala:964)

...

at com.sageserpent.americium.java.tupleTrials$Tuple2Trials$$anon$3.supplyTo(tupleTrials.scala:73)
at com.sageserpent.americium.java.TrialsApiTests.thisBreaksTrialsWithAnInternalAssertionFailure(TrialsApiTests.java:318)

Not awe-inspiring.

[sageserpent-open]

The bug reproduction can be reduced to:

@Test
void minimiseBug() {
    stringsOfSize(1, 10, api.integers().map(index -> (char) (int) index))
            .withLimit(50)
            .supplyTo(only -> assertThat(
                    "Strings aren't allowed to be of length 4" +
                    " or 5 characters" + " in this test.",
                    4 > only.length() ||
                    5 < only.length()));
}

Initial investigation shows that the changes made in issue #29 have introduced this bug - specifically, the complexity wall is partially waived in TrialsImplementation.lotsOfSize that underpins the generation of trials of fixed sized collections / strings etc. This means that the number of decision stages in a case resulting from the trials can exceed the complexity wall, and that can break the shrinkage invariant that any further shrunk case must have no more decision stages than the caller's case.

[sageserpent-open]

Fixed in commit dfa513577b78bb76640a563724cdd6e2d2d20ed1, but the now succeeding test case has exposed another unrelated problem - shrinkage takes forever when we have a streaming trials whose cases' behaviour in a test aren't really influenced by the factory input - and Trials.strings is a perfect example.

What happens here is that the shrinkage of factory input values has to continue to the bitter end, as it will have no effect whatsoever on whether the test passes or fails - so the shrinkage has to pussyfoot through a long series of pointless shrinkage steps, when it has already minimised the complexity.

EDIT - see the following comment. It turns out that the aforementioned case would probably be OK - the shrinkage would slog its way through reasonably fast by virtue of the final stage shrinkage becoming increasingly impatient.

What we have here is a different situation layered on top, read on ...

[sageserpent-open]

Further investigation reveals that the trials used in the slow test example itself is pathological - it uses a shrinkable-valued size to control how large its final cases are - once these are shrunk beyond a threshold, it is no longer possible to generate any cases that would meet the fixed complexity required when doing the final stages of shrinkage.

[sageserpent-open]

Fixed in release 1.0.1.

Waiting on successful promotion from Sonatype before closing ticket.

[sageserpent-open]

As a passing thought, there is now a commented-out pathologically slow test case in the shrinkage test in TrialsSpec. One to address in the future. The Java test inspired by Jqwik does complete very rapidly, but there is a fly in the ointment where if the string sizes are allowed to be shrinkable (now done as of commit 988cf584e2292974c3031d622a945f60c27a78a1), then the shrinkage is spoilt somewhat, which is another side of the same problem discussed above.

[sageserpent-open]

It's been published now, so closing this ticket.