Add API to automatically generate a regression test based on the property failure

[vlsi]

Testing Problem

Jqwik is fine to catch failures, however, it does not seem to cover the aspect of adding regression tests. Well, there's @Property(seed="4242", however, it does not cover the requirements for regression tests.

1. seed-based test is obscure. It is hard to understand the nature of the test. The test code does not show input data
2. seed-based test it is hard to debug and modify as one can't modify the part of the code that generates input values
3. seed-based tests do not survive the change in the Arbitrary generators. For instance, if I change weights (or add an edge case), then behaviour of all the tests change
4. seed does not survive changes in the random generator. Even though random generators do not change often, it is sad to lose all the seed-based regression tests when the generator improves

Suggested Solution

Implement an API so the user can register "unparser" or "test printer" that would automatically generate a valid test code.

The same "test printer" might be helpful even for regular "property failed" messages. In other words, suppose a test fails in CI. Then it would be nice if the failure message would contain a source code for the reproducer.

It would be nice if jqwik could integrate printers (Java, Kotlin) for common classes like Integer, Long, List, Map, Set, and so on.

See also: https://github.com/jlink/jqwik/issues/428#issuecomment-1373552260

[jlink]

I think I have an inkling of what you mean, but could you add an example or two to make sure we're thinking of the same thing?

[vlsi]

Apache Calcite has expression simplifier.

Here's a typical hand-crafted test:

https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/test/java/org/apache/calcite/rex/RexProgramTest.java#L1826-L1836

  @Test void testSimplifyEqOrIsNullAndEqSame() {
    // (deptno = 10 OR deptno IS NULL) AND deptno = 10
    //   ==>
    // false
    final RexNode e =
        and(
            or(eq(vInt(), literal(10)),
                isNull(vInt())),
        eq(vInt(), literal(10)));
    checkSimplify(e, "=(?0.int0, 10)");
  }

A few years ago I added a trivial random-based expression generator and a property test so it generates expressions and checks if the simplification results is sane. Here's one of the checks: https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/test/java/org/apache/calcite/test/fuzzer/RexProgramFuzzyTest.java#L221-L227

The generated messages include copy-paste ready code that generates and expression which triggered the failure. See nodeToString which boils down to https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/test/java/org/apache/calcite/test/fuzzer/RexToTestCodeShuttle.java

For instance, here's a test case that was found by the fuzzer: https://github.com/apache/calcite/blob/b9c2099ea92a575084b55a206efc5dd341c0df62/core/src/test/java/org/apache/calcite/rex/RexProgramTest.java#L513-L520

  @Test void reproducerFor3457() {
    // Identified with RexProgramFuzzyTest#testFuzzy, seed=4887662474363391810L
    checkSimplify(
        eq(unaryMinus(abstractCast(literal(1), tInt(true))),
          unaryMinus(abstractCast(literal(1), tInt(true)))),
        "true");
  }

The part eq(unaryMinus(abstractCast(literal(1), tInt(true))), unaryMinus(abstractCast(literal(1), tInt(true)))) was included in to the exception message, so I copied it from the failure message and added @Test, checkSimplify, etc.

Of course, for jqwik-like integration, it might be fun to produce @Example functions that generate the code via literals and call the original test method.

For instance:

@Example void fuzzyCase12374253dsfdfj() {
  Expression expr = eq(unaryMinus(abstractCast(literal(1), tInt(true))), unaryMinus(abstractCast(literal(1), tInt(true))));

  callOriginalPropertyMethod("propertyName", expr, ... /*other args*/);
}

Then, the users would be able to keep the test as @Example, or they could re-parameterize it:

@Property void fuzzyCase12374253dsfdfj(@ForAll boolean bool) {
  Expression expr = eq(unaryMinus(abstractCast(literal(1), tInt(bool))), unaryMinus(abstractCast(literal(1), tInt(bool))));

  callOriginalPropertyMethod("propertyName", expr, ... /*other args*/);
}

[jlink]

I think, this idea requires a link from a shrinkable back to its arbitrary or at least to what you call the unparser. Having this backlink to the arbitrary would have other benefits as well, but would also increase the memory footage of each and every test-run.

I guess some experiments are warranted to find out how much memory we're actually talking about.

[vlsi]

Probably a slightly better term would be serializer. In other words, it serializes Java objects to Java source or Kotlin source formats.