Test suite fails when seed does not produce a counterexample

[roberth]

In this build (look for "previous attempt"), the test suite fails and logs:

        Use --quickcheck-replay=116302 to reproduce. (expected failure)
       Interchange: (x *> (y <*? z)) == ((x *> y) <*? z):                                                                    OK (unexpected)
         +++ OK, passed 100 tests.
          (unexpected success)
         Use -p '/Validation.Theorems.Interchange: (x *> (y <*? z)) == ((x *> y) <*? z)/' to rerun this test only.

This is theorem6 in validationTheorems :: Tests, which is an expectFailure test. This means that it is using a property test to find a counterexample for the interchange theorem. If it fails to find one, it fails the test suite.

Failing the test suite doesn't seem like the right thing to do here, because the absence of such a property can hardly be a requirement for the library to function correctly.

[snowleopard]

Thanks, I agree. This also annoyed me a few times. I guess we have two options here:

• Make an unexpected success a warning, i.e. do not fail the test suite when it happens.

• Instead of using the expected failures mechanism, record counterexamples in the test suite (the test suite would simply ensure that counter-examples indeed violate the corresponding properties).

The first option is easier to implement. The second one seems like a better solution.

[snowleopard]

I'll try to find time to implement one of the two options soon. Happy to review a PR if anyone beats me to it!