Tests that fail due to "exhausted arguments" are marked as "passed"

[sol]

Here is an example:

import Test.Framework
import Test.Framework.Providers.QuickCheck2 (testProperty)
import Test.QuickCheck

main :: IO ()
main = defaultMain [ testProperty "reverse" prop_reverse]

prop_reverse :: [Int] -> Property
prop_reverse xs = False ==> xs == (reverse . reverse) xs

quickCheck prop_reverse fails on this with

*** Gave up! Passed only 0 tests.

But test-framework marks it as passed.

[batterseapower]

Fixed this in test-framework-quickcheck-0.2.8 and test-framework-quickcheck2-0.2.11

[sol]

@batterseapower Hmm, that was not obvious from my example. But my point was that any property that fails due to "exhausted arguments" should be marked as failed (yes, even if some tests passed). At least that is what QuickCheck does.

I'd even argue that "exhausted arguments" is a misnomer, and that the terminology used by QuickCheck ("Gave up!") is much clearer. I think smallcheck-like approaches may "exaust arguments", but random testing (at least as implemented by QuickCheck) can not.

[batterseapower]

Sorry, I'm not sure what you are replying to. I agree with you that "exhausted arguments" should cause the test to fail. Is that not the behaviour I implemented when I changed the code 3 months ago?

[sol]

@batterseapower If I understand correctly, it's conditional.

PropertyArgumentsExhausted -> maybe False (/= 0) mb_n

(887368265d15d4ea644ef0414c1adb14b70fdfaa)

[batterseapower]

OK, now I understand! I'll make the change.

I've published it as test-framework-quickcheck-0.2.8.1 and test-framework-quickcheck2-0.2.12.2