nick8325 / quickcheck

Automatic testing of Haskell programs.
Other
713 stars 119 forks source link

hspec's `around` doesn't wrap shrunk quickcheck tests #413

Open ChickenProp opened 1 month ago

ChickenProp commented 1 month ago

At first I thought this was a bug in hspec, I opened an issue at https://github.com/hspec/hspec/issues/899. But as things currently stand I'm not sure how hspec could fix it without changes to quickcheck. So I'm posting here too to try to get more eyes.

Short version is that hspec has an around function that lets you wrap every run of a quickcheck test. But if the test fails and shrinks, it doesn't wrap the shrunk attempts. For example:

  let hook act = do
        putStrLn "before"
        act
        putStrLn "after"
  around_ hook $ it "xxx" $ do
    let shr = \case
          2 -> [1, 0]
          _ -> []
    forAllShrink getSize shr $ \someInt -> ioProperty $ do
      putStrLn $ "Testing " <> show someInt
      someInt `shouldNotBe` 2
before     
Testing 0
after
before
Testing 1
after
before
Testing 2
after
Testing 1
Testing 0

We have two successful tests that get wrapped, a failing test that gets wrapped, then two shrinks of the failing test that don't get wrapped.

The way this works is that hspec examples can be of type a -> Property, and hooks can provide the a. a -> Property is essentially a -> QCGen -> Int -> Rose Result, where the root Result is the result of the first test and children are the results of any shrinks. (Even if shrinking is unnecessary, every shrink is in the tree, though not actually evaluated.) But that type can't do what we want, since we have a single a to construct the whole tree. Currently it essentially calls hook $ \a -> reduceRose (r a), which means the function passed to the hook

Which gives us the behavior observed, that the hook wraps the initial test but not any shrinks.

I don't see any way to get around this with the types as they are. Switching to QCGen -> Int -> Rose (a -> IO Result) might work, but it feels like that would need large changes to both quickcheck and hspec, and I can't rule out that it would fail for some other reason.

Any suggestions?