NorfairKing / sydtest

A modern testing framework for Haskell with good defaults and advanced testing features.
113 stars 25 forks source link

Show single quickcheck labels #33

Closed rynoV closed 2 years ago

rynoV commented 2 years ago

Hi, thanks for making this tool!

I noticed when a quickcheck property only has a single label, the label is not shown in the sydtest output. It would be useful to me if the label is still shown because I have some tests where there are multiple possible labels, and if only one of them applies to a given test I would like to know which one.

Example

prop_reverse :: Property
prop_reverse =
  forAll (arbitrary) $
    \ls ->
      collect (if null ls then ("empty" :: String) else "not empty") $
        reverse (reverse (ls :: [Int])) `shouldBe` ls

spec :: Spec
spec = do
  describe "reverse" $
    specify
      "reversing twice is the same as not reversing"
      prop_reverse

will output:

reverse
    ✓ reversing twice is the same as not reversing                         0.85 ms
      passed for all of 100 inputs.
      labels
         8.00% "\"empty\""
        92.00% "\"not empty\""

But if we change the property to only output empty lists:

prop_reverse :: Property
prop_reverse =
  forAll (arbitrary `suchThat` null) $
    \ls ->
      collect (if null ls then ("empty" :: String) else "not empty") $
        reverse (reverse (ls :: [Int])) `shouldBe` ls

The "empty" label won't be shown:

reverse
    ✓ reversing twice is the same as not reversing                         1.25 ms
      passed for all of 100 inputs.

I believe this is the relevant line: https://github.com/NorfairKing/sydtest/blob/127362d7e48d9b983f61e358966af71003f95026/sydtest/src/Test/Syd/Output.hs#L217 I think changing it to just check if the labels are empty might be what I need (I tried to setup the development environment as described in CONTRIBUTING.md, but ran into errors building persistent-postgresql, persistent-sqlite, wai-app-static, shakespeare, path-io, so I haven't been able to test this).

I would like something similar to what quickcheck itself does:

λ> quickCheck prop_reverse
+++ OK, passed 100 tests (100% "empty").
NorfairKing commented 2 years ago

Hi! Gerne

For which project are you using sydtest?

I'd be happy with a PR that changes the <= to a < or the 1 to a 0 and to have CI take care of the rest..

rynoV commented 2 years ago

I'm working on a library for generating test cases for parsers based on simple grammars (using quickcheck), and depending on the grammar sometimes only invalid test cases can be generated, or only valid test cases, so when I label the generated cases as invalid or valid I want it to be clear when this happens.

Opened the PR!