search

hspec / hspec-hedgehog

A library to integrate hedgehog tests into your hspec test suite.
https://hackage.haskell.org/package/hspec-hedgehog
Other
28 stars 6 forks source link

'GaveUp' when moving from tasty-hedgehog #9

Open vrom911 opened 4 years ago

vrom911 commented 4 years ago

First of all, thanks a lot for this handy library! I am using it a lot already.

However, I am facing an issue when trying to move from tasty-* family to hspec-* family. Specifically, I want to move from tasty-hedgehog to hspec-hedgehog and the process itself is straightforward. I am not changing any hedgehog specific functions or any of the Gens. But I get GaveUp in my property tests now. I manage to fix a few such tests with downgrading some ranges in generators from 0 10 to 0 5, which seems odd.

Do you know about any limitations hspec-hedgehog puts on discarding or any other limits? Would really appreciate your insights on why this regression can happen :pray:

(This is a commit with the first attempts of migration to hspec-hedgehog: https://github.com/kowainik/tomland/commit/8755e76d21ca25ddeeb2e8ddb184db8a773326b9 )

parsonsmatt commented 4 years ago

GaveUp occurs when the propertyDiscardLimit is hit while shrinking, as far as I can tell (see the source of hedgehog:Hedgehog.Internal.Runner). You can configure this with the maxDiscardRatio function in Test.Hspec.QuickCheck.

vrom911 commented 4 years ago

Shrinking is not the only reason. It is mentioned in this comment on the hedgehog library issues: https://github.com/hedgehogqa/haskell-hedgehog/issues/186#issuecomment-391822861

But the most mystical part is that it was working before, and stopped working now, even though I didn't configure anything differently anywhere. The question is more about if there are some limits in hspec-hedgehog itself to make this happen?

parsonsmatt commented 4 years ago

The problem is that QuickCheck has a default discard ratio of 10, while hedgehog has a default discard limit of 100. I think we could make this more seamless by adding 90 to the QuickCheck configuration -> Hedgehog configuration.

vrom911 commented 4 years ago

Thank you, this totally solves the problem.

Would you like to see the PR with the changed config where maxDiscardRatio is set to 100? Or maybe there could be a better solution for that? :thinking:

parsonsmatt commented 4 years ago

Well, setting it directly to 100 is bad, because people will use things like maxDiscardRatio (* 123), and having the discard ratio be 100 would be surprising. That's why I think (+ 90) is appropriate - if you expect 100 and get 10, that's bad, but if you expect 1000 and get 1090, that's not too bad. But if you expect eg < 10 and get 95, that may be bad.

So maybe what the logic should do is: