hedgehogqa / fsharp-hedgehog

Release with confidence, state-of-the-art property testing for .NET.
https://hedgehogqa.github.io/fsharp-hedgehog/
Other
271 stars 31 forks source link

Recheck runs property multiple times #420

Closed dharmaturtle closed 1 year ago

dharmaturtle commented 1 year ago

Stealing code from this issue:

    testCase "my test2" <| fun () ->
        property {
            let! i = Gen.int32 (Range.linear 1 10)
            System.Diagnostics.Debugger.Break()
            Expect.isTrue (i <= 5)
        }
        |> Property.recheck "59_15683818216235017248_6890634259096173777_"

The breakpoint is hit multiple times. I think Hedgehog is rerunning the shrink runs. The values of i in chronological order are as follows: 6, 1, 3, 5. I'm guessing this isn't expected behavior?

dharmaturtle commented 1 year ago

It isn't obvious to me that reportRecheckWith should followShrinkPath: https://github.com/hedgehogqa/fsharp-hedgehog/blob/b48f662ba42541dd51b0a116cca2489339d14d6a/src/Hedgehog/Property.fs#L250

Not sure what Status should be though. Pattern match on splitAndRun's result?

                let journal, outcome = root.Value
                match outcome with
                | Failure ->
                    { Shrinks = 0<shrinks>
                      Journal = journal
                      RecheckInfo = None }
                    |> Failed
                | Success _ -> OK
                | Discard -> failwith "Unexpected 'Discard' result when rechecking. This should never happen."

A general gut feeling answer would suffice. I'm mostly trying to figure out the raison d'etre for followShrinkPath.

TysonMN commented 1 year ago

I think you need to add return on the last line of your CE. This is a limitation imposed by F#.

Can you check and report back?

TysonMN commented 1 year ago

I don't understand your second comment. I will call you later to discuss.

dharmaturtle commented 1 year ago

return Expect.isTrue (i <= 5) does indeed only run once with recheck.

Kinda wish there was some way to make the dev experience on this better.