Mutable structures not always recreated during shrink

[marklam]

If (say) an array is declared before the generator binds in a property computation expression, it doesn't get cleared when shrinking after a test failure. If the declaration is moved after the let! then it's ok.

This can produce misleading failure messages where the reported state is a combination of what happened during the shrinkage, not what caused the original failure.

As a (contrived) repro case, this test takes 10 integers and copies the distinct values into an array of int options. It then tests that the Some values are the sorted distinct values (so it fails).

let copyIntoArray (target : array<_>) source =
    source |> Array.iteri (fun i x -> target.[i] <- Some x)

[<Test>]
let ``strange`` () =
    property {
        let buffer = Array.zeroCreate 10

        let! input =
            Gen.int (Range.constant 1 100)
            |> Gen.array (Range.constant 10 10)

        // Failure message is sensible if buffer is declared here instead
        // let buffer = Array.zeroCreate 10

        let actual =                 
            input  |> Array.distinct |> copyIntoArray buffer
            buffer |> Array.choose id

        let expected = 
            input  |> Array.distinct |> Array.sort

        printfn "testing %A vs %A" actual expected

        test <@ actual = expected @>
    } |> Property.check

During shrinkage, the input array is made shorter but the buffer array has items left-over. The output reads:

testing [|38; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|13; 21; 23; 38; 43; 44; 56; 59; 80; 99|]
testing [|1; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|1; 13; 21; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 80; 43; 23; 59; 13; 56; 44; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 43; 23; 59; 13; 56; 44; 99; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 99|]
testing [|1; 23; 59; 13; 56; 44; 99; 99; 99; 99|] vs [|1; 13; 23; 44; 56; 59; 99|]
testing [|1; 59; 13; 56; 44; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 59; 99|]
testing [|1; 13; 56; 44; 99; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 99|]
testing [|1; 56; 44; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 56; 99|]
testing [|1; 44; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1|]
Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 10 shrinks):
[|1; 1; 1; 1; 1; 1; 1; 1; 1; 1|]
NUnit.Framework.AssertionException: 

[|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] = [|1|]
false

But if the buffer declaration is moved, the shrinkage works properly and the message reads:

Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 24 shrinks):
[|1; 1; 1; 1; 1; 1; 3; 2; 1; 1|]
NUnit.Framework.AssertionException: 

[|1; 3; 2|] = [|1; 2; 3|]
false

Even if this is by-design behaviour, it might be worth noting it in the docs

[moodmosaic]

Thank you for reporting this 💯 I'll have a look hopefully in the next few days and get back to you.

[moodmosaic]

I apologize for not getting back to this sooner, but now https://github.com/hedgehogqa/fsharp-hedgehog/issues/192#issuecomment-573017758 acts as a strong reminder.

[moodmosaic]

@marklam, you may want to follow https://github.com/hedgehogqa/fsharp-hedgehog/issues/192#issuecomment-575535702 until we get back on this one.