Generators generate the same values for each test?

[AlexeyRaga]

I have two tests that are defined as:

[Property]
public async Task GetUserById(User user) { ... }

[Property]
public async Task GetUsersById(User user1, User user2) { ... }

The generator is defined using LINQ like:

    public static Gen<User> User() =>
        from username in Gen.AlphaNumeric.String(Range.LinearInt32(5, 255))
        from firstName in Gen.AlphaNumeric.String(Range.LinearInt32(3, 255))
        from lastName in Gen.AlphaNumeric.String(Range.LinearInt32(3, 255))
        select new User
        (
            Username: username,
            FirstName: firstName,
            LastName: lastName);

and the config is:

public static class UsersGenConfig
{
    [UsedImplicitly]
    public static AutoGenConfig Config() =>
        GenX.defaults
            .WithGenerator(UserGen.User())
}

I mark my tests with

[Properties(typeof(UsersGenConfig), Tests = 1, Shrinks = 0)]

But then when I run these tests in the same test run, I see that GetUserById and GetUsersByIdget the sameUser`!

GetUsersById receives two different users, then GetUserById receives one user, which is one of these that GetUsersById previously received.

I played with it a bit, asking for more users, and it seems like the test that gets called second, would get the parameters that were previously passed to the test that was called first.

Is it by design? It looks like the splitting some random generator issue to me? Like if it was split, but then the wrong part was passed down...

[TysonMN]

Is this behavior bad or just surprising to you?

[AlexeyRaga]

Both.

It is bad because it makes the amount of variation smaller. It also not integration-tests friendly at all (and this is where I had to stop using Hedgehog in this case).

It is also extremely surprising because when asking to generate something I would totally expect at least some variation and not a full copy of what I already had before in a previous test.

Are you saying that it is by design @TysonMN ?

[TysonMN]

I think the first item generated is always the smallest element in the sample space. More generally, there is a size parameter that controls how big the sample space is. It starts out small and typically grows with each test case. Initially preferring small values improves the result found during shrinking.

Your code has extreme behavior because you specified that each test should only consider one test case.

[dharmaturtle]

Your code has extreme behavior because you specified that each test should only consider one test case.

Yep.

GetUsersById receives two different users, then GetUserById receives one user, which is one of these that GetUsersById previously received.

My question would be is if this is always the case or often the case. I would expect often, but not always.

it seems like the test that gets called second, would get the parameters that were previously passed to the test that was called first. Is it by design?

No. I would expect diverging results as the size increases. They only collide with test=1 because the size is so small.

Edit: If you want to run just one test because integration tests are slow, you can pin the size to a larger value as documented here.

[AlexeyRaga]

My question would be is if this is always the case or often the case. I would expect often, but not always.

Out of all the runs that I did manually it was always the case. But, of course, I cannot say that there cannot be a run in which it won't be the case.

If you want to run just one test because integration tests are slow, you can pin the size to a larger value

Thanks, I will try that!

[TysonMN]

If you want to run just one test because integration tests are slow, you can pin the size to a larger value

I don't expect that will work though, because I expect the generator always returns the smallest value as it's first sample regardless of size.

Instead, if you only want to run a slow test one time, then I think you should hardcore the values for that test.

[dharmaturtle]

Hm, maybe that's the case in Hedgehog proper, but empirically its not the case in Hedgehog.Xunit:

Out of all the runs that I did manually it was always the case.

I do find that surprising. Running the above test a few dozen times with 0 size I always observed 0 for int and "" for string, but float always changed. Were you seeing just the same value over and over again?

[AlexeyRaga]

if you only want to run a slow test one time

I do not. I certainly don't want to run them 100 times, but 1 was just me debugging the issue. I normally go around 5-10 for integration tests.

Were you seeing just the same value over and over again?

No, I have two tests (two properties) in one run. The values always change between runs, but within the same run the property that runs second always observes the value that the first property has seen.

[AlexeyRaga]

Consider this example:

open System
open Hedgehog
open Hedgehog.Xunit

type User = { id: Guid; name: String; age: int }

[<Properties(Tests = 1<tests>, Shrinks=0<shrinks>)>]
module Tests =

    [<Property>]
    let ``Test A`` (user1: User) =
        user1.id = user1.id

    [<Property>]
    let ``Test B`` (user1: User, user2: User) =
        user1.id = user1.id

When I run it I always see that user2 in Test b is the same as user1 in Test A. Even if I specify a very large Size.

[AlexeyRaga]

I just tested it with a "plain" Hedgehog (without Hedgehog.Xunit) and it has the same behaviour...

[AlexeyRaga]

I just quickly checked Haskell Hedgehog, and it doesn't have this behaviour. As expected, each test Perhaps that's why it was so surprising to me, I have never seen that before.

Here is a reference test:

module TestSpec where

import HaskellWorks.Hspec.Hedgehog
import Hedgehog
import qualified Hedgehog.Gen                as Gen
import qualified Hedgehog.Range              as Range
import Test.Hspec
import Debug.Trace

data User = User {
  name :: String,
  age :: Int }
  deriving (Show, Eq)

userGen :: MonadGen m => m User
userGen = User
  <$> Gen.string (Range.linear 1 10) Gen.alpha
  <*> Gen.int (Range.linear 0 100)

{- HLINT ignore "Redundant do"        -}
spec :: Spec
spec = describe "First Spec" $ do
  it "test A" $ require $ withTests 1 $ withShrinks 0 $ property $ do
    user1 <- forAll userGen
    traceShowId user1 === user1

  it "test B" $ require $ withTests 1 $ withShrinks 0 $ property $ do
    user1 <- forAll userGen
    user2 <- forAll userGen
    traceShowId user1 === traceShowId user2

[TysonMN]

...1 was just me debugging the issue.

Then you should be calling Property.recheck. I implemented (what I call) "efficient rechecking" to make this experience great.

With this library, you do that be adding the Recheck attribute to your test.

If you are only debugging, then why do you care that the two tests in the same run are given the same input?

[AlexeyRaga]

Then you should be calling Property.recheck. I implemented (what I call) "efficient rechecking" to make this experience great.

Sorry for not being clear in my message. Recheck is fine, and it is nothing to do with the rechecking. What I meany was that I was debugging this issue. The issue, to me, is that the second test was getting the same generated values as the first one. It happens with tests=100 or tests=5, or anything. I just tests=1 for the ease of looking into the generated values.

So let's not concentrate on the number of tests that are being run per property. The issue is that previously generated values are re-used again for subsequent properties. It doesn't seem to have anything to do with the number of tests or the value of Size.

[TysonMN]

I think this is the intended behavior. As I said before:

I think the first item generated is always the smallest element in the sample space.

[AlexeyRaga]

I think this is the intended behavior

That's sad. I hoped that it is more accidental than intentional... Because it is hard for me to guess what would be the (non-technical) reasoning behind the decision to diverge from the original (Haskell) Hedgehog behaviour...

[AlexeyRaga]

I think the first item generated is always the smallest element in the sample space.

I do not think that it is about the first element only. It then keeps generating the same values.

In the example below, there are two tests receiving two User parameters, each runs 5 time. The output confirms that both tests always receive the same values in the same order.

So we are essentially running tests with the same generated values.

open System
open Hedgehog
open Hedgehog.Xunit
open Xunit.Abstractions

type User = { id: Guid; name: String; age: int }

[<Properties(Tests = 5<tests>, Shrinks=0<shrinks>)>]
type Tests(output: ITestOutputHelper) =

    [<Property>]
    let ``Test A`` (user1: User, user2: User) =
        output.WriteLine ($"{user1.id}, {user2.id}")
        user1.id = user1.id

    [<Property>]
    let ``Test B`` (user1: User, user2: User) =
        output.WriteLine ($"{user1.id}, {user2.id}")
        user1.id = user1.id

Output:

Test A:
7f6a6fbc-5eb7-d400-1287-fc44810c7145, e1916763-61f3-d3e6-1937-2d0c6f057070
3878dd01-bf21-0b58-06d4-2b5a555ec0fd, 822eef00-5682-aae4-8736-b1e560341949
e008cdcc-c2c0-18f4-e64b-641052b5515b, 781bf826-81a3-2aed-5091-d3f718ab07be
3df9967b-5bc0-56a3-6fe4-4d33b603404d, 10b5a7f6-8a30-6d9a-18e1-9d58ee88f625
6679d369-9022-c4ce-11f5-b0a06bb0893f, 70ba9843-0d6a-af64-dbeb-f359e854078a

Test B:
7f6a6fbc-5eb7-d400-1287-fc44810c7145, e1916763-61f3-d3e6-1937-2d0c6f057070
3878dd01-bf21-0b58-06d4-2b5a555ec0fd, 822eef00-5682-aae4-8736-b1e560341949
e008cdcc-c2c0-18f4-e64b-641052b5515b, 781bf826-81a3-2aed-5091-d3f718ab07be
3df9967b-5bc0-56a3-6fe4-4d33b603404d, 10b5a7f6-8a30-6d9a-18e1-9d58ee88f625
6679d369-9022-c4ce-11f5-b0a06bb0893f, 70ba9843-0d6a-af64-dbeb-f359e854078a

[TysonMN]

Now this looks like a bug to me.

Can you achieve the same behavior without Hedgehog.Xunit?

[AlexeyRaga]

Apparently yes.

Back to basics:

open System
open Xunit
open Hedgehog
open Xunit.Abstractions

type User = { id: Guid; name: String; age: int }

let propertyConfig = PropertyConfig.defaultConfig |> PropertyConfig.withTests 5<tests> |> PropertyConfig.withoutShrinks

let userGen = 
    gen {
        let! id = Gen.guid
        let! name = Gen.string (Range.linear 0 100) Gen.alpha
        let! age = Gen.int32 (Range.linear 0 100)
        return { id = id; name = name; age = age }
    }

type PureTests(output: ITestOutputHelper) =

    [<Fact>]
    let ``Test A`` () =
        property {
            let! user1 = userGen
            let! user2 = userGen
            output.WriteLine ($"{user1}, {user2}")
            user1.id = user1.id
        } |> Property.checkBoolWith propertyConfig

    [<Fact>]
    let ``Test B`` () =
        property {
            let! user1 = userGen
            let! user2 = userGen
            output.WriteLine ($"{user1}, {user2}")
            user1.id = user1.id
        } |> Property.checkBoolWith propertyConfig

It prints exactly the same users from both tests.

This is why I suspect that we somewhere split the generator into two, use one part for a test, and pass it to the next test instead of the unused one...

[AlexeyRaga]

Any word on this one?

[TysonMN]

No progress. Sorry. I don't have much time to maintain this project any more.