Cannot reproduce tests with seed when restarting tests

[jonaslagoni]

I have this massive generator which creates an arbitrary complex struct however each time I run the test I get a different result. So I started implementing tests for the generators to see if I could reproduce results from the same generator using the same seed in in two instances of properties. This is the layout of the test and its support functions:

func SetupReproducibleProperties(seed *int64) (*gopter.Properties, *gopter.Properties) {
    var parameters1 *gopter.TestParameters
    var parameters2 *gopter.TestParameters
    if seed != nil {
        parameters1 = gopter.DefaultTestParametersWithSeed(*seed)
        parameters2 = gopter.DefaultTestParametersWithSeed(*seed)
    }else{
        parameters1 = gopter.DefaultTestParameters()
        parameters2 = gopter.DefaultTestParametersWithSeed(parameters1.Seed)
    }
    properties1 := gopter.NewProperties(parameters1)
    properties2 := gopter.NewProperties(parameters2)
    return properties1, properties2
}

func GetHashStringFromInterface(i interface{}) string {
    newJsonBytes, _ := json.Marshal(i)
    newHash := md5.Sum(newJsonBytes)
    return string(newHash[:])
}

func Test(t *testing.T) {
    var seedToUse *int64
    //seedToUse = Utils.Int64ToPtr(1604056858868077464)
    properties1, properties2 := SetupReproducibleProperties(seedToUse)
    generatorToUse := SomeGenerator
    var oldGens []GeneratedStruct
    properties1.Property("Generate first iteration", prop.ForAll(
        func(value GeneratedStruct) bool {
            oldGens = append(oldGens, value)
            return true
        },
        generatorToUse,
    ))
    counter := Utils.IntToPtr(0)
    properties2.Property("Comparing hashes should be a match", prop.ForAll(
        func(value GeneratedStruct) bool {
            currentHash := Utils.GetHashStringFromInterface(value)
            storedValue := oldGens[*counter]
            storedHash := Utils.GetHashStringFromInterface(storedValue)
            counter = Utils.IntToPtr(*counter + 1)
            if storedHash == currentHash {
                return true
            }
            return false
        },
        generatorToUse,
    ))
    properties1.TestingRun(t)
    properties2.TestingRun(t)
}

Running this test always succeeds. Same goes if you replace GeneratedStruct with string and SomeGenerator with gen.AnyString(). However when I close the test down and start it up again I start seeing different results. If I use the AnyString example it generates the same value when I run the test and restart it. If I do it with my custom generated struct it does not generate the same result.

One of the generators which cannot be reproduced, I do apologise it is quite comprehensive.

#### The Structure ##### schema.SelectOption ```golang type SelectOption struct { // gopg specific fields to make database management easier tableName struct{} `pg:"selectoptions"` // Fields specified in AsyncAPI configuration Value string `json:"value" ` Selected *bool `json:"selected,omitempty" ` Name string `json:"name" ` SubsetQuestions *QuestionsKeyObject `json:"subsetQuestions,omitempty" ` } ``` ##### schema.Question The **QuestionsKeyObject** and **Mapper** are never generated so ignore them ```golang type Question struct { // gopg specific fields to make database management easier tableName struct{} `pg:"questions"` // Fields specified in AsyncAPI configuration Id *string `json:"id,omitempty" ` Type QuestionTypesEnum `json:"type" ` Hint *string `json:"hint,omitempty" ` Placeholder *string `json:"placeholder,omitempty" ` DefaultValue *string `json:"defaultValue,omitempty" ` Options []SelectOption `json:"options,omitempty" ` CheckedSubsetQuestions *QuestionsKeyObject `json:"checkedSubsetQuestions,omitempty" ` UncheckedSubsetQuestions *QuestionsKeyObject `json:"uncheckedSubsetQuestions,omitempty" ` Questions *QuestionsKeyObject `json:"questions,omitempty" ` Mappers []Mapper `json:"mappers,omitempty" ` } ``` ##### schema.QuestionTypesEnum ```golang type QuestionTypesEnum int type questionTypesStruct struct { Text QuestionTypesEnum `json:"text"` Select QuestionTypesEnum `json:"select"` Container QuestionTypesEnum `json:"container"` Textarea QuestionTypesEnum `json:"textarea"` Checkbox QuestionTypesEnum `json:"checkbox"` _values [5]QuestionTypesEnum _names [5]string } var QuestionTypes = &questionTypesStruct{ Text: 0, Select: 1, Container: 2, Textarea: 3, Checkbox: 4, _values: [5]QuestionTypesEnum{ 0, 1, 2, 3, 4}, _names: [5]string{ "text", "select", "container", "textarea", "checkbox", }, } ``` #### The Test ```golang func TestQuestionGenShouldReproduceResults(t *testing.T) { var seedToUse *int64 //seedToUse = Utils.Int64ToPtr(1604056858868077464) properties1, properties2 := SetupReproducibleProperties(seedToUse) generatorToUse := QuestionGenWithOptions() var oldGens []schema.Question properties1.Property("Generate first iteration", prop.ForAll( func(question schema.Question) bool { oldGens = append(oldGens, question) return true }, generatorToUse, )) counter := Utils.IntToPtr(0) properties2.Property("Comparing hashes should be a match", prop.ForAll( func(question schema.Question) bool { questionHash := Utils.GetHashStringFromInterface(question) storedQuestion := oldGens[*counter] storedHash := Utils.GetHashStringFromInterface(storedQuestion) counter = Utils.IntToPtr(*counter+1) if storedHash == questionHash { return true } return false }, generatorToUse, )) properties1.TestingRun(t) properties2.TestingRun(t) } func QuestionGenWithOptions() gopter.Gen { questionGenOptionsGen := gen.PtrOf(QuestionGenOptionsGen()).WithLabel("QuestionGenOptionsGenPtrOf") questionGenOptionsGen = questionGenOptionsGen.FlatMap(func(i interface{}) gopter.Gen { var questionGenOptions *QuestionGenOptions if i != nil { questionGenOptions = i.(*QuestionGenOptions) } return SpecificQuestionGenWithOptions(schema.Question{}, questionGenOptions) }, reflect.TypeOf(schema.Question{})).WithLabel("QuestionGenOptionsGenFlatMap") return questionGenOptionsGen } func QuestionIdGenOptionsGen() gopter.Gen { presetGens := map[string]gopter.Gen{ "DifferentThen": gen.PtrOf(gen.SliceOfN(10, gen.PtrOf(gen.AnyString()))).WithLabel("QuestionIdGenOptionsGenDifferentThen"), "NotNil": gen.PtrOf(gen.Bool()).WithLabel("QuestionIdGenOptionsGenNotNil"), "UseNil": gen.PtrOf(gen.Bool()).WithLabel("QuestionIdGenOptionsGenUseNil"), "GenUniqueId": gen.PtrOf(gen.Bool()).WithLabel("QuestionIdGenOptionsGenGenUniqueId"), } return gen.Struct(reflect.TypeOf(QuestionIdGenOptions{}), presetGens).WithLabel("QuestionIdGenOptionsGen") } func QuestionGenOptionsGen() gopter.Gen { presetGens := map[string]gopter.Gen{ "IdGenOptions": gen.PtrOf(QuestionIdGenOptionsGen()).WithLabel("QuestionGenOptionsGenIdGenOptions"), "ExcludeId": gen.PtrOf(gen.Bool()).WithLabel("QuestionGenOptionsGenExcludeId"), } return gen.Struct(reflect.TypeOf(QuestionGenOptions{}), presetGens).WithLabel("QuestionGenOptionsGen") } ``` #### The Generator to test ```golang type QuestionGenOptions struct { //Use a custom generator for ids IdGen *gopter.Gen //Affect the current id generator by providing options IdGenOptions *QuestionIdGenOptions //Dont generate id, it will then always get the value nil ExcludeId *bool } type QuestionIdGenOptions struct { //Used to not generate the same id DifferentThen *[]*string //Used to ensure that nil are not generated, the generator returned are no longer a pointer NotNil *bool UseNil *bool //Generate unique ids GenUniqueId *bool alreadyGeneratedQuestions *[]string } func QuestionIdGen(questionIdGenOptions *QuestionIdGenOptions) gopter.Gen { if questionIdGenOptions != nil && questionIdGenOptions.alreadyGeneratedQuestions == nil { var newAlreadyGeneratedQuestions = []string{} questionIdGenOptions.alreadyGeneratedQuestions = &newAlreadyGeneratedQuestions } var pgen gopter.Gen if questionIdGenOptions != nil && questionIdGenOptions.UseNil != nil && *questionIdGenOptions.UseNil == true { var data *string = nil pgen = gen.Const(data).WithLabel("QuestionIdGenNil") }else{ pgen = gen.PtrOf(gen.AnyString()).WithLabel("QuestionIdGenPtrOf") } if questionIdGenOptions != nil { pgen = pgen.SuchThat( func(generatedId *string) bool { //If they want nil dont make it different then if (questionIdGenOptions.UseNil == nil || questionIdGenOptions.UseNil != nil && *questionIdGenOptions.UseNil == false) && questionIdGenOptions.DifferentThen != nil && generatedId != nil && Utils.ContainsPointers(*questionIdGenOptions.DifferentThen, generatedId) { return false } if questionIdGenOptions.NotNil != nil && *questionIdGenOptions.NotNil == true && generatedId == nil { return false } //Even if the characters are the same the bytes are not i.e. it is therefore unique. //if questionIdGenOptions.GenUniqueId != nil && *questionIdGenOptions.GenUniqueId == true && (generatedId == nil || Utils.Contains(*questionIdGenOptions.alreadyGeneratedQuestions, *generatedId)) { // return false //} if generatedId != nil { newAlreadyGeneratedQuestions := append(*questionIdGenOptions.alreadyGeneratedQuestions, *generatedId) questionIdGenOptions.alreadyGeneratedQuestions = &newAlreadyGeneratedQuestions } return true }).WithLabel("QuestionIdGenSuchThat") } return pgen.WithLabel("QuestionIdGen") } func SpecificQuestionGenWithOptions(presetQuestions schema.Question, options *QuestionGenOptions) gopter.Gen { presetGens := map[string]gopter.Gen{ "Type": gen.OneConstOf( schema.QuestionTypes.Checkbox, schema.QuestionTypes.Select, schema.QuestionTypes.Container, schema.QuestionTypes.Text, schema.QuestionTypes.Textarea).WithLabel("QuestionTypeGen"), } //Based on the already existing presetQuestions decide if we should keep the values if presetQuestions.Id != nil { presetGens["Id"] = gen.Const(presetQuestions.Id).WithLabel("QuestionIdGen1") } else { if options != nil && options.IdGen != nil { presetGens["Id"] = (*options.IdGen).WithLabel("QuestionIdGen2") } else if options != nil { presetGens["Id"] = QuestionIdGen(options.IdGenOptions).WithLabel("QuestionIdGen3") }else{ presetGens["Id"] = QuestionIdGen(nil).WithLabel("QuestionIdGen4") } } if presetQuestions.Hint != nil { presetGens["Hint"] = gen.Const(presetQuestions.Hint).WithLabel("QuestionHintGen1") } else { presetGens["Hint"] = gen.PtrOf(gen.AnyString()).WithLabel("QuestionHintGen2") } if presetQuestions.Placeholder != nil { presetGens["Placeholder"] = gen.Const(presetQuestions.Placeholder).WithLabel("QuestionHintGen1") } else { presetGens["Placeholder"] = gen.PtrOf(gen.AnyString()).WithLabel("QuestionHintGen1") } if presetQuestions.DefaultValue != nil { presetGens["DefaultValue"] = gen.Const(presetQuestions.DefaultValue).WithLabel("QuestionDefaultValueGen1") } else { presetGens["DefaultValue"] = gen.PtrOf(gen.AnyString()).WithLabel("QuestionDefaultValueGen1") } if presetQuestions.Questions != nil { presetGens["Questions"] = gen.Const(presetQuestions.Questions).WithLabel("QuestionQuestionsGen") } if presetQuestions.CheckedSubsetQuestions != nil { presetGens["CheckedSubsetQuestions"] = gen.Const(presetQuestions.CheckedSubsetQuestions).WithLabel("QuestionCheckedSubsetQuestionsGen") } if presetQuestions.Mappers != nil { presetGens["Mappers"] = gen.Const(presetQuestions.Mappers).WithLabel("QuestionMappersGen") } if presetQuestions.UncheckedSubsetQuestions != nil { presetGens["UncheckedSubsetQuestions"] = gen.Const(presetQuestions.UncheckedSubsetQuestions).WithLabel("QuestionUncheckedSubsetQuestionsGen") } standardQuestion := gen.Struct(reflect.TypeOf(schema.Question{}), presetGens).WithLabel("QuestionStandardGen") if presetQuestions.Options == nil { standardQuestion = standardQuestion.FlatMap( func(generatedValue interface{}) gopter.Gen { generatedQuestion := generatedValue.(schema.Question) presetGens := map[string]gopter.Gen{ "Type": gen.Const(generatedQuestion.Type), "Hint": gen.Const(generatedQuestion.Hint), "Id": gen.Const(generatedQuestion.Id), "Placeholder": gen.Const(generatedQuestion.Placeholder), "DefaultValue": gen.Const(generatedQuestion.DefaultValue), "Questions": gen.Const(generatedQuestion.Questions), "CheckedSubsetQuestions": gen.Const(generatedQuestion.CheckedSubsetQuestions), "Mappers": gen.Const(generatedQuestion.Mappers), "Options": gen.Const(generatedQuestion.Options), "UncheckedSubsetQuestions": gen.Const(generatedQuestion.UncheckedSubsetQuestions), } if presetQuestions.Options != nil { presetGens["Options"] = gen.Const(presetQuestions.Options).WithLabel("QuestionOptionsGen") }else{ switch generatedQuestion.Type { case schema.QuestionTypes.Select: presetGens["Options"] = generateOptions() break } } return gen.Struct(reflect.TypeOf(schema.Question{}), presetGens) }, reflect.TypeOf(schema.Question{}), ).WithLabel("QuestionOptionsGenFlatMap") } return standardQuestion } func generateOptions() gopter.Gen { optionsGen := func(genParams *gopter.GenParameters) *gopter.GenResult { genParams.MinSize = 1 genParams.MaxSize = 10 presetGens := map[string]gopter.Gen{ "Value": gen.AnyString().WithLabel("OptionsSliceValueGen"), "Name": gen.AnyString().WithLabel("OptionsSliceNameGen"), } return gen.SliceOf(gen.Struct(reflect.TypeOf(schema.SelectOption{}), presetGens).WithLabel("OptionGen")).WithLabel("OptionsSliceGen")(genParams) } return optionsGen } ```

Do you have any idea what might do this? I'll investigate further and keep adding comments, but thought it was time to share the bug or if it is my wrong doing. Keep in mind Go is new for me so if anything looks weird I apologise :smile:

[untoldwind]

This is indeed quite complex ;)

I do not know what Utils.GetHashStringFromInterface actually does, especially since your generated structs contain pointer (which naturally differ for each run).

To narrow down the problem, I'd suggest to marshal the struct to a json-string and compare that instead of the hash (i.e. just use json.Marshal instead of Utils.GetHashStringFromInterface) If there is a difference you can dump both string that and we might get an idea witch of the many nested generators went haywire.

[jonaslagoni]

This is indeed quite complex ;)

I do not know what Utils.GetHashStringFromInterface actually does, especially since your generated structs contain pointer (which naturally differ for each run).

To narrow down the problem, I'd suggest to marshal the struct to a json-string and compare that instead of the hash (i.e. just use json.Marshal instead of Utils.GetHashStringFromInterface) If there is a difference you can dump both string that and we might get an idea witch of the many nested generators went haywire.

Hey @untoldwind the Utils.GetHashStringFromInterface method can be seen in the main test example as the method GetHashStringFromInterface. Furthermore it is not the actual test that fails when I restart the test, that succeeds perfectly fine. However when I manually inspect the test run between first time and second time that the results varies with the custom generator. i.e. this is what I did:

1. Debug the test and see what the result of the first generated value is. i.e. one property of the struct GeneratedStruct could be "somerandomString"
2. Stop the test and debug it again.
3. Debug test and see what the result of the first generated value is. i.e. one property of the struct GeneratedStruct could be "somerandomString2" however when you first tried it it was "somerandomString" even though it is the same seed. This changes for each time I run it.

Hope that clarified my problem a bit more.

[untoldwind]

Sorry, I probably still don't get it.

I thought the problem is that in your example properties1.Property and properties2.Property where generating different GeneratedStruct even though the both use the same random seed. If this is the case then it would be helpful to see the actual difference via json.

Otherwise if you un-comment seedToUse = Utils.Int64ToPtr(1604056858868077464) the test should always have a fixed seed and always produce the same result. If not ... again it would be helpful to have a json to see where exactly things went wrong.

[jonaslagoni]

Had to double check that I wasn't that stupid...

Checked that I did have the correct seed in the tests and checked pointer and ensured they weren't affecting the generations in any way and trying to narrow down where it goes wrong, tried duplicating tests with custom values etc. So it all came down to how I set the seed... Which was different between tests (did it one way, learned about a new and continued using that)

The function I used to set the parameters and seed for the example was

func SetupReproducibleProperties(seed *int64) (*gopter.Properties, *gopter.Properties) {
    var parameters1 *gopter.TestParameters
    var parameters2 *gopter.TestParameters
    if seed != nil {
        parameters1 = gopter.DefaultTestParametersWithSeed(*seed)
        parameters2 = gopter.DefaultTestParametersWithSeed(*seed)
    }else{
        parameters1 = gopter.DefaultTestParameters()
        parameters2 = gopter.DefaultTestParametersWithSeed(parameters1.Seed)
    }
    properties1 := gopter.NewProperties(parameters1)
    properties2 := gopter.NewProperties(parameters2)
    return properties1, properties2
}

And in the other tests I did something like the following:

func SetupReproducibleProperties(seed *int64) (*gopter.Properties, *gopter.Properties) {
    var parameters1 *gopter.TestParameters
    var parameters2 *gopter.TestParameters
    if seed != nil {
        parameters1 = gopter.DefaultTestParameters()
        parameters1.Seed = *seed
        parameters2 = gopter.DefaultTestParameters()
        parameters2.Seed = *seed
    }else{
        parameters1 = gopter.DefaultTestParameters()
        parameters2 = gopter.DefaultTestParameters()
        parameters2.Seed = parameters1.Seed
    }
    properties1 := gopter.NewProperties(parameters1)
    properties2 := gopter.NewProperties(parameters2)
    return properties1, properties2
}

Changing to this implementation fails the test immediately.

:facepalm: setting the Seed manually on the default test parameters does of course not work since it does not regenerate the underlying Rng property... Kinda assumed that parsing the parameters to the NewProperties method would auto generate the underlying Rng property, sorry for have wasted your time...

Thought a little about whether there should be some comments explaining this, but dont see where, what you think?

[jonaslagoni]

We could move when the Rng gets initialized i.e. to https://github.com/leanovate/gopter/blob/a173b3f6792cb4c0055d8f94a1faefd6d41c18b5/properties.go#L18 with something like

if parameters == nil {
    parameters = DefaultTestParameters()
} else {
        parameters.Rng = rand.New(NewLockedSource(parameters.seed))
}

Then it should be me safe if no one uses the Rng before it is wrapped in a property :smile:

[untoldwind]

I'm glad it turned out to be something simple.

Unluckily there are already some examples using parameters.Rng.Seed(number) which would break when moving the creation of the Rng to NewProperties. Instead I decided to make the Seed property of TestParameters private and added a setter for it.

I hope this fixes your problem just as well. (At the very least you get an easily fixable compile error)

[jonaslagoni]

Unluckily there are already some examples using parameters.Rng.Seed(number) which would break when moving the creation of the Rng to NewProperties. Instead I decided to make the Seed property of TestParameters private and added a setter for it.

I hope this fixes your problem just as well. (At the very least you get an easily fixable compile error)

Perfect, thanks!

[untoldwind]

Closing the issue then