How to write your own (complex) generator?

[meling]

I've been playing around with gopter for a little while now, trying to understand how to write my own generator for my use case, which is not as straight forward as those in the repo. My use case is the following; I want to test a function (ReadQF) that should return a value and true when enough, i.e. a quorum of replies have been received and passed in to the ReadQF function. It should return false otherwise.

I've hacked together something that seems to work in the following:

https://github.com/relab/byzq/blob/master/authdataspec_property_test.go#L37 https://github.com/relab/byzq/blob/master/authdataspec_property_test.go#L63

However, I suspect it isn't quite in the spirit of property-based testing, and I'm struggling to break it up into multiple generators, since the input parameter n to the NewAuthDataQ constructor that creates a qspec object and computes the parameter q, which is used to decide the minimal/maximal length of the replies array. And furthermore, I need access to the qspec object in the end to decide if a quorum has been received.

I would really appreciate to get some feedback on my two tests linked above, especially, if you can provide some advice on how to decouple things.

(Below is an initial attempt at writing a generator, but I don't know how to get both the quorumSize and qspec parameters out of the generator for consumption in the condition function passed to prop.ForAll().)

func genQuorums(min, max int, hasQuorum bool) gopter.Gen {
    return func(genParams *gopter.GenParameters) *gopter.GenResult {
        rangeSize := uint64(max - min + 1)
        n := int(uint64(min) + (genParams.NextUint64() % rangeSize))
        qspec, err := NewAuthDataQ(n, priv, &priv.PublicKey)
        if err != nil {
            panic(err)
        }
        // initialize as non-quorum
        minQuorum, maxQuorum := math.MinInt32, qspec.q
        if hasQuorum {
            minQuorum, maxQuorum = qspec.q+1, qspec.n
        }
        rangeSize = uint64(maxQuorum - minQuorum + 1)
        quorumSize := int(uint64(minQuorum) + (genParams.NextUint64() % rangeSize))

        genResult := gopter.NewGenResult(qspec, gopter.NoShrinker)
        return genResult
    }
}

[untoldwind]

On first sight, the generator does not look that wrong to me. Though using the Sample() function in tests is kind of an anti-pattern since - in theory - it may not create a result for all generators (i.e. generators that have a SuchThat(...) sieve).

Unluckily go has not language support for tuples, so writing a generator for two or more parameters always requires some kind of wrapper "object".

So: If you need qspec and quorumSize the probably best way is to have a

struct qfParams {
  quorumSize int
  qspec AuthDataQ
}

in your tests and adapt the generator accordingly.

A somewhat better approach might be to combine generators via. Map(...) and FlatMap(...)

E.g.

gen.IntRange(min, max).FlatMap(func (_n interface{}) {
  n := _n.(int)
...
  return gen.IntRange(minQurom, maxQurom).Map(func (_quorumSize interface{}) {
    quorumSize := _quorumSize.(int)

    return &qfParams{
...
    }
  }
}

Hope that helps, otherwise I might take a closer look at a less stressful moment than right now ;)

[meling]

Thanks for the input and proposed combined generator; much appreciated. I had been looking at the Map and FlatMap before, but found it a bit difficult to understand without a good example that matched the complexity I needed.

Also, I agree that my use of Sample was perhaps the one thing that I disliked the most with my first approach and why I wanted to improve it.

Anyway, I tried to set it up as you suggested (barring a few adjustments to satisfy the API):

gen.IntRange(4, 100).FlatMap(func(n interface{}) gopter.Gen {
    qspec, err := NewAuthDataQ(n.(int), priv, &priv.PublicKey)
    if err != nil {
        t.Fatalf("failed to create quorum specification for size %d", n)
    }
    return gen.IntRange(qspec.q+1, qspec.n).Map(func(quorumSize interface{}) gopter.Gen {
        return func(*gopter.GenParameters) *gopter.GenResult {
            return gopter.NewGenResult(&qfParams{quorumSize.(int), qspec}, gopter.NoShrinker)
        }
    })
}, reflect.TypeOf(&qfParams{})),

See here for the full code: https://github.com/relab/byzq/blob/master/authdataspec_property_test.go#L100

I'm not quite sure what I'm doing wrong, but I get the following (partial) stack trace. I suspect that it is related to the reflect.TypeOf() at the end. Would appreciate some input on this, if possible. Thanks!!

! testing -- sufficient replies guarantees a quorum: Error on property
   evaluation after 0 passed tests: Check paniced: reflect: Call using
   gopter.Gen as type *byzq.qfParams goroutine 6 [running]:
runtime/debug.Stack(0xc420051640, 0x14ce120, 0xc4203f00c0)
    /usr/local/Cellar/go/1.9.2/libexec/src/runtime/debug/stack.go:24 +0xa7
github.com/leanovate/gopter.SaveProp.func1.1(0xc420051c40)
    /Users/meling/Dropbox/work/go/src/github.com/leanovate/gopter/prop.go:19
   +0x6e
panic(0x14ce120, 0xc4203f00c0)
    /usr/local/Cellar/go/1.9.2/libexec/src/runtime/panic.go:491 +0x283
reflect.Value.call(0x14dc040, 0xc420011b70, 0x13, 0x1595205, 0x4,
   0xc420153c80, 0x1, 0x1, 0x14cd6a0, 0xc4201a9a18, ...)

[untoldwind]

At first glance, I would say that gen.IntRange(qspec.q+1, qspec.n).Map(func ... needs to be a FlatMap since your returning a generator instead of a value.

But you're right, the error is not very helpful, I'll look into that. Unluckily reflection seems to be the only way to have the required flexibility ... alas, it's also a highly reusable booby trap ...

[untoldwind]

Just checked your example:

This here seems to work:

        gen.IntRange(4, 100).FlatMap(func(n interface{}) gopter.Gen {
            qspec, err := NewAuthDataQ(n.(int), priv, &priv.PublicKey)
            if err != nil {
                t.Fatalf("failed to create quorum specification for size %d", n)
            }
            return gen.IntRange(qspec.q+1, qspec.n).FlatMap(func(quorumSize interface{}) gopter.Gen {
                return func(*gopter.GenParameters) *gopter.GenResult {
                    return gopter.NewGenResult(&qfParams{quorumSize.(int), qspec}, gopter.NoShrinker)
                }
            }, reflect.TypeOf(&qfParams{}))
        }, reflect.TypeOf(&qfParams{})),

Though I think this one is what you're actually looking for:

        gen.IntRange(4, 100).FlatMap(func(n interface{}) gopter.Gen {
            qspec, err := NewAuthDataQ(n.(int), priv, &priv.PublicKey)
            if err != nil {
                t.Fatalf("failed to create quorum specification for size %d", n)
            }
            return gen.IntRange(qspec.q+1, qspec.n).Map(func(quorumSize interface{}) *qfParams {
                return &qfParams{quorumSize.(int), qspec}
            })
        }, reflect.TypeOf(&qfParams{})),

But it actually might be a good idea to allow the map function to accept GenResult as well ... and there has to be some better error reporting ...

[meling]

Thanks! I actually just discovered the same myself after you pointed out that I was returning a generator, which I thought was awkward... So that solved it!! Thanks for helping me with this. Moving on to testing more interesting properties. Feel free to close the issue.

[talgendler]

@meling I think you should use gen.Struct or gen.StructPtr for your case. You can look at the examples here: https://github.com/leanovate/gopter/blob/master/gen/struct_test.go#L18

[maurerbot]

I'm having a similar but much simpler problem. I want to generate a bunch of values to restrict ranges in generators. However, the Samples() I'm asking for are returning 0 always. I believe this is due to no RNG being set at the time I'm calling sample. What is the approach to make this work?

func genPreset() gopter.Gen {
    arbitraries := arbitrary.DefaultArbitraries()

    floatGen := gen.Float64Range(0.01, 1)
    arbitraries.RegisterGen(floatGen)
    sl, _ := floatGen.Sample()
    slv, _ := sl.(float64)
    ThresholdGen := gen.Float64Range(slv, 1)
    windowGen := gen.Float64Range(0, 120)
    aw, _ := windowGen.Sample()
    awv, _ := aw.(float64)
    windowGen = gen.Float64Range(awv, 120)
    dw, _ := windowGen.Sample()
    dwv, _ := dw.(float64)
    windowGen = gen.Float64Range(dwv, 120)
    sw, e := windowGen.Sample()
    swv, _ := sw.(float64)

    presetGens := map[string]gopter.Gen{
        "TargetRatioA":   floatGen,
        "TargetRatioD":   floatGen,
        "TargetRatioR":   floatGen,
        "ThresholdLevel": ThresholdGen,
    }
    return gen.StructPtr(reflect.TypeOf(&ADSRBasePreset{
        SustainLevel: slv, // value is 0
        Awindow:      awv, // value is 0
        Dwindow:      dwv, // value is 0
        Swindow:      swv, // value is 0
    }), presetGens)
}

func TestADSRBasePreset_SetupGraph(t *testing.T) {
    arbitraries := arbitrary.DefaultArbitraries()
    arbitraries.RegisterGen(genPreset())
    arbitraries.RegisterGen(genMag())
    arbitraries.RegisterGen(genHours())
    properties := gopter.NewProperties(nil)
    properties.Property("prop gen respects rules", arbitraries.ForAll(
        func(preset *ADSRBasePreset) bool {
            if preset.ThresholdLevel < preset.SustainLevel {
                return false
            }
            if preset.Awindow > preset.Dwindow || preset.Dwindow > preset.Swindow {
                return false
            }
            return true
        },
    ))

    properties.Property("SetupGraph()", arbitraries.ForAll(
        func(preset *ADSRBasePreset, mag float64, hours int64) bool {
            graph := preset.SetupGraph(float64(hours), mag)
            if graph.Decay.Coef > 0 || graph.Release.Coef > 0 || graph.Attack.Coef < 0 {
                return false
            }
            if graph.Attack.Base <= 0 {
                return false
            }
            return true
        },
    ))

    properties.TestingRun(t)

}

[talgendler]

@adrianmaurer there is also another way to create custom structs a more typesafe one:

func FullNameGen() gopter.Gen {
    return gopter.DeriveGen(
        func(first, last string) string {
            return first + " " + last
        },
        func(fullName string) (string, string) {
            split := strings.Split(fullName, " ")
            return split[0], split[1]
        },
        FirstNameGen(),
        LastNameGen(),
    )

Derive function accepts any number of generators - FirstNameGen,LastNameGen and runs them before creating the struct. Their artifacts are used as function arguments