leanovate / gopter

GOlang Property TestER
MIT License
600 stars 40 forks source link

Generating values which depend on each other fails #79

Open alfert opened 3 years ago

alfert commented 3 years ago

I am looking for a way to generate values which depend on each other. In particular, I want to recreate the following simple example from ScalaCheck's User Guide (https://github.com/typelevel/scalacheck/blob/main/doc/UserGuide.md#generators), where two integer are generated, the first in the range of 10..20, the second has a lower bound which is twice as large as the first value:

// ScalaCheck Example
val myGen = for {
  n <- Gen.choose(10,20)
  m <- Gen.choose(2*n, 500)
} yield (n,m)

My impression was that the gen.FlatMap() should provide the required functionality (in Scala, <- is a monadic assignment, implemented by FlatMap), but I failed to find a way to succeed.

I defined a simple struct to generate two values which can be fed into the property:

type IntPair struct {
        Fst int
        Snd int
    }
properties.Property("ScalaCheck example for a pair", prop.ForAll(
        func(p IntPair) bool {
            a := p.Fst
            b := p.Snd
            return a*2 <= b
        },
        genIntPairScala(),
    ))

The generator is a straight translation of the Scala code, first generating an integer and then generating a second via accessing the generated value of the first. Both generators are finally stored in the struct generator:

genIntPairScala := func() gopter.Gen {
        n := gen.IntRange(10, 20).WithLabel("n (fst)")
        m := n.FlatMap(func(v interface{}) gopter.Gen {
            k := v.(int)
            return gen.IntRange(2*k, 50)
        }, reflect.TypeOf(int(0))).WithLabel("m (snd)")

        var gen_map = map[string]gopter.Gen{"Fst": n, "Snd": m}
        return gen.Struct(
            reflect.TypeOf(IntPair{}),
            gen_map,
        )
    }

However, it does not work:

=== RUN   TestGopterGenerators
! ScalaCheck example for a pair: Falsified after 10 passed tests.
n (fst), m (snd): {Fst:17 Snd:32}
n (fst), m (snd)_ORIGINAL (1 shrinks): {Fst:19 Snd:32}
Elapsed time: 233.121µs
    properties.go:57: failed with initial seed: 1617636578517672000

Remark: I set the upper bound to 50 instead of 500. The property must still hold, but the generator has a smaller pool to pick suitable values: setting the upper bound to 500 often results in a passing property!

untoldwind commented 3 years ago

The problem here is that the final generator is not the result of a FlatMap. I.e. "n" and "m" are completely independent generators within the struct-generator

I thing the correct way would look something like this:

gen.IntRange(10, 20).FlatMap(func(v interface{}) gopter.Gen {
   n := v.(int)
   var gen_map = map[string]gopter.Gen{"Fst": gen.Const(n), "Snd": gen.IntRange(2*k, 50) }
   return gen.Struct(
            reflect.TypeOf(IntPair{}),
            gen_map,
        )
}

Hope this makes sense

alfert commented 3 years ago

Thanks, that works indeed. Here is the solution a bit reformatted:

genIntPair := func() gopter.Gen {
        return gen.IntRange(10, 20).FlatMap(func(v interface{}) gopter.Gen {
            k := v.(int)
            n := gen.Const(k)
            m := gen.IntRange(2*k, 50)
            var gen_map = map[string]gopter.Gen{"Fst": n, "Snd": m}
            return gen.Struct(
                reflect.TypeOf(IntPair{}),
                gen_map,
            )
        },
        reflect.TypeOf(int(0)))
    }

So the trick is that the first generated integer value must be re-introduced as generator by applying Const, the trivial generator (akin to return in a monadic setting).

If you have more dependencies some syntactic sugar would be nice, but this seems to be difficult in Go.

untoldwind commented 3 years ago

Scalacheck works well because of scala's for-comprehention notation, which is a very nice way to write these map/flatMap cascades. Your example

val myGen = for {
  n <- Gen.choose(10,20)
  m <- Gen.choose(2*n, 500)
} yield (n,m)

actually expands to something like:

Gen.choose(10, 20).flatMap(n -> Gen.choose(2*n, 500).map(m -> (n,m))

I think you could write it like this in go as well, though you have to be very careful when using external variables in anonymous functions.

alfert commented 3 years ago

I like your FlatMap -> Map approach. This boils down to

genIntPair := func() gopter.Gen {
        return gen.IntRange(10, 20).FlatMap(func(v interface{}) gopter.Gen {
            k := v.(int)
            return gen.IntRange(2*k, 50).Map(func(m int) IntPair {
                return IntPair{Fst: k, Snd: m}
            })
        },
            reflect.TypeOf(int(0)))
    }

This is still baroque, but way more to the point than the first version. I will update my example PR #80