search

flyingmutant / rapid

Rapid is a modern Go property-based testing library
https://pkg.go.dev/pgregory.net/rapid
Mozilla Public License 2.0
579 stars 25 forks source link

Model based/stateful testing with .Repeat does not test sufficiently different distributions of each operation #69

Open danwt opened 1 month ago

danwt commented 1 month ago

Because of this it is necessary to have bogus logic to implement it yourself

flyingmutant commented 1 month ago

Can you expand a bit, what kind of distribution do you want to achieve?

danwt commented 1 month ago

Hi thanks for getting back so quick

So for example, given operations A,B, a program bug may be exposed only with a sequence of operations

AAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAA

or other such 'skewed' distributions

Obviously it's not easy to guess the distribution that would find the bug beforehand, that's the goal of the library in this case.

So I would suppose a naive approach, but one that might be useful in practice, is to have distributions such as

A,B,C
0.33,0.33,0.33
1,0,0
0,1,0
0,0,1
0.5,0.5,0
0.5,0,0.5
0,0.5,0.5
0.99,0.01,0.01
0.01,0.99,0.01
0.01,0.01,0.99
...
etc

I appreciate this is not an easy problem, per say

flyingmutant commented 1 month ago

Yep, that's definitely something I would like to add/improve. Are you aware of any formalization that can be used as the basis for implementation instead of doing it completely ad-hoc?

The only paper on the subject that I remember right now is this one, but I'd say the method used in it is rather ad-hoc.

danwt commented 1 month ago

I'm not sure to be honest it's been a while since I was looking into the area. I recall some papers on doing a series of flips on operations. There is also the whole field of model checking to draw from. A good search term is 'test case diversity'