Closed austb closed 6 months ago
The sets the bounds to accommodate all the failures seen in 10,000 iterations. To avoid simply increasing the bounds in all cases I track the number of adds and deletes to make the upper bound flexible. In one or two cases I saw ~1000 extra additions than deletes so this avoids hitting a transient failure in those cases and the bounds being needlessly wide for most test runs.
This passed 1,000 iterations locally
$ lein test :only puppetlabs.puppetdb.cli.benchmark-test/rand-catalog-mutation-test
lein test puppetlabs.puppetdb.cli.benchmark-test
Ran 1 tests containing 9990 assertions.
0 failures, 0 errors.
We should be getting a balanced number of additions and deletions by using rand-nth, but if there's only 1 resource remaining deletes become a modification. We start with 4 resources, so if we get a string of 7 deletes followed by a string of 7 additions, we have a balanced number of operations, but we will end with 8 resources. This shifts our expected end number of resources to the right of a simple binomial distribution.