puppetlabs / puppetdb

Centralized Puppet Storage
http://docs.puppetlabs.com/puppetdb
Apache License 2.0
301 stars 226 forks source link

(maint) benchmark_test: track add/del operations to set max expected resources #3978

Closed austb closed 6 months ago

austb commented 6 months ago

We should be getting a balanced number of additions and deletions by using rand-nth, but if there's only 1 resource remaining deletes become a modification. We start with 4 resources, so if we get a string of 7 deletes followed by a string of 7 additions, we have a balanced number of operations, but we will end with 8 resources. This shifts our expected end number of resources to the right of a simple binomial distribution.

austb commented 6 months ago

The sets the bounds to accommodate all the failures seen in 10,000 iterations. To avoid simply increasing the bounds in all cases I track the number of adds and deletes to make the upper bound flexible. In one or two cases I saw ~1000 extra additions than deletes so this avoids hitting a transient failure in those cases and the bounds being needlessly wide for most test runs.

austb commented 6 months ago

This passed 1,000 iterations locally

$ lein test :only puppetlabs.puppetdb.cli.benchmark-test/rand-catalog-mutation-test

lein test puppetlabs.puppetdb.cli.benchmark-test

Ran 1 tests containing 9990 assertions.
0 failures, 0 errors.