search

melix / jmh-gradle-plugin

Integrates the JMH benchmarking framework with Gradle
Apache License 2.0
662 stars 88 forks source link

Confidence Interval Uncertainty #192

Closed lutacluny closed 3 years ago

lutacluny commented 3 years ago

I am not sure whether the resulting confidence interval really corresponds to 99.9%. Consider the following output for 30 Iterations:

90574.608 ±(99.9%) 9287.329 ops/s [Average] (min, avg, max) = (39404.385, 90574.608, 97347.465), stdev = 13900.838 CI (99.9%): [81287.279, 99861.937] (assumes normal distribution)

Following the Z-table from https://www.statisticshowto.com/tables/z-table/, alpha=0.999 corresponds to z=3.3 which leads to an error margin of 8375.183. That is definitely not 9287.329 as it is given by the output.

Did I miss something here?

melix commented 3 years ago

Thanks, but this is an issue for JMH itself, not the Gradle plugin.