zombocom / derailed_benchmarks

Go faster, off the Rails - Benchmarks for your whole Rails app
2.96k stars 144 forks source link

Check multiple confidence intervals #165

Closed schneems closed 4 years ago

schneems commented 4 years ago

I want to know if a patch stands up to a more rigorous statistical check. Instead of forcing the user to specify what confidence interval they want, we default to 95% then if it passes we try to check again at 99%.

If the 99% check fails then we rely on the 95% check, but note the different confidence interval in the output.