zombocom / derailed_benchmarks

Go faster, off the Rails - Benchmarks for your whole Rails app
2.96k stars 141 forks source link

Don't use statistical significance as a stopping condition #164

Closed schneems closed 4 years ago

schneems commented 4 years ago

The goal of adding the statistical significane check was to determine if it was "safe" to stop early, but this method of constantly checking for significance and then stopping produces less than stellar results. It's better to default to running the full test.

As a compromise the intermediate results are also output so if someone wants to use this as a quick and dirty guide for if they're on the right track or not, they can but they should know it might not hold up to scrutiny.