Option to benchmark until confidence value is reached

bencheeorg / benchee

Easy and extensible benchmarking in Elixir providing you with lots of statistics!

MIT License

1.42k stars 66 forks source link

Option to benchmark until confidence value is reached #9

Open PragTob opened 8 years ago

PragTob commented 8 years ago

I'm not sure how this works, but there are measures in statistics to be confident of your results (standard deviation we have and I think it is one of them) - with this given we could add an option that says benchmark until this confidence level is reached, with some timeout though so it doesn't run forever if results vary too much naturally.

PragTob commented 6 years ago

https://github.com/jlouis/eministat seems to implement a confidence interval in Erlang + even outlier analysis. Potential reuse and inspiration over there - thanks/via @michalmuskala