andrewrk / poop

Performance Optimizer Observation Platform
MIT License
788 stars 50 forks source link

add 95% confidence interval to delta #18

Closed candrewlee14 closed 1 year ago

candrewlee14 commented 1 year ago

image An important missing feature is a confidence interval on the delta, since insignificant run-to-run changes might wrongly show up as meaningful.

This PR adds that. Notice the cache_misses - 3.3% (which would've been considered significant before) is no longer significant since the standard deviation was high.

Related to #7 and #2

Resources:

andrewrk commented 1 year ago

Thanks!