search

Bodigrim / tasty-bench

Featherlight benchmark framework, drop-in replacement for criterion and gauge.
https://hackage.haskell.org/package/tasty-bench
MIT License
80 stars 11 forks source link

Estimate standard deviation for memory statistics #38

Closed brprice closed 1 year ago

brprice commented 1 year ago

(Disclaimer: I am not particularly well-versed in benchmarking or statistics, so there may be reasons this is a bad idea.)

Consider the output

All fibonacci numbers fifth: OK (2.13s) 63 ns ± 3.4 ns, 223 B allocated, 0 B copied, 2.0 MB peak memory

Notice that only timing has an estimated (double) standard deviation. Could we add exactly the same analysis to allocation and copying statistics? (I am not sure what variance information would be useful for "peak memory", as we report the maximum, rather than the mean.)

I guess in the majority of programs GHC is deterministic enough for these variations to be very small, so perhaps it is not worth doing?

Bodigrim commented 1 year ago

That's probably possible, but will require an overhaul of console and CSV reporting. Given that normally allocation count is deterministic, I'm a bit reluctant to bother.

Bodigrim commented 1 year ago

Closing, since no one is particularly interested to work on this low-priority feature.