search

BurntSushi / cargo-benchcmp

A small utility to compare Rust micro-benchmarks.
The Unlicense
342 stars 21 forks source link

use more rigourous statistics #4

Open dgryski opened 8 years ago

dgryski commented 8 years ago

Russ Cox has written https://godoc.org/rsc.io/benchstat which does the same thing but with better math.

BurntSushi commented 8 years ago

Yes, I've seen that and I definitely want to add better math.

mgeisler commented 7 years ago

This might not be necessary. As far as I understand, go test -bench does not do any kind of statistics itself -- it just runs the benchmark code repeated until the running time exceeds 1 sec (by default) and outputs the average: https://golang.org/src/testing/benchmark.go

So this means that the Go world needs extra tools to do statistics on multiple runs. However, that is not the case for cargo bench which comes with some builtin statistics and outlier detection: https://github.com/rust-lang/rust/blob/master/src/libtest/stats.rs#L308

This suggests to me that cargo benchcmp should be careful with comparing values that are already filtered and adjusted by cargo bench -- similar to how an average of an average doesn't give you a proper average of the underlying values. Perhaps cargo bench itself could be improved with more fancy statistic -- I don't know much about that since I'm not a statistician :-)