I think it would be nice to indicate differences inside the control error somehow. Perhaps colour them orange/warning-colour instead of red when a subsequent bench is slower than the control but inside the error.
For example a control
test benchmark_a ... bench: 24,000,000 ns/iter (+/- 500,000)
test benchmark_b ... bench: 24,000,000 ns/iter (+/- 5,000)
...and the latest bench
test benchmark_a ... bench: 24,150,000 ns/iter (+/- 480,000)
test benchmark_b ... bench: 24,150,000 ns/iter (+/- 5,500)
Will produce this output:
name control ns/iter latest ns/iter diff ns/iter diff % speedup
benchmark_a 24,000,000 24,150,000 150,000 0.62% x 0.99 // colour orange
benchmark_b 24,000,000 24,150,000 150,000 0.62% x 0.99 // colour red
benchmark_a will be a warning as the diff is inside the control error. benchmark_b will be red as before as the diff is outside the error.
Perhaps you could also allow an option, similar to --threshold, to ignore warnings/ diffs inside errors.
Thanks for this utility!
I think it would be nice to indicate differences inside the control error somehow. Perhaps colour them orange/warning-colour instead of red when a subsequent bench is slower than the control but inside the error.
For example a control
...and the latest bench
Will produce this output:
benchmark_a will be a warning as the diff is inside the control error. benchmark_b will be red as before as the diff is outside the error.
Perhaps you could also allow an option, similar to
--threshold, to ignore warnings/ diffs inside errors.