Open dweiller opened 1 year ago
I agree about the hyperfine-style times faster
/times slower
being easier to understand. My suggestion would be to change the header to:
times faster/slower
instead of
ratio
since the 'times faster/slower' part is necessary context for what something like ⚡2.0x
means
faster/slower
only works for the wall-time measurement. I think I'd just go for showing the ratio measurement / reference
, so it will be good/green if it's significantly < 1 and bad/red if it's significantly > 1 and gray when it's not significantly different from 1. My example wasn't great as everything was worse than the reference.
I think that percentages are not easier to comprehend than ratios, especially when the delta is quite big. An example:
Here is the same benchmark run with the worst one first:
I think something like this is much easier to grok:
(I didn't properly convert the numbers on the confidence intervals to a ratio, so they'll be a bit off)
The ratio will be even easier to read (relative to the delta) if you also truncate some of the less significant figures in which case the ratio will need fewer digits than the delta to represent the performance differences (assuming we don't want to use scientific notation for the delta).