benchstat: New mean represented as 0.00ms instead of 1.46µs

[mewmew]

When the benchmark delta end up changing the unit of the new mean, the value is still presented using the same unit as the old mean. This may be intended behaviour, but if that is not the case I've included a benchmark example which demonstrates this behaviour.

Example:

• old.txt

BenchmarkEncGlobalNameNoReplace  1000000          1094 ns/op
BenchmarkEncGlobalNameReplace       2000       1164383 ns/op
BenchmarkEncLocalNameNoReplace   1000000          1095 ns/op
BenchmarkEncLocalNameReplace        2000       1161360 ns/op
BenchmarkEncGlobalNameNoReplace  1000000          1063 ns/op
BenchmarkEncGlobalNameReplace       1000       1194576 ns/op
BenchmarkEncLocalNameNoReplace   1000000          1067 ns/op
BenchmarkEncLocalNameReplace        2000       1176277 ns/op
BenchmarkEncGlobalNameNoReplace  1000000          1078 ns/op
BenchmarkEncGlobalNameReplace       2000       1146668 ns/op
BenchmarkEncLocalNameNoReplace   1000000          1070 ns/op
BenchmarkEncLocalNameReplace        2000       1146975 ns/op
BenchmarkEncGlobalNameNoReplace  1000000          1096 ns/op
BenchmarkEncGlobalNameReplace       1000       1183679 ns/op
BenchmarkEncLocalNameNoReplace   1000000          1084 ns/op
BenchmarkEncLocalNameReplace        1000       1181105 ns/op
BenchmarkEncGlobalNameNoReplace  1000000          1096 ns/op
BenchmarkEncGlobalNameReplace       1000       1239623 ns/op
BenchmarkEncLocalNameNoReplace   1000000          1135 ns/op
BenchmarkEncLocalNameReplace        1000       1301538 ns/op

• new.txt

BenchmarkEncGlobalNameNoReplace  2000000           753 ns/op
BenchmarkEncGlobalNameReplace    1000000          1456 ns/op
BenchmarkEncLocalNameNoReplace   2000000           741 ns/op
BenchmarkEncLocalNameReplace     1000000          1483 ns/op
BenchmarkEncGlobalNameNoReplace  2000000           726 ns/op
BenchmarkEncGlobalNameReplace    1000000          1454 ns/op
BenchmarkEncLocalNameNoReplace   2000000           712 ns/op
BenchmarkEncLocalNameReplace     1000000          1446 ns/op
BenchmarkEncGlobalNameNoReplace  2000000           743 ns/op
BenchmarkEncGlobalNameReplace    1000000          1471 ns/op
BenchmarkEncLocalNameNoReplace   2000000           711 ns/op
BenchmarkEncLocalNameReplace     1000000          1449 ns/op
BenchmarkEncGlobalNameNoReplace  2000000           723 ns/op
BenchmarkEncGlobalNameReplace    1000000          1479 ns/op
BenchmarkEncLocalNameNoReplace   2000000           706 ns/op
BenchmarkEncLocalNameReplace     1000000          1443 ns/op
BenchmarkEncGlobalNameNoReplace  2000000           735 ns/op
BenchmarkEncGlobalNameReplace    1000000          1446 ns/op
BenchmarkEncLocalNameNoReplace   2000000           714 ns/op
BenchmarkEncLocalNameReplace     1000000          1465 ns/op

Notice the new mean 0.00ms of EncGlobalNameReplace and EncLocalNameReplace. The same value could be represented as 1.46µs.

$ benchstat old.txt new.txt 
name                    old mean              new mean              delta
EncGlobalNameNoReplace  1.09µs × (0.98,1.01)  0.74µs × (0.98,1.02)  -32.19% (p=0.000)
EncGlobalNameReplace    1.19ms × (0.97,1.05)  0.00ms × (0.99,1.01)  -99.88% (p=0.000)
EncLocalNameNoReplace   1.09µs × (0.98,1.04)  0.72µs × (0.98,1.03)  -34.25% (p=0.000)
EncLocalNameReplace     1.19ms × (0.96,1.09)  0.00ms × (0.99,1.02)  -99.88% (p=0.000)

[rsc]

It's intended, for better or worse. The rationale is that this way you can rely on the fact that the first digit in the old column and the first digit in the new column have the same meaning. If there was a unit change or even motion of the decimal point it could be easily missed and you might be comparing numbers that are not actually directly comparable. It does mean that if you make something 1000x faster, the new speed prints as 0.00 in the new column, but that's something I'm willing to live with. :-)

In this case the thing to do is probably to follow that command by a plain 'benchstat new.txt' to show the new data (alone) in more natural units.

[mewmew]

I agree with the rationale as the trade-off seems worth it. If the decimal point moved or the unit changed, the output might be misread. And the drawback may be mitigated, as you already mentioned, by invoking benchstat FILE in the rare cases when the mean is truncated to 0.00.

Thanks for a great tool Russ! For the longest of time, my friend Daniel and I were convinced that you were an AI cluster at Google rather than a single human being. This was the only reasonable explanation we could conjure which would account for all the work produced under the alias rsc. Our favorite moment was when Brad Fitzpatrick proposed read-only slices on golang-dev, which received initial discussions regarding syntax, intermediate discussions regarding potential benefit (merging the bytes and strings packages into a single package, ...), and final discussions regarding your forked implementation of Go which implemented the proposal to thoroughly review its implications. Whomever you are, we admire your efforts rsc.