Upgrade benchviz to understand benchstat output.

[randall77]

benchstat is an upgraded benchcmp which can handle multiple benchmark runs and performs statistics on those runs.

https://godoc.org/rsc.io/benchstat

[ajstarks]

this change breaks benchcmp compatibility and does not handle bare ~. working on a fix.

[mamyn0va]

this change breaks benchcmp compatibility and does not handle bare ~. working on a fix.

Is there any update on benchviz update? It would be great!

Thanks

[ajstarks]

merged.

[mamyn0va]

Hi, thanks a lot for the update!

It seems that I have not the same results with benchstat as with benchcmp for the same given bench output.

bench.1

go test -bench=. -count=1 -run=^\$ component/... | grep "^Benchmark" > bench.1

BenchmarkComponent_Func1-8              3804            345522 ns/op         517.44 MB/s
BenchmarkComponent_Func2-8               3750            302372 ns/op         657.30 MB/s
BenchmarkComponent_Func3-8               2232            593357 ns/op        1508.42 MB/s

bench.2

go test -bench=. -count=1 -run=^\$ component/... | grep "^Benchmark" > bench.2

BenchmarkComponent_Func1-8              2427            424328 ns/op         268.82 MB/s
BenchmarkComponent_Func2-8               3956            384277 ns/op         545.62 MB/s
BenchmarkComponent_Func3-8               1742            669751 ns/op        1042.99 MB/s

benchcmp

benchcmp bench.1 bench.2 > benchcmp.out

benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
benchmark                                  old ns/op     new ns/op     delta
BenchmarkComponent_Func1-8     345522        424328        +22.81%
BenchmarkComponent_Func2-8                      302372        384277        +27.09%
BenchmarkComponent_Func3-8                      593357        669751        +12.87%

benchmark                                  old MB/s     new MB/s     speedup
BenchmarkComponent_Func1-8     517.44       268.82       0.52x
BenchmarkComponent_Func2-8                      657.30       545.62       0.83x
BenchmarkComponent_Func3-8                      1508.42      1042.99      0.69x

benchstat

benchstat bench.1 bench.2 > benchstat.out

name                           old time/op    new time/op    delta
BenchmarkComponent_Func1-8     346µs ± 0%     424µs ± 0%   ~     (p=1.000 n=1+1)
BenchmarkComponent_Func2-8                      302µs ± 0%     384µs ± 0%   ~     (p=1.000 n=1+1)
BenchmarkComponent_Func3-8                      593µs ± 0%     670µs ± 0%   ~     (p=1.000 n=1+1)

name                           old speed      new speed      delta
BenchmarkComponent_Func1-8   517MB/s ± 0%   269MB/s ± 0%   ~     (p=1.000 n=1+1)
BenchmarkComponent_Func2-8                    657MB/s ± 0%   546MB/s ± 0%   ~     (p=1.000 n=1+1)
BenchmarkComponent_Func3-8                   1.51GB/s ± 0%  1.04GB/s ± 0%   ~     (p=1.000 n=1+1)

benchviz from benchcmp

cat benchcmp.out | benchviz > benchviz_cmp.svg

<?xml version="1.0"?>
<!-- Generated by SVGo -->
<svg width="1024" height="351"
     xmlns="http://www.w3.org/2000/svg"
     xmlns:xlink="http://www.w3.org/1999/xlink">
<rect x="0" y="0" width="1024" height="351" style="fill:white;stroke-width:2px;stroke:lightgray" />
<g style="font-size:20px;font-family:sans-serif">
<text x="100" y="50" style="font-size:150%" ></text>
<g style="font-style:italic;font-size:75%">
<rect x="544" y="118" width="68" height="20" style="fill-opacity:0.3;fill:red" />
<text x="539" y="133" style="text-anchor:end" >+22.81%</text>
</g>
<text x="100" y="138" style="text-anchor:start" >BenchmarkComponent_Func1-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="531" y="144" width="81" height="20" style="fill-opacity:0.3;fill:red" />
<text x="526" y="159" style="text-anchor:end" >+27.09%</text>
</g>
<text x="100" y="164" style="text-anchor:start" >BenchmarkComponent_Func2-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="574" y="170" width="38" height="20" style="fill-opacity:0.3;fill:red" />
<text x="569" y="185" style="text-anchor:end" >+12.87%</text>
</g>
<text x="100" y="190" style="text-anchor:start" >BenchmarkComponent_Func3-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="248" width="386" height="20" style="fill-opacity:0.3;fill:green" />
<text x="1003" y="263" style="text-anchor:start" >0.52x</text>
</g>
<text x="100" y="268" style="text-anchor:start" >BenchmarkComponent_Func1-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="274" width="386" height="20" style="fill-opacity:0.3;fill:green" />
<text x="1003" y="289" style="text-anchor:start" >0.83x</text>
</g>
<text x="100" y="294" style="text-anchor:start" >BenchmarkComponent_Func2-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="300" width="386" height="20" style="fill-opacity:0.3;fill:green" />
<text x="1003" y="315" style="text-anchor:start" >0.69x</text>
</g>
<text x="100" y="320" style="text-anchor:start" >BenchmarkComponent_Func3-8</text>
</g>
</svg>

benchviz from benchstat

cat benchstat.out | benchviz > benchviz_stat.svg

<?xml version="1.0"?>
<!-- Generated by SVGo -->
<svg width="1024" height="351"
     xmlns="http://www.w3.org/2000/svg"
     xmlns:xlink="http://www.w3.org/1999/xlink">
<rect x="0" y="0" width="1024" height="351" style="fill:white;stroke-width:2px;stroke:lightgray" />
<g style="font-size:20px;font-family:sans-serif">
<text x="100" y="50" style="font-size:150%" ></text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="118" width="0" height="20" style="fill-opacity:0.3;fill:green" />
<text x="617" y="133" style="text-anchor:start" >~</text>
</g>
<text x="100" y="138" style="text-anchor:start" >BenchmarkComponent_Func1-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="144" width="0" height="20" style="fill-opacity:0.3;fill:green" />
<text x="617" y="159" style="text-anchor:start" >~</text>
</g>
<text x="100" y="164" style="text-anchor:start" >BenchmarkComponent_Func2-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="170" width="0" height="20" style="fill-opacity:0.3;fill:green" />
<text x="617" y="185" style="text-anchor:start" >~</text>
</g>
<text x="100" y="190" style="text-anchor:start" >BenchmarkComponent_Func3-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="248" width="0" height="20" style="fill-opacity:0.3;fill:green" />
<text x="617" y="263" style="text-anchor:start" >~</text>
</g>
<text x="100" y="268" style="text-anchor:start" >BenchmarkComponent_Func1-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="274" width="0" height="20" style="fill-opacity:0.3;fill:green" />
<text x="617" y="289" style="text-anchor:start" >~</text>
</g>
<text x="100" y="294" style="text-anchor:start" >BenchmarkComponent_Func2-8</text>
<g style="font-style:italic;font-size:75%">
<rect x="612" y="300" width="0" height="20" style="fill-opacity:0.3;fill:green" />
<text x="617" y="315" style="text-anchor:start" >~</text>
</g>
<text x="100" y="320" style="text-anchor:start" >BenchmarkComponent_Func3-8</text>
</g>
</svg>

[ajstarks]

I appears to me that the data you posted does not show a delta, and therefore benchviz has nothing to plot.

[mamyn0va]

Thanks, I misused benchstat. I need to run my bench a lot more times (x10) to get significant p-results.