Running go benchmarks: Use `-count` or not?

[ankon]

We're setting up this action for our performance-critical go applications, and notice some variance in the results. In one situation it got so bad (or good) that PR branches now report performance regressions because the master branch by chance recorded a really good value.

I was wondering whether we can "smooth" that a bit by using -count when running the benchmarks, but am not sure how/if this action will handle that? If it "should just work", do you think it's a good idea?

[ktrz]

Hi @ankon

I'm not sure what do you mean by "smooth" in this context. But as long as the output of the benchmark run is in the same format it should be fine.

[ankon]

In the case of running using -count=10 we get this output: Basically one line identifying the benchmark, and then it repeats once for each requested count. The question is whether the action will parse this format, and understand to aggregate the data in some way (average it?)

goos: linux
goarch: amd64
pkg: github.com/.../tmp
cpu: AMD Ryzen 9 7940HS w/ Radeon 780M Graphics     
BenchmarkBranchless
BenchmarkBranchless/toUpper1
BenchmarkBranchless/toUpper1-16             88331236           13.18 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             99448578           12.76 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             96488515           12.92 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             84861313           12.38 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             82347253           12.23 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             92781146           13.04 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             97147456           12.94 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             93339778           12.69 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             85405495           13.66 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper1-16             83599527           12.25 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2
BenchmarkBranchless/toUpper2-16             51006874           21.01 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             51493095           20.49 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             52621446           20.44 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             51095416           20.32 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             51120625           20.61 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             53014428           20.54 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             52577970           20.73 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             53823154           20.41 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             50077378           20.84 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper2-16             51951420           20.87 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3
BenchmarkBranchless/toUpper3-16             73775959           15.37 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             77838273           15.27 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             69781880           15.46 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             65597184           15.25 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             68476039           15.30 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             71289954           15.36 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             69229252           15.00 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             80217844           15.47 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             70269200           15.31 ns/op        0 B/op          0 allocs/op
BenchmarkBranchless/toUpper3-16             71143648           15.36 ns/op        0 B/op          0 allocs/op
PASS

[ktrz]

I don't think the action will support that output right now. I'm not sure whether it would fail or only use the last value for each benchmark name. If you have some time to submit a fix for that then I would greatly appreciate it. Otherwise, I will try to pick it up when I manage to get some time.

Just to clarify what would be the expected result here. I suppose the solution should combine all the metrics using a weighted average (based on number of iterations per run)