search

moby / buildkit-bench

Set of benchmarks for BuildKit
https://moby.github.io/buildkit-bench/
Apache License 2.0
0 stars 2 forks source link

Tool to parse `go test` output #5

Closed crazy-max closed 2 months ago

crazy-max commented 2 months ago

In our pipeline we are currently running tests without processing the result output:

+ docker run --rm --privileged -v /tmp --volumes-from=buildkit-bench-cache -e BUILDKIT_REF_RANDOM -e BUILDKIT_RUN_COUNT -e REGISTRY_MIRROR_DIR=/root/.cache/registry buildkit-bench go test -bench . -benchmem -mod=vendor -v ./...
=== RUN   TestBinary
=== RUN   TestBinary/TestDaemonVersion/ref=master/run=1
    binary_test.go:36: repo: github.com/moby/buildkit
    binary_test.go:37: version: 49f3d8f
    binary_test.go:38: commit: 49f3d8f8294d40a3c37a1e2ea3053be7c88f47cb
=== RUN   TestBinary/TestBinarySize/ref=master/run=1
    binary_test.go:45: 64558110
=== RUN   TestBinary/TestDaemonVersion/ref=v0.13/run=1
    binary_test.go:36: repo: github.com/moby/buildkit
    binary_test.go:37: version: v0.13
    binary_test.go:38: commit: 2e18d709fefdcc2db20853ee241c75b058189d39
=== RUN   TestBinary/TestBinarySize/ref=v0.13/run=1
    binary_test.go:45: 63332106
--- PASS: TestBinary (0.26s)
    --- PASS: TestBinary/TestDaemonVersion/ref=master/run=1 (0.13s)
    --- PASS: TestBinary/TestBinarySize/ref=master/run=1 (0.00s)
    --- PASS: TestBinary/TestDaemonVersion/ref=v0.13/run=1 (0.13s)
    --- PASS: TestBinary/TestBinarySize/ref=v0.13/run=1 (0.00s)
goos: linux
goarch: amd64
pkg: github.com/moby/buildkit-bench/test
cpu: AMD Ryzen 9 5950X 16-Core Processor
BenchmarkBinary
BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=1
BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=1-32                    27          39062493 ns/op            2659 B/op         31 allocs/op
BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=2
BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=2-32                    30          35656674 ns/op            2680 B/op         31 allocs/op
BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=3
BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=3-32                    31          38740524 ns/op            2647 B/op         31 allocs/op
BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=1
BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=1-32                     33          35300426 ns/op            2674 B/op         31 allocs/op
BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=2
BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=2-32                     32          35783239 ns/op            2651 B/op         31 allocs/op
BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=3
BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=3-32                     33          34556691 ns/op            2645 B/op         31 allocs/op
PASS
ok      github.com/moby/buildkit-bench/test     9.948s
?       github.com/moby/buildkit-bench/util/testutil    [no test files]
+ docker rm -v buildkit-bench-cache
buildkit-bench-cache

A tool to parse go test output is needed to collect stats. https://pkg.go.dev/golang.org/x/tools@v0.24.0/benchmark/parse and https://cs.opensource.google/go/x/perf/+/783bcb78:benchfmt/reader.go;l=66 (.Result() seem to be what we could use for this case. Or we could just use -json flag to handle that ourselves which seems better:

$ TESTFLAGS="-v -json" BUILDKIT_RUN_COUNT=3 BUILDKIT_REFS=master,v0.13 ./hack/test
...
+ docker run --rm --privileged -v /tmp --volumes-from=buildkit-bench-cache -e BUILDKIT_REF_RANDOM -e BUILDKIT_RUN_COUNT -e REGISTRY_MIRROR_DIR=/root/.cache/registry buildkit-bench go test -bench . -benchmem -mod=vendor -v -json ./...
{"Time":"2024-08-20T13:33:06.682136464Z","Action":"start","Package":"github.com/moby/buildkit-bench/test"}
{"Time":"2024-08-20T13:33:06.684150934Z","Action":"run","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary"}
{"Time":"2024-08-20T13:33:06.684173818Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary","Output":"=== RUN   TestBinary\n"}
{"Time":"2024-08-20T13:33:06.684182374Z","Action":"run","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=master/run=1"}
{"Time":"2024-08-20T13:33:06.68418579Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=master/run=1","Output":"=== RUN   TestBinary/TestDaemonVersion/ref=master/run=1\n"}
{"Time":"2024-08-20T13:33:06.79389955Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=master/run=1","Output":"    binary_test.go:36: repo: github.com/moby/buildkit\n"}
{"Time":"2024-08-20T13:33:06.793943344Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=master/run=1","Output":"    binary_test.go:37: version: 49f3d8f\n"}
{"Time":"2024-08-20T13:33:06.793950297Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=master/run=1","Output":"    binary_test.go:38: commit: 49f3d8f8294d40a3c37a1e2ea3053be7c88f47cb\n"}
{"Time":"2024-08-20T13:33:06.793958052Z","Action":"run","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=master/run=1"}
{"Time":"2024-08-20T13:33:06.793961959Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=master/run=1","Output":"=== RUN   TestBinary/TestBinarySize/ref=master/run=1\n"}
{"Time":"2024-08-20T13:33:06.793965626Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=master/run=1","Output":"    binary_test.go:45: 64558110\n"}   
{"Time":"2024-08-20T13:33:06.794010151Z","Action":"run","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=v0.13/run=1"}
{"Time":"2024-08-20T13:33:06.794028085Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=v0.13/run=1","Output":"=== RUN   TestBinary/TestDaemonVersion/ref=v0.13/run=1\n"}
{"Time":"2024-08-20T13:33:06.90214838Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=v0.13/run=1","Output":"    binary_test.go:36: repo: github.com/moby/buildkit\n"}
{"Time":"2024-08-20T13:33:06.902182465Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=v0.13/run=1","Output":"    binary_test.go:37: version: v0.13\n"}
{"Time":"2024-08-20T13:33:06.902186473Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=v0.13/run=1","Output":"    binary_test.go:38: commit: 2e18d709fefdcc2db20853ee241c75b058189d39\n"}
{"Time":"2024-08-20T13:33:06.902193827Z","Action":"run","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1"}
{"Time":"2024-08-20T13:33:06.902196652Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"=== RUN   TestBinary/TestBinarySize/ref=v0.13/run=1\n"}
{"Time":"2024-08-20T13:33:06.902203134Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"    binary_test.go:45: 63332106\n"}    
{"Time":"2024-08-20T13:33:06.902236498Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary","Output":"--- PASS: TestBinary (0.22s)\n"}
{"Time":"2024-08-20T13:33:06.90224283Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=master/run=1","Output":"    --- PASS: TestBinary/TestDaemonVersion/ref=master/run=1 (0.11s)\n"}
{"Time":"2024-08-20T13:33:06.902245646Z","Action":"pass","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=master/run=1","Elapsed":0.11}
{"Time":"2024-08-20T13:33:06.902302855Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=master/run=1","Output":"    --- PASS: TestBinary/TestBinarySize/ref=master/run=1 (0.00s)\n"}
{"Time":"2024-08-20T13:33:06.902307494Z","Action":"pass","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=master/run=1","Elapsed":0}
{"Time":"2024-08-20T13:33:06.902310179Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=v0.13/run=1","Output":"    --- PASS: TestBinary/TestDaemonVersion/ref=v0.13/run=1 (0.11s)\n"}
{"Time":"2024-08-20T13:33:06.902315429Z","Action":"pass","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestDaemonVersion/ref=v0.13/run=1","Elapsed":0.11}
{"Time":"2024-08-20T13:33:06.902318244Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"    --- PASS: TestBinary/TestBinarySize/ref=v0.13/run=1 (0.00s)\n"}
{"Time":"2024-08-20T13:33:06.90422401Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"goos: linux\n"}
{"Time":"2024-08-20T13:33:06.904248788Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"goarch: amd64\n"}
{"Time":"2024-08-20T13:33:06.904252375Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"pkg: github.com/moby/buildkit-bench/test\n"}
{"Time":"2024-08-20T13:33:06.904254699Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"cpu: AMD Ryzen 9 5950X 16-Core Processor            \n"}
{"Time":"2024-08-20T13:33:06.904258867Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary\n"}
{"Time":"2024-08-20T13:33:06.904798297Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=1\n"}
{"Time":"2024-08-20T13:33:09.103931422Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=1-32         \t      33\t  35090991 ns/op\t    2659 B/op\t      31 allocs/op\n"}
{"Time":"2024-08-20T13:33:09.103972Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=2\n"}
{"Time":"2024-08-20T13:33:11.265319857Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=2-32         \t      33\t  35357671 ns/op\t    2669 B/op\t      31 allocs/op\n"}
{"Time":"2024-08-20T13:33:11.265360936Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=3\n"}
{"Time":"2024-08-20T13:33:12.428929955Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=master/run=3-32         \t      32\t  35181297 ns/op\t    2681 B/op\t      31 allocs/op\n"}
{"Time":"2024-08-20T13:33:12.428968899Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=1\n"}
{"Time":"2024-08-20T13:33:14.549796892Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=1-32          \t      32\t  34712195 ns/op\t    2686 B/op\t      31 allocs/op\n"}
{"Time":"2024-08-20T13:33:14.549836077Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=2\n"}
{"Time":"2024-08-20T13:33:15.636078265Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=2-32          \t      30\t  34860998 ns/op\t    2659 B/op\t      31 allocs/op\n"}
{"Time":"2024-08-20T13:33:15.636120034Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=3\n"}
{"Time":"2024-08-20T13:33:17.805295348Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Output":"BenchmarkBinary/BenchmarkBinaryVersion/ref=v0.13/run=3-32          \t      32\t  35875905 ns/op\t    2651 B/op\t      31 allocs/op\n"}
{"Time":"2024-08-20T13:33:17.805331167Z","Action":"pass","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinarySize/ref=v0.13/run=1","Elapsed":0}
{"Time":"2024-08-20T13:33:17.805336627Z","Action":"pass","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary","Elapsed":0.22}
{"Time":"2024-08-20T13:33:17.805340825Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Output":"PASS\n"}
{"Time":"2024-08-20T13:33:17.806147513Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Output":"ok  \tgithub.com/moby/buildkit-bench/test\t11.124s\n"}
{"Time":"2024-08-20T13:33:17.806175266Z","Action":"pass","Package":"github.com/moby/buildkit-bench/test","Elapsed":11.124}
{"Time":"2024-08-20T13:33:17.807246129Z","Action":"start","Package":"github.com/moby/buildkit-bench/util/testutil"}
{"Time":"2024-08-20T13:33:17.807299631Z","Action":"output","Package":"github.com/moby/buildkit-bench/util/testutil","Output":"?   \tgithub.com/moby/buildkit-bench/util/testutil\t[no test files]\n"}
{"Time":"2024-08-20T13:33:17.807304961Z","Action":"skip","Package":"github.com/moby/buildkit-bench/util/testutil","Elapsed":0}
+ docker rm -v buildkit-bench-cache
buildkit-bench-cache