search

moby / buildkit-bench

Set of benchmarks for BuildKit
https://moby.github.io/buildkit-bench/
Apache License 2.0
0 stars 2 forks source link

Unexpected go test output #39

Closed crazy-max closed 2 weeks ago

crazy-max commented 2 weeks ago

It seems go test output for benchmark is randomly malformed: https://github.com/moby/buildkit-bench/actions/runs/10682680331/job/29609222191#step:7:132

goos: linux
goarch: amd64
pkg: github.com/moby/buildkit-bench/test
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkBinary
BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-21/run=1
BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-21/run=1-4          
1000000000           0.05120 ns/op    64558108 bytes           0 B/op          0 allocs/op
BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-22/run=1
BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-22/run=1-4          1000000000           0.05157 ns/op    64567205 bytes           0 B/op          0 allocs/op
BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-23/run=1
BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-23/run=1-4          1000000000           0.05129 ns/op    64572191 bytes           0 B/op          0 allocs/op

There is an extra new line for BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-21/run=1-4 and therefore result is not set for this run:

gotestmetrics: error: BenchmarkBinary/BenchmarkBinarySize missing bytes value for ref 2024-08-21

Not yet sure how to repro consistently.

crazy-max commented 2 weeks ago

Was able to debug this issue with https://github.com/moby/buildkit-bench/pull/40 and it seems to be an issue with go test: https://github.com/moby/buildkit-bench/actions/runs/10685110494/job/29617285283#step:7:334

{"Time":"2024-09-03T14:40:53.713670716Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-27/run=1\n"}
{"Time":"2024-09-03T14:40:53.715533088Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-27/run=1-4         \t1000000000\t         0.0000220 ns/op\t  64572183 bytes\t       0 B/op\t       0 allocs/op\n"}
{"Time":"2024-09-03T14:40:53.715590676Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-28/run=1\n"}
{"Time":"2024-09-03T14:40:53.717329904Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-28/run=1-4         \t1000000000\t         0.0000142 ns/op\t  64567941 bytes\t       0 B/op\t       0 allocs/op\n"}
{"Time":"2024-09-03T14:40:53.717415164Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-30/run=1\n"}
{"Time":"2024-09-03T14:40:53.719272012Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=2024-08-30/run=1-4         \t"}
{"Time":"2024-09-03T14:40:53.719285247Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"1000000000\t         0.0000200 ns/op\t  64572325 bytes\t       0 B/op\t       0 allocs/op\n"}
{"Time":"2024-09-03T14:40:53.719361062Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=master/run=1\n"}
{"Time":"2024-09-03T14:40:53.721342422Z","Action":"output","Package":"github.com/moby/buildkit-bench/test","Test":"TestBinary/TestBinaryVersion/ref=v0.15.2/run=1","Output":"BenchmarkBinary/BenchmarkBinarySize/ref=master/run=1-4             \t1000000000\t         0.0000208 ns/op\t  64572702 bytes\t       0 B/op\t       0 allocs/op\n"}