prattmic
commented
2 years ago cc @mknyszek
Open bcmills opened 2 years ago
prattmic
commented
2 years ago cc @mknyszek
prattmic
commented
2 years ago The secondary issue here is that we are apparently not logging the stderr output from the go command, so the actual error is missing.
prattmic
commented
2 years ago [sweet] Work directory: /tmp/gosweet2197573647
[sweet] Benchmarks: gvisor
[sweet] Setting up benchmark: gvisor
[shell] git clone -b go https://github.com/google/gvisor /tmp/gosweet2197573647/gvisor/src
[shell] git -C /tmp/gosweet2197573647/gvisor/src checkout 6392b0f3bea052af0de9d95677233dd9e442dbd5
[shell] mkdir -p /usr/local/google/home/mpratt/src/benchmarks/sweet/results/gvisor
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/goroot/bin
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/src
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/goroot/tmp
[shell] mkdir -p /tmp/gosweet2197573647/gvisor/goroot/assets
[shell] cd /usr/local/google/home/mpratt/src/benchmarks/sweet/benchmarks/gvisor
[shell] GOROOT=/usr/local/google/home/mpratt/src/go/ /usr/local/google/home/mpratt/src/go/bin/go build -o /tmp/gosweet2197573647/gvisor/goroot/bin/gvisor-bench
[shell] cd /usr/local/google/home/mpratt/src/benchmarks/sweet
[shell] cd /tmp/gosweet2197573647/gvisor/src/runsc
[shell] CGO_ENABLED=0 GOROOT=/usr/local/google/home/mpratt/src/go/ /usr/local/google/home/mpratt/src/go/bin/go build -o /tmp/gosweet2197573647/gvisor/goroot/bin/runsc
package gvisor.dev/gvisor/runsc
imports gvisor.dev/gvisor/runsc/cli
imports gvisor.dev/gvisor/pkg/coverage
imports gvisor.dev/gvisor/pkg/hostarch
imports gvisor.dev/gvisor/pkg/gohacks: build constraints exclude all Go files in /tmp/gosweet2197573647/gvisor/src/pkg/gohacks
package gvisor.dev/gvisor/runsc
imports gvisor.dev/gvisor/runsc/cli
imports gvisor.dev/gvisor/runsc/cmd
imports gvisor.dev/gvisor/pkg/sentry/control
imports gvisor.dev/gvisor/pkg/sentry/fdimport
imports gvisor.dev/gvisor/pkg/sentry/fs/host
imports gvisor.dev/gvisor/pkg/sentry/kernel
imports gvisor.dev/gvisor/pkg/goid: build constraints exclude all Go files in /tmp/gosweet2197573647/gvisor/src/pkg/goid
[shell] cd /usr/local/google/home/mpratt/src/benchmarks/sweet
[sweet] error: build gvisor for goroot: exit status 1
[sweet] error: failed to execute benchmarks, see log for detailsThis is failing due to https://go.dev/cl/388376 [1]. gVisor has negative go1.19 build tags due to a limitation in their static analysis tool. We'll need to disable this benchmark until gVisor is updated.
[1] I don't understand why this CL wasn't specifically tested on https://build.golang.org/?repo=golang.org%2fx%2fbenchmarks.
gopherbot
commented
2 years ago Change https://go.dev/cl/389754 mentions this issue: sweet: disable gvisor benchmark
gopherbot
commented
2 years ago Change https://go.dev/cl/389774 mentions this issue: sweet/common: include stderr in error string
prattmic
commented
2 years ago https://go.dev/cl/389754 fixes the builders by disabling the gvisor benchmark. Keeping this open to track reenabling the benchmarks once google/gvisor#7252 goes in.
gopherbot
commented
2 years ago Change https://go.dev/cl/390954 mentions this issue: sweet/harnesses: update gVisor for 1.19 compatibility
gopherbot
commented
2 years ago Change https://go.dev/cl/390955 mentions this issue: Revert "sweet: disable gvisor benchmark"
gopherbot
commented
2 years ago Change https://go.dev/cl/391194 mentions this issue: sweet: disable gvisor benchmark
bcmills
commented
2 years ago Unfortunately still failing at head (https://build.golang.org/log/65cac28815309e7fe290362018f8929ca2204c58):
--- FAIL: TestSweetEndToEnd (222.47s)
integration_test.go:136: output for gvisor:
integration_test.go:144: === Benchmark stdout+stderr ===
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
error: exit status 137
integration_test.go:150: command output:
[sweet] Work directory: /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5
[sweet] Benchmarks: gvisor
[sweet] Setting up benchmark: gvisor
[shell] git clone -b go https://github.com/google/gvisor /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src
[shell] git -C /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src checkout adc7bb5e1baf4a7489e428e1fad756e5e2aa3410
[shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/results-5/gvisor
[shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin
[shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src
[shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/tmp
[shell] mkdir -p /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/assets
[shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/benchmarks/gvisor
[shell] /workdir/go/bin/go build -o /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/gvisor-bench
[shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/cmd/sweet
[shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/src/runsc
[shell] CGO_ENABLED=0 /workdir/go/bin/go build -o /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/runsc
[shell] cd /workdir/gopath/src/golang.org/x/benchmarks/sweet/cmd/sweet
[shell] chmod 755 /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/runsc
[shell] unzip /tmp/go-sweet-assets/assets-v0.3.0.zip 'gvisor/*' -d /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/assets
[sweet] Running benchmark gvisor for go: run 1
[shell] /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/gvisor-bench -runsc /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/bin/runsc -assets-dir /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/assets -tmp /workdir/gopath/src/golang.org/x/benchmarks/sweet/tmp/tmp-5/gvisor/go/tmp -short
[sweet] error: run benchmark gvisor for config go: exit status 1
[sweet] error: failed to execute benchmarks, see log for details
integration_test.go:151: exit status 1
FAIL
FAIL golang.org/x/benchmarks/sweet/cmd/sweet 222.488s
mknyszek
commented
2 years ago This test only runs on the longtest builders, so generally modifications to Sweet should check longtest as a slowbot.
I have been considering making the test run on the usual trybots.
mknyszek
commented
2 years ago To be clear, I don't think anyone did anything wrong here (except maybe me, for marking this test as "long" đŸ˜… ). I do think this is pretty good evidence to suggest this test should probably just run on the regular trybots. We have plenty of timeout headroom, though it will add a few minutes of latency to the trybots.
bcmills
commented
2 years ago This test only runs on the longtest builders, so generally modifications to Sweet should check longtest as a slowbot.
It may be possible to modify x/build/cmd/coordinator to do that; compare #42661, #43719.
aclements
commented
2 years ago What's the status on this? The x/benchmarks repo isn't failing at this point. Is the gvisor benchmark still disabled? If so, we should retitle this to make it about re-enabling this benchmark.
mknyszek
commented
2 years ago The current status is that after the previous fix (https://go.dev/cl/390954) there were still failures, so the benchmark was disabled again. This needs to be investigated again, I believe.
bcmills
commented
2 years ago I had labeled this as a release-blocker because the builders were failing, but now the benchmark is disabled and the builder is passing, so it won't mask any other regressions and from my perspective isn't as urgent.
Is it important to re-enable this benchmark for the Go 1.19 release, or should we remove the release-blocker label?
mknyszek
commented
2 years ago I don't think there's any particular reason fixing this should be tied to the release cycle. I think putting it in Unreleased makes sense.
prattmic
commented
2 years ago I've tested the latest gVisor. It is currently blocked on https://github.com/google/gvisor/issues/7930, which causes us to see spurious SIGKILL exit status.
gopherbot
commented
2 years ago Change https://go.dev/cl/444758 mentions this issue: sweet: update and re-enable gvisor benchmark
prattmic
commented
2 years ago Now we are blocked on https://github.com/google/gvisor/issues/8111. :(
TestSweetEndToEndis persistently failing on thelinux-amd64-longtestandlinux-amd64-perfbuilders since CL 386698 (attn @dr2chase, @prattmic, @aclements).greplogs --dashboard -md -l -e 'error: build gvisor for .*: exit status 1'