search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
121.62k stars 17.41k forks source link

x/build: instances of all internet-facing tests failing #54279

Open dmitshur opened 1 year ago

dmitshur commented 1 year ago 
#!watchflakes
post <- repo == "build" && `https:\/\/maintner\.golang\.org\/.+: 502 Bad Gateway`

Issue created to collect instances where internet-facing tests fail with errors:

[...]
2022/07/28 00:33:59 syncing segment 136 did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs/136: 502 Bad Gateway
2022/07/28 00:34:08 Downloading 15228181 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:17 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs/136: 502 Bad Gateway"}
2022/07/28 00:34:17 syncing segment 136 did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs/136: 502 Bad Gateway
2022/07/28 00:34:33 Downloading 15228181 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:34 Unknown error type *errors.errorString: &errors.errorString{s:"corrupt download"}
2022/07/28 00:34:34 Corpus GetMutations: corrupt download
--- FAIL: TestCorpusCheck (112.99s)
    godata_test.go:52: getting corpus: corrupt download
2022/07/28 00:34:34 Loading data from log *maintner.netMutSource ...
2022/07/28 00:34:34 Downloading 15186069 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:35 wrote /root/.cache/golang-maintner/0136.growing.mutlog
2022/07/28 00:35:16 Reloaded data from log *maintner.netMutSource.
FAIL
FAIL    golang.org/x/build/maintner/godata  156.319s
[...]
--- FAIL: TestFindWork (9.05s)
    coordinator_test.go:309: bad http response: 500 Internal Server Error
2022/07/28 00:33:04 gRPCServer.ClearResults: bad status from dashboard: 400 ("400 Bad Request")
2022/07/28 00:33:04 gRPCServer.ClearResults: bad status from dashboard: 308 ("308 Permanent Redirect")
FAIL
FAIL    golang.org/x/build/cmd/coordinator  9.164s
[...]
dmitshur commented 1 month ago

The likely explanation here is that the k8s cluster was undergoing an upgrade, resulting in brief windows of service unavailability.

Let's reuse this issue to catch future instances, so that watchflakes doesn't keep opening new issues per package/test.

gopherbot commented 1 month ago

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "build" && `https:\/\/maintner\.golang\.org\/.+: 502 Bad Gateway`
2024-06-17 17:07 x_build-go1.21-windows-amd64-longtest build@def93586 release-branch.go1.21@74ac37e9 x/build/maintner/godata.TestCorpusCheck (log) === RUN TestCorpusCheck 2024/06/17 10:16:19 Loading data from log *maintner.netMutSource ... 2024/06/17 10:16:28 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:28 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:16:38 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:38 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:16:51 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:51 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:09 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:09 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:34 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:34 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway --- FAIL: TestCorpusCheck (75.14s)
2024-06-17 17:07 x_build-go1.21-windows-amd64-longtest build@def93586 release-branch.go1.21@74ac37e9 x/build/maintner/maintnerd/maintapi.TestGetRef (log) === RUN TestGetRef 2024/06/17 10:16:20 Loading data from log *maintner.netMutSource ... 2024/06/17 10:16:29 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:29 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:16:39 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:39 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:52 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:10 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:10 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:35 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:35 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway api_test.go:475: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway --- FAIL: TestGetRef (75.12s)
2024-06-17 17:07 x_build-go1.22-darwin-amd64-longtest build@def93586 release-branch.go1.22@fe9b3c33 x/build/maintner/godata.TestCorpusCheck (log) === RUN TestCorpusCheck 2024/06/17 10:16:26 Loading data from log *maintner.netMutSource ... 2024/06/17 10:16:35 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:35 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:16:45 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:45 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:16:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:58 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:16 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:16 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:41 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:41 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway --- FAIL: TestCorpusCheck (75.37s)
2024-06-17 17:07 x_build-gotip-windows-amd64-longtest build@def93586 go@8f5c6904 x/build/maintner/godata.TestGerritForeachNonChangeRef (log) === RUN TestGerritForeachNonChangeRef 2024/06/17 10:16:43 Loading data from log *maintner.netMutSource ... 2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:52 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:02 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:02 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:15 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:15 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:33 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:33 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:58 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway --- FAIL: TestGerritForeachNonChangeRef (75.02s)
2024-06-17 17:07 x_build-gotip-windows-amd64-longtest build@def93586 go@8f5c6904 x/build/maintner/maintnerd/maintapi.TestTryBotStatus (log) === RUN TestTryBotStatus 2024/06/17 10:16:43 Loading data from log *maintner.netMutSource ... 2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:16:52 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:02 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:02 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:15 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:15 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:33 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:33 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway 2024/06/17 10:17:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"} 2024/06/17 10:17:58 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway api_test.go:475: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway --- FAIL: TestTryBotStatus (75.04s)

— watchflakes