Open bcmills opened 2 years ago
rsc.io is just about the most trivial App Engine server ever. Except for App Engine taking it down for being too old a Go version, it's never had any appreciable downtime. It's possible that App Engine really is hanging up on the connection, but it seems equally possible that the VM is hanging up on the connection. I wonder if we should respond to ECONNRESET by trying one more time?
One more, from a SlowBot on CL 376655: https://storage.googleapis.com/go-build-log/2af499b1/linux-amd64-longtest_dabf7ba2.log
It's possible that App Engine really is hanging up on the connection, but it seems equally possible that the VM is hanging up on the connection. I wonder if we should respond to ECONNRESET by trying one more time?
It's curious that this failure mode happens so much for frequently for rsc.io
than for other services (like gopkg.in
, golang.org/x
, or vcs-test.golang.org
).
Scanning the failures over the last year:
greplogs --dashboard -md -l -e 'https://.*: connection reset by peer(?:.*\n)*FAIL\s+cmd/go' --since=2021-01-01
2021-12-08T15:30:52-9fe77de/linux-386-longtest (rsc.io
)
2021-12-01T10:19:34-0e1d553/linux-386-longtest (rsc.io
)
2021-11-24T20:51:25-5d8c49a/linux-386-longtest (rsc.io
)
2021-11-16T17:13:33-29ec902/linux-386-longtest (rsc.io
)
2021-11-15T19:24:28-9265558/linux-386-longtest (rsc.io
)
2021-09-30T18:10:18-205640e/linux-amd64-longtest (rsc.io
)
2021-09-02T13:45:48-90ed541/linux-386-longtest (github.com
, and gopkg.in
reporting its own error connecting to github.com
, both coinciding with https://www.githubstatus.com/incidents/lfvwm5qydw6r)
2021-02-24T19:25:49-bf48163/linux-386-longtest (rsc.io
)
2021-02-23T21:49:20-0458d8c/linux-amd64-longtest (rsc.io
)
2021-02-13T15:15:13-66c2709/linux-amd64-longtest (rsc.io
)
To me that suggests a problem specific to the App Engine service — perhaps either a bug in the App Engine version of the Go runtime, or flakiness in some network infrastructure sitting it front of the service.
greplogs -l -e 'https://.*: connection reset by peer(?:.*\n)*FAIL\s+cmd/go' --since=2021-12-09
2022-05-05T18:17:08-7b1f8b6/linux-amd64-longtest (rsc.io
, but with a failed git pull
to github.com/rsc/pdfxxx
in another test)
2022-04-26T22:08:00-79db59d/linux-386-longtest (rsc.io
)
2022-03-30T18:02:38-a7e76b8/linux-386-longtest (rsc.io
)
2022-03-28T16:29:15-b10164b/linux-386-longtest (rsc.io
)
2022-02-18T18:46:44-d93cc8c/linux-386-longtest (rsc.io
)
2022-02-09T21:52:21-2e2ef31/linux-386-longtest (rsc.io
)
2022-01-26T17:03:32-0f4bd92/linux-amd64-longtest (swtch.com
)
2022-01-24T17:07:30-f88c3b9/linux-386-longtest (rsc.io
)
greplogs -l -e 'https://.*: connection reset by peer(?:.*\n)*FAIL\s+cmd/go' --since=2022-05-06
2022-08-22T23:36:38-0f42e35/linux-386-longtest
2022-06-28T16:41:48-47e792e/linux-amd64-longtest
2022-05-17T00:58:22-f2b1cde/linux-386-longtest
2022-05-10T21:29:25-d28616d/linux-386-longtest
2022-05-08T17:19:07-4c565a5/linux-amd64-longtest
And a related failure mode:
greplogs -l -e 'https://.*: network is unreachable(?:.*\n)*FAIL\s+cmd/go' --since=2022-05-06
2022-09-07T05:37:53-c011270/linux-386-longtest
2022-09-06T21:53:23-86f8b8d/linux-386-longtest
2022-09-02T17:45:18-274528e/linux-amd64-longtest
Found new matching dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `Get "https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `Get "https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
Found new dashboard test flakes for:
#!watchflakes
post <- pkg ~ `cmd/go` && `(reading |Get ")https://(swtch.com|rsc.io).*(connection reset by peer|network is unreachable|Internal Server Error)`
greplogs --dashboard -md -l -e 'Get "https://rsc\.io/.*read: connection reset by peer' --since=2021-01-01
2021-12-01T10:19:34-0e1d553/linux-386-longtest 2021-11-24T20:51:25-5d8c49a/linux-386-longtest 2021-11-16T17:13:33-29ec902/linux-386-longtest 2021-11-15T19:24:28-9265558/linux-386-longtest 2021-09-30T18:10:18-205640e/linux-amd64-longtest 2021-02-24T19:25:49-bf48163/linux-386-longtest 2021-02-23T21:49:20-0458d8c/linux-amd64-longtest 2021-02-13T15:15:13-66c2709/linux-amd64-longtest