Open bcmills opened 4 years ago
Whatever the source of memory corruption here is, it may also account for some or all of the crashes observed in #35541. Several of those include an exprInternal
stack frame:
2020-02-26T17:58:41-a1103dc/linux-mips64le-mengzhuo 2020-02-21T22:00:03-240498d/linux-mips64le-mengzhuo 2019-12-20T23:07:52-4d5bb9c/linux-mips64le-mengzhuo
CC @mengzhuo
I think this is unrelated to #36605, because, as you mentioned, the underlying issue for that was fixed in 1.14. But the go/types code is complex, so I wouldn't be surprised at another race lurking there.
2020-04-24T21:12:21-943a0d0/linux-mips64le-mengzhuo 2020-04-17T22:12:02-4d9ecde/openbsd-arm-jsing 2020-04-14T19:01:47-334d410/linux-mips64le-mengzhuo 2020-04-08T20:57:58-b2790a2/linux-mips64le-mengzhuo 2020-04-07T18:19:55-7ee8467/linux-mips64le-mengzhuo 2020-04-07T17:18:30-64f19d7/linux-mips64le-mengzhuo 2020-04-02T22:10:27-2bed279/linux-mips64le-mengzhuo 2020-03-25T04:06:34-6b6414c/linux-mips64le-mengzhuo 2020-03-24T22:21:50-9f343b1/linux-mips64le-mengzhuo 2020-03-19T16:17:59-f9f57c4/linux-mips64le-mengzhuo 2020-03-04T19:06:44-68fea52/linux-mips64le-mengzhuo 2020-03-04T18:09:01-588ee79/linux-mips64le-mengzhuo 2020-03-03T11:36:47-bd6f4cd/linux-mips64le-mengzhuo 2020-03-01T21:04:30-a4f7b08/darwin-amd64-nocgo
2020-06-10T21:39:03-6aadfcd/netbsd-amd64-9_0 2020-05-30T03:20:54-1ed57c1/openbsd-arm-jsing 2020-05-13T12:28:04-866d71a/openbsd-386-64 2020-05-07T19:23:25-6441d34/openbsd-386-64 2020-04-27T20:59:12-352a540/openbsd-386-64 2020-04-27T16:39:59-9ea0146/openbsd-386-64 2020-04-27T15:30:19-a90b730/openbsd-386-64 2020-04-27T14:36:10-bed5418/openbsd-386-64
2021-01-12T21:39:59-ba76567/darwin-amd64-10_14 2020-12-03T19:32:01-9bc6a97/openbsd-386-64 2020-12-01T06:44:07-fd09bd9/openbsd-386-64 2020-12-01T01:28:06-bf23c54/openbsd-386-64 2020-11-25T22:00:14-2e53cd6/openbsd-386-64 2020-11-24T12:36:21-a966329/openbsd-386-64 2020-11-23T21:59:33-b5535b1/openbsd-386-64 2020-11-19T13:27:11-4783bc9/openbsd-386-64 2020-11-18T17:45:08-6ed8ff9/openbsd-386-64 2020-11-17T22:57:35-f544f6c/openbsd-386-64 2020-11-17T15:25:13-9036a0f/openbsd-386-64 2020-11-16T00:27:33-ac45abd/openbsd-386-64 2020-11-13T20:20:37-1643af1/openbsd-386-64 2020-11-12T18:51:08-eeaa07d/openbsd-386-64 2020-10-30T14:32:52-cf7a54d/openbsd-386-64 2020-10-28T02:59:01-8cd080b/openbsd-386-64 2020-10-27T23:31:11-0b86805/openbsd-386-64 2020-10-21T21:49:18-23787c0/darwin-amd64-nocgo 2020-10-21T00:02:07-d49c4ed/openbsd-386-64 2020-10-17T00:14:24-6003fad/darwin-amd64-10_15
This one has a SIGSEGV
in the collector, but exprInternal
on one of the other call stacks:
https://build.golang.org/log/dc8b837f7f5de247bcd9f07144f6d18bec4fe2db
Funky. Only seems to persist on openbsd-386-64
now.
2021-03-16T19:06:39-9e9211a/openbsd-386-64 2021-03-11T22:40:02-44abc2a/openbsd-386-64 2021-03-10T19:04:36-11e8f6b/openbsd-386-64 2021-03-03T21:54:20-376db57/openbsd-386-64 2021-03-03T20:04:07-dafbee5/openbsd-386-64 2021-03-02T22:01:38-2ac05c8/openbsd-386-64 2021-03-02T17:33:32-16b2c87/openbsd-386-64 2021-03-02T01:39:42-f9c628b/openbsd-386-64 2021-02-27T20:30:37-f5a4005/openbsd-386-64 2021-02-25T15:03:53-54dc8c5/openbsd-386-64 2021-02-25T05:05:20-24aca17/openbsd-386-64 2021-02-23T16:31:27-f48e60b/openbsd-386-64 2021-02-19T19:26:32-bdaa8bf/openbsd-386-64 2021-02-18T20:23:11-4b19790/openbsd-386-64 2021-02-18T03:45:40-67e49ef/openbsd-386-64 2021-02-17T20:31:47-4534fc3/openbsd-386-64 2021-02-17T17:17:01-35839b7/openbsd-386-64 2021-02-12T16:44:11-5bd3da9/openbsd-386-64 2021-02-11T05:13:29-706a59c/openbsd-386-64 2021-02-11T00:57:35-8316e56/openbsd-386-64 2021-02-04T18:06:13-842a928/openbsd-386-64 2021-02-02T23:37:55-e7dfe02/openbsd-386-64 2021-02-02T11:32:59-c3402e3/openbsd-386-64 2021-02-01T17:10:16-f1f686b/openbsd-386-64 2021-02-01T17:07:00-d8a2a07/openbsd-386-64 2021-01-26T13:52:46-c2bea79/openbsd-386-64 2021-01-25T18:56:18-68bf78a/openbsd-386-64 2021-01-22T19:37:57-e13398c/openbsd-386-64 2021-01-22T17:08:14-cf1022a/openbsd-386-64 2021-01-21T18:28:06-bec622c/openbsd-386-64 2021-01-19T22:25:03-fe37c9e/openbsd-386-64 2021-01-15T20:22:50-e0d2015/openbsd-386-64 2021-01-14T06:55:38-d78b04b/openbsd-386-64 2021-01-13T18:03:00-f964368/openbsd-386-64
OTOH, the darwin
failures were always pretty infrequent. Maybe those builders have just been luckier since January.
greplogs --dashboard -md -l -e 'SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)' --since=2022-02-01
2022-02-15T22:30:13-11109f6-6e82ff8/linux-ppc64-buildlet
greplogs --dashboard -md -l -e 'SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)' --since=2021-03-17 --omit=\(\?:openbsd-\|netbsd-\).\*
2021-10-25T21:03:33-5a40697-60c3069/freebsd-amd64-race 2021-09-09T21:49:46-295a39b-c981874/linux-riscv64-unmatched 2021-08-31T08:41:16-3e0d083-f118d14/linux-arm-arm5spacemonkey 2021-08-31T08:41:16-3e0d083-f118d14/linux-arm64-packet 2021-08-31T04:26:31-3e0d083-d384ebd/linux-arm-arm5spacemonkey 2021-08-30T22:09:22-3e0d083-7622e41/linux-arm-arm5spacemonkey 2021-08-30T20:08:45-3e0d083-61120c6/linux-arm-arm5spacemonkey 2021-08-30T17:13:00-3e0d083-b602dae/linux-arm-arm5spacemonkey 2021-08-30T02:40:46-3e0d083-56c3856/linux-arm-arm5spacemonkey 2021-08-30T01:55:58-3e0d083-a29d9aa/linux-arm-arm5spacemonkey 2021-08-25T03:13:06-c32dd46-e1fcf88/linux-arm-arm5spacemonkey 2021-08-25T03:07:54-c32dd46-41b99da/linux-arm-arm5spacemonkey 2021-08-24T22:23:12-c32dd46-54cdef1/linux-arm-arm5spacemonkey 2021-08-24T18:31:34-c32dd46-c2f96e6/linux-arm-arm5spacemonkey 2021-08-24T14:51:57-c32dd46-8eeb1bf/linux-arm-arm5spacemonkey 2021-08-24T00:01:29-d4cc65f-8eeb1bf/linux-arm-arm5spacemonkey 2021-08-23T21:22:58-d4cc65f-8157960/linux-arm-arm5spacemonkey 2021-08-23T19:27:46-d4cc65f-3081f81/linux-arm-arm5spacemonkey 2021-08-23T03:39:14-d4cc65f-457418b/linux-arm-arm5spacemonkey 2021-08-22T23:49:55-d4cc65f-86ee892/linux-arm-arm5spacemonkey 2021-05-05T22:35:07-dd255f2-ba7cac4/linux-arm-scaleway 2021-04-30T16:06:43-9b9633e-72ccabc/linux-arm-scaleway 2021-04-08T19:22:05-be791d0-8c163e8/linux-arm-scaleway 2021-04-08T02:08:45-b261fe9-8c163e8/linux-arm-scaleway 2021-04-07T10:44:45-3ac76b8-8c163e8/linux-arm-scaleway
The latest failure here was #59910 (just to point it out explicitly). @cherrymui notes she tried reproducing this in the past on a gomote but couldn't. I'm not sure what the next steps are here. Is #59910 the only one in recent memory? I'll take a look at the history.
At face value, the pattern here seems to maybe be weak memory architectures?
cc @findleyr for visibility. @findleyr has been working on parallelizing some of the tests but I believe that CL has not yet landed, so can't be the culprit.
Is #59910 the only one in recent memory?
From a cursory GitHub search,
is the only other recent report that looks related.
FWIW #59080 also looks like an inexplicable NPE on ppc64.
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
These recent failures to me indicate a bug on ppc64.
CC @golang/ppc64
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Two recent failures on darwin/amd64
. That doesn't seem consistent with a PPC64-specific bug, but maybe we're seeing similar symptoms from two different bugs.
(Perhaps the PPC64 failures were another symptom of #60368? If so, that suggests that the darwin/amd64
failures may also be symptomatic of register corruption.)
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
The linux-riscv64-unmatched failure is #61932.
Is a go/types race implicated here? Could/would that cause the types of SIGSEGVs we see in garbage collection?
Another consideration is that go/types is the leading cause of allocation in a lot of tests.
Personally my theory is that go/types
is triggering some kind of defer
bug in the runtime and/or compiler.
It is, to my knowledge, one of the only places in the standard library where we use a recover
of an explicit panic
in normal operation:
https://cs.opensource.google/go/go/+/master:src/go/types/check.go;l=329-341;drc=2420cc0d00fccb28c4d155941b2853c14bee4e34
(fmt
does recover panics too, but they're usually nil-deferences rather than explicit panic
statements and the defer
/recover
pairing is much more immediate.)
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
Found new dashboard test flakes for:
#!watchflakes
post <- log ~ `SIGSEGV(.*\n)*\ngoroutine \d+ \[.+\]:\n(\s*.+\n)*go/types\.\(\*Checker\)\.(?:rawExpr|exprInternal)`
2020-03-01T21:04:30-a4f7b08/darwin-amd64-nocgo
This may be related to #36605, although the failures reported for that issue are supposed to be fixed in 1.14 and at tip (https://github.com/golang/go/issues/36605#issuecomment-576924501).
CC @matloob @griesemer