search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.98k stars 17.67k forks source link

runtime: memory corruption on darwin/amd64 #60449

Open gopherbot opened 1 year ago

gopherbot commented 1 year ago 
#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))

Issue created automatically to collect these failures.

Example (log):

— watchflakes

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "exp" && section == "go test -short ./..." && test == ""
2023-05-22 17:56 darwin-amd64-12_0 exp@2e198f4a go@f0e356a0 (log)

— watchflakes

bcmills commented 1 year ago

Runtime heap corruption in cmd/go during go test. Looks like a runtime bug to me?

mknyszek commented 1 year ago

@prattmic notes that this is a failure on Darwin and there's a fork/exec goroutine here. Sounds like an existing issue possibly (#59995)?

mknyszek commented 1 year ago

We should probably aggregate such issues into a single issue.

bcmills commented 1 year ago

This failure mode looks substantially different from the other Darwin cluster described in https://github.com/golang/go/issues/54461#issuecomment-1591922415 (which includes #59995).

That cluster of failures is characterized by a goroutine stuck copying a pipe connected to a dead child process: either stdout, stderr, or (more rarely) the one created by syscall.forkExecPipe, but there is typically no sign of memory corruption. In contrast, the log attached to this issue shows memory corruption, and has a syscall.Wait4 frame on the stack that suggests that the child process is probably still alive.

So I think this issue is probably independent of #59995.

mknyszek commented 1 year ago

Agreed. I changed my mind after posting the last message and looking at the other failures again, but then I had a meeting. :P

mknyszek commented 1 year ago

I was going back through issues assigned to me (oops, I have a backlog) and I noticed that #58715 is also memory corruption, but none of those failures include forkExec. Just mentioning it here since it might be related.

bcmills commented 1 year ago

https://github.com/golang/go/issues/55167#issuecomment-1595404481 looks like plausibly the same kind of corruption; updated the watchflakes pattern to catch it more specifically.

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && `fatal error: found pointer to free object`
2023-06-16 20:47 darwin-amd64-10_15 go@0183c1aa cmd/go.TestScript (log) vcs-test.golang.org rerouted to http://127.0.0.1:54179 https://vcs-test.golang.org rerouted to https://127.0.0.1:54180 go test proxy running at GOPROXY=http://127.0.0.1:54181/mod runtime: marked free object in span 0x1e6f868, elemsize=48 freeindex=106 (bad use of unsafe.Pointer? try -d=checkptr) 0xc000042000 alloc marked 0xc000042030 alloc marked 0xc000042060 alloc marked 0xc000042090 alloc marked 0xc0000420c0 alloc marked 0xc0000420f0 alloc marked ... fatal error: found pointer to free object runtime stack: runtime.throw({0x17b70aa?, 0xc000043d10?}) /tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x70000d701c50 sp=0x70000d701c20 pc=0x103b47c runtime.(*mspan).reportZombies(0x1e6f868) /tmp/buildlet/go/src/runtime/mgcsweep.go:861 +0x2e5 fp=0x70000d701cd0 sp=0x70000d701c50 pc=0x102a745 runtime.(*sweepLocked).sweep(0x2030002?, 0x0) /tmp/buildlet/go/src/runtime/mgcsweep.go:644 +0xa88 fp=0x70000d701dd8 sp=0x70000d701cd0 pc=0x1029f68 runtime.(*mcentral).uncacheSpan(0x70000d701e50?, 0x1031946?) ... /tmp/buildlet/go/src/os/file_unix.go:315 +0x75 fp=0xc000856d10 sp=0xc000856cb8 pc=0x10b9815 os.(*File).Close(...) /tmp/buildlet/go/src/os/file_posix.go:23 os.WriteFile({0xc000b882a0?, 0x64?}, {0xc00080de81, 0xe, 0xf}, 0x18e2578?) /tmp/buildlet/go/src/os/file.go:766 +0x7d fp=0xc000856d58 sp=0xc000856d10 pc=0x10b819d cmd/go/internal/script.(*State).ExtractFiles(0xc000e7ad00, 0xc000538a80?) /tmp/buildlet/go/src/cmd/go/internal/script/state.go:171 +0x1c9 fp=0xc000856e20 sp=0xc000856d58 pc=0x16421c9 cmd/go_test.TestScript.func3(0xc0005d7ba0) /tmp/buildlet/go/src/cmd/go/script_test.go:128 +0x2a5 fp=0xc000856f70 sp=0xc000856e20 pc=0x166ff25 testing.tRunner(0xc0005d7ba0, 0xc0004daeb0)

— watchflakes

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap`)
2023-07-24 17:08 darwin-amd64-12_0 tools@67ba5997 go@df0a1297 x/tools/go/packages.TestAll (log) runtime: pointer 0xc01477ee10 to unused region of span span.base()=0xc01477e000 span.limit=0xc01477ffe0 span.state=1 runtime: found in object at *(0xc0394e8000+0x17d8) object=0xc0394e8000 s.base()=0xc039408000 s.limit=0xc03a370000 s.spanclass=0 s.elemsize=16154624 s.state=mSpanInUse *(object+0) = 0x4 *(object+8) = 0x1350008 *(object+16) = 0x14d37c0 *(object+24) = 0x1351c90 *(object+32) = 0xc037d90e38 *(object+40) = 0x5 *(object+48) = 0x13500f8 ... fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0x12f6b3f?, 0x6?}) /tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x700005b78d90 sp=0x700005b78d60 pc=0x10360dc runtime.badPointer(0x5f407c8, 0x700005b78e10?, 0xc0394e8000, 0xc00791dfe0?) /tmp/buildlet/go/src/runtime/mbitmap.go:321 +0x165 fp=0x700005b78de8 sp=0x700005b78d90 pc=0x1013885 runtime.findObject(0xc02fca9280?, 0x10563de?, 0x106c4a0?) /tmp/buildlet/go/src/runtime/mbitmap.go:364 +0xa6 fp=0x700005b78e20 sp=0x700005b78de8 pc=0x1013a06 runtime.scanobject(0xc0394e8000, 0xc000030140) ... /tmp/buildlet/go/src/runtime/chan.go:442 +0x12 fp=0xc00010ac98 sp=0xc00010ac70 pc=0x1005f12 testing.(*T).Run(0xc00cc761a0, {0x12e60ab?, 0xc000171e20?}, 0xc00d581278) /tmp/buildlet/go/src/testing/testing.go:1649 +0x3c8 fp=0xc00010ad58 sp=0xc00010ac98 pc=0x10eec48 golang.org/x/tools/go/packages_test.testLoadDifferentPatterns(0xc00cc761a0, {0x1350d40, 0x150c880}) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/overlay_test.go:997 +0x517 fp=0xc00010aed8 sp=0xc00010ad58 pc=0x1257a97 golang.org/x/tools/go/packages_test.testAllOrModulesParallel.func1(0xc00cc761a0, {0x1350d40, 0x150c880}) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages_test.go:81 +0x138 fp=0xc00010af30 sp=0xc00010aed8 pc=0x1258f78 golang.org/x/tools/go/packages/packagestest.TestAll.func1(0xc00cc761a0) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packagestest/export.go:162 +0x51 fp=0xc00010af70 sp=0xc00010af30 pc=0x1247c91 testing.tRunner(0xc00cc761a0, 0xc00e8fd5c0)

— watchflakes

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty`)
2023-06-23 21:36 darwin-amd64-10_15 telemetry@5a4cd7a1 go@bca81759 (log) fatal error: workbuf is not empty runtime stack: runtime.throw({0x16519f6?, 0x700003f65e08?}) /tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x700003f65de0 sp=0x700003f65db0 pc=0x103449d runtime.(*workbuf).checkempty(0x4582128?) /tmp/buildlet/go/src/runtime/mgcwork.go:344 +0x36 fp=0x700003f65e00 sp=0x700003f65de0 pc=0x1025a16 runtime.getempty() /tmp/buildlet/go/src/runtime/mgcwork.go:357 +0x1e8 fp=0x700003f65e60 sp=0x700003f65e00 pc=0x1025c28 runtime.handoff(0xc001c81000) /tmp/buildlet/go/src/runtime/mgcwork.go:439 +0x25 fp=0x700003f65e90 sp=0x700003f65e60 pc=0x1025e25 runtime.(*gcWork).balance(0xc000035238) /tmp/buildlet/go/src/runtime/mgcwork.go:297 +0x45 fp=0x700003f65ea8 sp=0x700003f65e90 pc=0x1025905 runtime.gcDrain(0xc000035238, 0x3) /tmp/buildlet/go/src/runtime/mgcmark.go:1085 +0x125 fp=0x700003f65f08 sp=0x700003f65ea8 pc=0x101e7e5 runtime.gcBgMarkWorker.func2() /tmp/buildlet/go/src/runtime/mgc.go:1348 +0xad fp=0x700003f65f58 sp=0x700003f65f08 pc=0x101af6d runtime.systemstack() /tmp/buildlet/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x700003f65f60 sp=0x700003f65f58 pc=0x10655c9

— watchflakes

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty`)
2023-07-27 16:59 darwin-amd64-12_0 tools@bacac149 go@77863e42 x/tools/gopls/test.TestStringChanges (log) runtime: marked free object in span 0x6bf8728, elemsize=16 freeindex=10 (bad use of unsafe.Pointer? try -d=checkptr) 0xc00027c000 alloc marked 0xc00027c010 alloc marked 0xc00027c020 alloc marked 0xc00027c030 alloc marked 0xc00027c040 alloc marked 0xc00027c050 alloc marked 0xc00027c060 alloc marked 0xc00027c070 alloc marked 0xc00027c080 alloc marked ... fatal error: found pointer to free object runtime stack: runtime.throw({0x1248123?, 0xc00027d4d0?}) /tmp/buildlet/go/src/runtime/panic.go:1077 +0x5c fp=0x700010f8b8d8 sp=0x700010f8b8a8 pc=0x103ac7c runtime.(*mspan).reportZombies(0x6bf8728) /tmp/buildlet/go/src/runtime/mgcsweep.go:861 +0x2e5 fp=0x700010f8b958 sp=0x700010f8b8d8 pc=0x102a025 runtime.(*sweepLocked).sweep(0x700010f8ba90?, 0x0) /tmp/buildlet/go/src/runtime/mgcsweep.go:644 +0xa88 fp=0x700010f8ba60 sp=0x700010f8b958 pc=0x1029848 runtime.(*mcentral).uncacheSpan(0xc000042900?, 0x700010f8bab8?) ... /tmp/buildlet/gopath/pkg/mod/github.com/google/go-cmp@v0.5.9/cmp/report.go:47 +0x89 fp=0xc0002a3ca8 sp=0xc0002a3c38 pc=0x11b38a9 github.com/google/go-cmp/cmp.Diff({0x121a3c0, 0xc00018ea80}, {0x121a3c0, 0xc00018ee00}, {0xc0001c2fc0?, 0x1252abc?, 0xc000736000?}) /tmp/buildlet/gopath/pkg/mod/github.com/google/go-cmp@v0.5.9/cmp/compare.go:130 +0x16f fp=0xc0002a3d08 sp=0xc0002a3ca8 pc=0x11aa0ef golang.org/x/tools/gopls/test_test.tryChange(0x133f, 0x1348, {0x1240e31, 0x2}) /tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:134 +0x366 fp=0xc0002a3e38 sp=0xc0002a3d08 pc=0x11dd026 golang.org/x/tools/gopls/test_test.allDeltas(0xc0000f0340, {0xc0000185a0, 0x12, 0x100000001?}, {0xc0002a3f40, 0x2, 0xfc90b1b56b?}) /tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:94 +0x136 fp=0xc0002a3f00 sp=0xc0002a3e38 pc=0x11dcb16 golang.org/x/tools/gopls/test_test.TestStringChanges(0x0?) /tmp/buildlet/gopath/src/golang.org/x/tools/gopls/test/json_test.go:67 +0x11c fp=0xc0002a3f70 sp=0xc0002a3f00 pc=0x11dc73c testing.tRunner(0xc0000f0340, 0x1253838)

— watchflakes

mknyszek commented 1 year ago

These represent more general runtime memory corruption, but I suspect this is all part of the same issue.

2023-08-21 23:20 darwin-amd64-11_0 go@e3c80f57 cmd/go.TestScript (log) 2023-08-22 08:38 darwin-amd64-longtest go@05f95115 runtime.TestEINTR (log)

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty` || `schedule: holding locks` || `fatal error: bad summary data`)
2023-08-21 23:20 darwin-amd64-11_0 go@e3c80f57 cmd/go.TestScript (log) vcs-test.golang.org rerouted to http://127.0.0.1:53045 https://vcs-test.golang.org rerouted to https://127.0.0.1:53046 go test proxy running at GOPROXY=http://127.0.0.1:53047/mod --- FAIL: TestScript (1.07s) --- FAIL: TestScript/mod_list_test (1.11s) script_test.go:132: 2023-08-21T23:59:53Z script_test.go:134: $WORK=/tmp/buildlet/tmp/cmd-go-test-340886821/tmpdir633785442/mod_list_test250661805 script_test.go:156: > env GO111MODULE=on # go list -compiled -test must handle test-only packages ... fatal error: bad summary data runtime stack: runtime.throw({0x880bda5?, 0x7000100efed0?}) /tmp/buildlet/go/src/runtime/panic.go:1018 +0x5c fp=0x7000100efe08 sp=0x7000100efdd8 pc=0x81c693c runtime.(*pageAlloc).alloc(0x8e82ca8, 0x4) /tmp/buildlet/go/src/runtime/mpagealloc.go:896 +0x26a fp=0x7000100efe50 sp=0x7000100efe08 pc=0x81bbe0a runtime.(*mheap).allocSpan(0x8e82ca0, 0x4, 0x0, 0x8a?) /tmp/buildlet/go/src/runtime/mheap.go:1247 +0x192 fp=0x7000100efef0 sp=0x7000100efe50 pc=0x81b8052 runtime.(*mheap).alloc.func1() ... cmd/go/internal/work.ActorFunc.Act(0x8a73460?, 0x8e99720?, {0x8a73460?, 0x8e99720?}, 0x0?) /tmp/buildlet/go/src/cmd/go/internal/work/action.go:78 +0x2d fp=0xc00009fd70 sp=0xc00009fd40 pc=0x870282d cmd/go/internal/work.(*Builder).Do.func3({0x8a73460, 0x8e99720}, 0xc000479080) /tmp/buildlet/go/src/cmd/go/internal/work/exec.go:151 +0x80c fp=0xc00009ff20 sp=0xc00009fd70 pc=0x871106c cmd/go/internal/work.(*Builder).Do.func4() /tmp/buildlet/go/src/cmd/go/internal/work/exec.go:219 +0xb9 fp=0xc00009ffe0 sp=0xc00009ff20 pc=0x8710699 runtime.goexit() /tmp/buildlet/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x81fd701 created by cmd/go/internal/work.(*Builder).Do in goroutine 1 /tmp/buildlet/go/src/cmd/go/internal/work/exec.go:205 +0x3fe
2023-08-22 08:38 darwin-amd64-longtest go@05f95115 runtime.TestEINTR (log) fatal error: schedule: holding locks runtime stack: runtime.throw({0x632a2a8?, 0x66cf960?}) /tmp/buildlet/go/src/runtime/panic.go:1018 +0x5c fp=0x7ff7baf93388 sp=0x7ff7baf93358 pc=0x5faa69c runtime.schedule() /tmp/buildlet/go/src/runtime/proc.go:3577 +0x276 fp=0x7ff7baf933c0 sp=0x7ff7baf93388 pc=0x5fb50d6 runtime.park_m(0xc0004cf520?) /tmp/buildlet/go/src/runtime/proc.go:3765 +0x12d fp=0x7ff7baf93408 sp=0x7ff7baf933c0 pc=0x5fb546d runtime.mcall() ... /tmp/buildlet/go/src/os/exec/exec.go:590 +0x2d fp=0xc00018cd70 sp=0xc00018cd50 pc=0x60b0eed os/exec.(*Cmd).CombinedOutput(0xc000610000) /tmp/buildlet/go/src/os/exec/exec.go:977 +0x94 fp=0xc00018cd98 sp=0xc00018cd70 pc=0x60b2b34 runtime_test.runBuiltTestProg(0xc000824d00, {0xc00002c500, 0x34}, {0x631da22, 0x5}, {0x0, 0x0, 0x0?}) /tmp/buildlet/go/src/runtime/crash_test.go:76 +0x327 fp=0xc00018ce80 sp=0xc00018cd98 pc=0x6265ae7 runtime_test.runTestProg(0xc000824d00, {0x6320719, 0xb}, {0x631da22, 0x5}, {0x0, 0x0, 0x0}) /tmp/buildlet/go/src/runtime/crash_test.go:59 +0x19e fp=0xc00018cef0 sp=0xc00018ce80 pc=0x626575e runtime_test.TestEINTR(0xc000824d00) /tmp/buildlet/go/src/runtime/crash_cgo_test.go:736 +0x5c fp=0xc00018cf70 sp=0xc00018cef0 pc=0x62647dc testing.tRunner(0xc000824d00, 0x6482498)

— watchflakes

mknyszek commented 1 year ago

https://github.com/golang/go/issues/62165 may be related.

mknyszek commented 1 year ago

Actually, it almost certainly is the same memory corruption. Folding it in.

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (`fatal error: found pointer to free object` || `fatal error: found bad pointer in Go heap` || `fatal error: workbuf is not empty` || `schedule: holding locks` || `fatal error: bad summary data` ||  (`panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))
2023-08-19 23:35 darwin-amd64-12_0 go@0b6ea6bb (log) panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x108 pc=0xab9995b] goroutine 3 [running]: panic({0xb2640c0?, 0xb652e40?}) runtime/panic.go:774 +0x158 fp=0xc0004b6ec8 sp=0xc0004b6e18 pc=0xaba9398 runtime.panicmem(...) runtime/panic.go:261 runtime.sigpanic() runtime/signal_unix.go:862 +0x378 fp=0xc0004b6f28 sp=0xc0004b6ec8 pc=0xabc0fd8 runtime.sweepone() runtime/mgcsweep.go:361 +0x1b fp=0xc0004b6f78 sp=0xc0004b6f28 pc=0xab9995b runtime.bgsweep(0x0?) runtime/mgcsweep.go:301 +0xff fp=0xc0004b6fc8 sp=0xc0004b6f78 pc=0xab997ff runtime.gcenable.gowrap1() runtime/mgc.go:203 +0x25 fp=0xc0004b6fe0 sp=0xc0004b6fc8 pc=0xab8e6e5 runtime.goexit() runtime/asm_amd64.s:1650 +0x1 fp=0xc0004b6fe8 sp=0xc0004b6fe0 pc=0xabdff61 created by runtime.gcenable in goroutine 1 runtime/mgc.go:203 +0x66
2023-09-02 22:09 darwin-amd64-race go@0cd473d9 (log) runtime: max = 8, npages = 4 runtime: searchIdx = 255, p.searchAddr = 0xc0001fe000 fatal error: bad summary data runtime stack: runtime.throw({0xbe124d7?, 0xb8c3f16?}) runtime/panic.go:1018 +0x5c fp=0x7ff7b566b570 sp=0x7ff7b566b540 pc=0xb8ccdbc runtime.(*pageAlloc).alloc(0xc3a8268, 0x4) runtime/mpagealloc.go:891 +0x26a fp=0x7ff7b566b5b8 sp=0x7ff7b566b570 pc=0xb8c3c6a runtime.(*mheap).allocSpan(0xc3a8260, 0x4, 0x0, 0x4d?) ... goroutine 26 [running]: goroutine running on other thread; stack unavailable created by cmd/go/internal/work.(*Builder).Do in goroutine 1 cmd/go/internal/work/exec.go:205 +0x3fe goroutine 27 [running]: goroutine running on other thread; stack unavailable created by cmd/go/internal/work.(*Builder).Do in goroutine 1 cmd/go/internal/work/exec.go:205 +0x3fe

— watchflakes

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    `fatal error: found pointer to free object` ||
    `fatal error: found bad pointer in Go heap` ||
    `fatal error: workbuf is not empty` ||
    `schedule: holding locks` ||
    `fatal error: bad summary data` ||
    `fatal: morestack on gsignal` ||
    (`panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))
2023-10-11 20:21 darwin-amd64-longtest go@53d000d7 runtime.TestEINTR (log) fatal: morestack on gsignal SIGSEGV: segmentation violation PC=0x4000 m=12 sigcode=1 addr=0x4000 rax 0xffffffffffffffff rbx 0x0 rcx 0x7ff80f87e0ce rdx 0xffffffffffffffff rdi 0x3c rsi 0x0 ... /tmp/buildlet/go/src/os/exec/exec.go:607 +0x2d fp=0xc000062d60 sp=0xc000062d48 pc=0xcf6c38d os/exec.(*Cmd).CombinedOutput(0xc00068c580) /tmp/buildlet/go/src/os/exec/exec.go:1012 +0x85 fp=0xc000062d88 sp=0xc000062d60 pc=0xcf6dfe5 runtime_test.runBuiltTestProg(0xc0001db520, {0xc000d7c100, 0x33}, {0xd1dfc06, 0x5}, {0x0?, 0x0, 0x0?}) /tmp/buildlet/go/src/runtime/crash_test.go:87 +0x327 fp=0xc000062e78 sp=0xc000062d88 pc=0xd128ae7 runtime_test.runTestProg(0xc0001db520, {0xd1e28ee, 0xb}, {0xd1dfc06, 0x5}, {0x0?, 0x0, 0x0}) /tmp/buildlet/go/src/runtime/crash_test.go:70 +0x1a5 fp=0xc000062ef0 sp=0xc000062e78 pc=0xd128745 runtime_test.TestEINTR(0xc0001db520) /tmp/buildlet/go/src/runtime/crash_cgo_test.go:736 +0x5c fp=0xc000062f70 sp=0xc000062ef0 pc=0xd12735c testing.tRunner(0xc0001db520, 0xd341d98)

— watchflakes

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    `fatal error: found pointer to free object` ||
    `fatal error: found bad pointer in Go heap` ||
    `fatal error: workbuf is not empty` ||
    `schedule: holding locks` ||
    `fatal error: bad summary data` ||
    `fatal: morestack on gsignal` ||
    (`panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))
2023-10-25 20:47 darwin-amd64-12_0 tools@672de526 go@d4863909 x/tools/go/packages.TestAll (log) runtime: pointer 0xc00c2ddf80 to unused region of span span.base()=0xc00c2dc000 span.limit=0xc00c2de000 span.state=1 runtime: found in object at *(0xc00565a000+0x7e78) object=0xc00565a000 s.base()=0xc0055fa000 s.limit=0xc005669f90 s.spanclass=0 s.elemsize=458752 s.state=mSpanInUse *(object+0) = 0x0 *(object+8) = 0x0 *(object+16) = 0x0 *(object+24) = 0x0 *(object+32) = 0x0 *(object+40) = 0x0 *(object+48) = 0x0 ... fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0x12e70e5?, 0x6?}) /tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x7000039c3d90 sp=0x7000039c3d60 pc=0x103517d runtime.badPointer(0xac04af8, 0x7000039c3e08?, 0xc00565a000, 0x7000039c3e98?) /tmp/buildlet/go/src/runtime/mbitmap.go:314 +0x150 fp=0x7000039c3de0 sp=0x7000039c3d90 pc=0x1013f50 runtime.findObject(0xc00bb179a0?, 0xc029e94000?, 0x0?) /tmp/buildlet/go/src/runtime/mbitmap.go:357 +0xa6 fp=0x7000039c3e18 sp=0x7000039c3de0 pc=0x10140e6 runtime.scanobject(0xc000030138?, 0xc000030138) ... /tmp/buildlet/go/src/runtime/chan.go:442 +0x18 fp=0xc00ac2ac98 sp=0xc00ac2ac70 pc=0x1005ff8 testing.(*T).Run(0xc016e0e680, {0x12d5ea0?, 0xc000287e20?}, 0xc01b3c4090) /tmp/buildlet/go/src/testing/testing.go:1630 +0x405 fp=0xc00ac2ad58 sp=0xc00ac2ac98 pc=0x10ee6e5 golang.org/x/tools/go/packages_test.testLoadDifferentPatterns(0xc016e0e680, {0x13387f8, 0x14e2568}) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/overlay_test.go:996 +0x529 fp=0xc00ac2aed8 sp=0xc00ac2ad58 pc=0x124c3a9 golang.org/x/tools/go/packages_test.testAllOrModulesParallel.func1(0xc016e0e680, {0x13387f8, 0x14e2568}) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages_test.go:80 +0x138 fp=0xc00ac2af30 sp=0xc00ac2aed8 pc=0x124d8b8 golang.org/x/tools/go/packages/packagestest.TestAll.func1(0xc016e0e680) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packagestest/export.go:161 +0x57 fp=0xc00ac2af70 sp=0xc00ac2af30 pc=0x123be97 testing.tRunner(0xc016e0e680, 0xc015703080)

— watchflakes

dmitshur commented 1 year ago

Possibly related, seen in https://storage.googleapis.com/go-build-log/08b2f1f7/darwin-amd64-13_5b47b2a9.log (from CL 538678):

fatal error: workbuf is empty (notably the opposite of "not empty")

gopherbot commented 1 year ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    `fatal error: found pointer to free object` ||
    `fatal error: found bad pointer in Go heap` ||
    `fatal error: workbuf is not empty` ||
    `schedule: holding locks` ||
    `fatal error: bad summary data` ||
    `fatal: morestack on gsignal` ||
    (`panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))
2023-11-09 18:02 darwin-amd64-11_0 tools@96815ee5 go@e1dc209b x/tools/gopls/internal/regtest/marker (log) runtime: pointer 0xc000ac2a80 to unused region of span span.base()=0xc000ac2000 span.limit=0xc000ac3ff0 span.state=1 runtime: found in object at *(0xc001e3a530+0x8) object=0xc001e3a530 s.base()=0xc001e3a000 s.limit=0xc001e3c000 s.spanclass=4 s.elemsize=16 s.state=mSpanInUse *(object+0) = 0x1eae368 *(object+8) = 0xc000ac2a80 <== fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0x1c63356?, 0x6?}) /tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x70000b184d90 sp=0x70000b184d60 pc=0x103677d runtime.badPointer(0x2abc2298, 0x70000b184e08?, 0xc001e3a530, 0x70000b184e58?) /tmp/buildlet/go/src/runtime/mbitmap.go:314 +0x150 fp=0x70000b184de0 sp=0x70000b184d90 pc=0x1014d50 runtime.findObject(0xc001e3a530?, 0xc00004ef00?, 0x100000004?) /tmp/buildlet/go/src/runtime/mbitmap.go:357 +0xa6 fp=0x70000b184e18 sp=0x70000b184de0 pc=0x1014ee6 runtime.scanobject(0xc000050138?, 0xc000050138) /tmp/buildlet/go/src/runtime/mgcmark.go:1335 +0x165 fp=0x70000b184ea8 sp=0x70000b184e18 pc=0x101f5c5 runtime.gcDrain(0xc000050138, 0x2) /tmp/buildlet/go/src/runtime/mgcmark.go:1103 +0x1d4 fp=0x70000b184f08 sp=0x70000b184ea8 pc=0x101ee34 runtime.gcBgMarkWorker.func2() /tmp/buildlet/go/src/runtime/mgc.go:1363 +0x157 fp=0x70000b184f58 sp=0x70000b184f08 pc=0x101b5b7 runtime.systemstack() /tmp/buildlet/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x70000b184f60 sp=0x70000b184f58 pc=0x106b5a9

— watchflakes

gopherbot commented 12 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))
2023-11-08 14:16 darwin-amd64-12_0 telemetry@d4a65260 go@7a1fce87 (log)

— watchflakes

gopherbot commented 12 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))
2023-11-10 20:32 darwin-amd64-13 exp@9a3e6036 go@a0df2388 x/exp/cmd/gorelease.TestRelease.testRelease (log) runtime: pointer 0xc0024e40c0 to unused region of span span.base()=0xc00372c000 span.limit=0xc00372df80 span.state=1 runtime: found in object at *(0xc002a00000+0x181c0) object=0xc002a00000 s.base()=0xc002a00000 s.limit=0xc002a6ff90 s.spanclass=0 s.elemsize=458752 s.state=mSpanInUse *(object+0) = 0x785243bce938bd *(object+8) = 0xc0016e3420 *(object+16) = 0xc00103d460 *(object+24) = 0xc0011157e0 *(object+32) = 0xc0018870e0 *(object+40) = 0xc000cd8720 *(object+48) = 0xc00115e5e0 ... fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0xe9ecc10?, 0x6?}) /tmp/buildlet/go/src/runtime/panic.go:1016 +0x5c fp=0x70000470cd88 sp=0x70000470cd58 pc=0xe76a91c runtime.badPointer(0x55cd33c8, 0xc0024e40c0, 0xc002a00000, 0x181c0) /tmp/buildlet/go/src/runtime/mbitmap.go:286 +0x165 fp=0x70000470cdd8 sp=0x70000470cd88 pc=0xe748305 runtime.findObject(0xc001b1ef60?, 0xc001ed3968?, 0x1?) /tmp/buildlet/go/src/runtime/mbitmap.go:329 +0xa6 fp=0x70000470ce10 sp=0x70000470cdd8 pc=0xe748486 runtime.scanobject(0xc000031240?, 0xc000031240) ... /tmp/buildlet/gopath/pkg/mod/golang.org/x/tools@v0.15.0/go/packages/packages.go:262 +0x5c fp=0xc0001c4c30 sp=0xc0001c4bb8 pc=0xe9b90dc golang.org/x/exp/cmd/gorelease.loadPackages({0xeac8d18?, 0xc003542480?}, {0x0?, 0xc002894150?}, {0xc002f4ff20, 0x53}, {0xc0019deb70?, 0xc0023e9996?}, {0xc00272b000, 0x3f, ...}, ...) /tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease.go:1334 +0x125 fp=0xc0001c4db0 sp=0xc0001c4c30 pc=0xe9caa45 golang.org/x/exp/cmd/gorelease.loadDownloadedModule({_, _}, {_, _}, {_, _}, {_, _}) /tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease.go:535 +0x797 fp=0xc0001c4f08 sp=0xc0001c4db0 pc=0xe9c3a37 golang.org/x/exp/cmd/gorelease.runRelease({0xeac8d18, 0xc003542480}, {0xeac5de8, 0xc003542660}, {0xc00241e140, 0x1b}, {0xc001771960?, 0x1, 0x1}) /tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease.go:246 +0xd1c fp=0xc0001c5e08 sp=0xc0001c4f08 pc=0xe9c117c golang.org/x/exp/cmd/gorelease.TestRelease.testRelease.func1(0xc00038f860) /tmp/buildlet/gopath/src/golang.org/x/exp/cmd/gorelease/gorelease_test.go:402 +0x7de fp=0xc0001c5f70 sp=0xc0001c5e08 pc=0xe9d247e testing.tRunner(0xc00038f860, 0xc00037c960)

— watchflakes

gopherbot commented 11 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`))
2023-11-29 14:02 darwin-amd64-13 tools@c8019b94 go@d77307f8 (log) fatal error: workbuf is not empty fatal error: workbuf is not empty runtime stack: runtime.throw({0x1651e3f?, 0x0?}) /tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x7000065f6d20 sp=0x7000065f6cf0 pc=0x103449d runtime.(*workbuf).checkempty(0x0?) /tmp/buildlet/go/src/runtime/mgcwork.go:344 +0x36 fp=0x7000065f6d40 sp=0x7000065f6d20 pc=0x1025a16 runtime.getempty() /tmp/buildlet/go/src/runtime/mgcwork.go:357 +0x1e8 fp=0x7000065f6da0 sp=0x7000065f6d40 pc=0x1025c28 ... runtime.greyobject(0xc000b9eaa0, 0x0?, 0x0?, 0x0?, 0x700006a91e58?, 0x1025dc5?) /tmp/buildlet/go/src/runtime/mgcmark.go:1506 +0x190 fp=0x700006a91e18 sp=0x700006a91dc8 pc=0x101f5f0 runtime.scanobject(0xc000040638?, 0xc000040638) /tmp/buildlet/go/src/runtime/mgcmark.go:1336 +0x187 fp=0x700006a91ea8 sp=0x700006a91e18 pc=0x101f047 runtime.gcDrain(0xc000040638, 0xb) /tmp/buildlet/go/src/runtime/mgcmark.go:1103 +0x1d4 fp=0x700006a91f08 sp=0x700006a91ea8 pc=0x101e894 runtime.gcBgMarkWorker.func2() /tmp/buildlet/go/src/runtime/mgc.go:1365 +0x8f fp=0x700006a91f58 sp=0x700006a91f08 pc=0x101af4f runtime.systemstack() /tmp/buildlet/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x700006a91f60 sp=0x700006a91f58 pc=0x10655c9

— watchflakes

gopherbot commented 11 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-11-29 15:35 darwin-amd64-11_0 tools@147b88d0 go@6111acc8 x/tools/go/packages (log) SIGSEGV: segmentation violation PC=0x8140827 m=12 sigcode=1 addr=0x0 goroutine 0 gp=0xc00edb4680 m=12 mp=0xc00137d008 [idle]: runtime.spanOfUnchecked(...) /tmp/buildlet/go/src/runtime/mheap.go:709 runtime.scanobject(0xc000031268?, 0xc000031268?) /tmp/buildlet/go/src/runtime/mgcmark.go:1407 +0x47 fp=0x700010391ea0 sp=0x700010391e10 pc=0x8140827 runtime.gcDrain(0xc000031268, 0x2) /tmp/buildlet/go/src/runtime/mgcmark.go:1242 +0x1f4 fp=0x700010391f08 sp=0x700010391ea0 pc=0x81401f4 ... r11 0x4c r12 0x700010391e90 r13 0x8190fe1 r14 0xc00edb4680 r15 0x1 rip 0x8140827 rflags 0x10246 cs 0x2b fs 0x0 gs 0x0

— watchflakes

gopherbot commented 10 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-12-07 19:37 darwin-amd64-11_0 pkgsite@00698da1 go@9b4b3e5a (log) runtime: pointer 0xc000ced4b9 to unused region of span span.base()=0xc000cea000 span.limit=0xc000cf9f00 span.state=1 runtime: found in object at *(0xc001265d60+0x0) object=0xc001265d60 s.base()=0xc001264000 s.limit=0xc001265fe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse *(object+0) = 0xc000ced4b9 <== *(object+8) = 0x6 *(object+16) = 0xc0004047e8 *(object+24) = 0x8 *(object+32) = 0xc00133e0a0 *(object+40) = 0x1d *(object+48) = 0xc0000280af ... fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0xf8793f0?, 0x6?}) runtime/panic.go:1023 +0x5c fp=0x700008ed5d88 sp=0x700008ed5d58 pc=0xf2f73dc runtime.badPointer(0x16cb2980, 0xc000ced4b9, 0xc001265d60, 0x0) runtime/mbitmap.go:286 +0x165 fp=0x700008ed5dd8 sp=0x700008ed5d88 pc=0xf2d4e05 runtime.findObject(0xc00002a080?, 0xc001265d60?, 0xf2e8545?) runtime/mbitmap.go:329 +0xa6 fp=0x700008ed5e10 sp=0x700008ed5dd8 pc=0xf2d4f86 runtime.scanobject(0xc00003f768?, 0xc00003f768) ... os/exec.(*Cmd).writerDescriptor.func1() os/exec/exec.go:577 +0x34 fp=0xc001a85f60 sp=0xc001a85f00 pc=0xf3e04f4 os/exec.(*Cmd).Start.func2(0xc0005382c0?) os/exec/exec.go:724 +0x2c fp=0xc001a85fc8 sp=0xc001a85f60 pc=0xf3e118c os/exec.(*Cmd).Start.gowrap1() os/exec/exec.go:736 +0x24 fp=0xc001a85fe0 sp=0xc001a85fc8 pc=0xf3e1124 runtime.goexit({}) runtime/asm_amd64.s:1695 +0x1 fp=0xc001a85fe8 sp=0xc001a85fe0 pc=0xf331c21 created by os/exec.(*Cmd).Start in goroutine 7451 os/exec/exec.go:723 +0x9ab

— watchflakes

gopherbot commented 10 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-12-18 20:56 darwin-amd64-13 tools@ee47fd47 go@c564d4ae (log)

— watchflakes

gopherbot commented 10 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-01-08 18:46 darwin-amd64-12_0 build@21e06540 go@59ecebf4 (log)

— watchflakes

bcmills commented 10 months ago

I wonder if this might be due to an uninitialized pointer field in a syscall result. That could explain the bad pointers, at least, but I guess it doesn't explain the other failure modes.

bcmills commented 10 months ago

darwin/amd64 is supposed to be a first class port.

@golang/runtime, should this be marked as a release-blocker for Go 1.22?

bcmills commented 10 months ago

Possibly related?

gopherbot commented 10 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2023-12-11 23:10 darwin-amd64-nocgo tools@f40889dc go@f4ff0e25 x/tools/go/analysis/passes/deepequalerrors (log) SIGSEGV: segmentation violation PC=0xe7c69a5 m=7 sigcode=1 addr=0x0 goroutine 0 gp=0xc0002081c0 m=7 mp=0xc000204008 [idle]: runtime.(*mheap).allocNeedsZero(0xe7a96f6?, 0x800?, 0xc004000000?) /tmp/buildlet/go/src/runtime/mheap.go:1024 +0x65 fp=0x70000176ee38 sp=0x70000176ee18 pc=0xe7c69a5 runtime.(*mheap).initSpan(0xeca5740, 0x1586cdc8, 0x0, 0xc, 0xc004010000, 0x1) /tmp/buildlet/go/src/runtime/mheap.go:1381 +0x51 fp=0x70000176ee70 sp=0x70000176ee38 pc=0xe7c71b1 runtime.(*mheap).allocSpan(0xeca5740, 0x1, 0x0, 0xc) /tmp/buildlet/go/src/runtime/mheap.go:1340 +0x43d fp=0x70000176ef10 sp=0x70000176ee70 pc=0xe7c6fbd ... r11 0x0 r12 0x80c00407ffff r13 0xec7bbe0 r14 0xc0002081c0 r15 0x3 rip 0xe7c69a5 rflags 0x10246 cs 0x2b fs 0x0 gs 0x0

— watchflakes

mknyszek commented 10 months ago

darwin/amd64 is supposed to be a first class port.

@golang/runtime, should this be marked as a release-blocker for Go 1.22?

I may be misunderstanding the policy, but I think technically it's not a release-blocker since we released Go 1.21 with the same issue, so it's not new to this release. (The fact that we released Go 1.21 without figuring out what it is is on it's own something that we should maybe revisit, but I think that's where we are today.) I don't want to be the sole judge on this so I'll add it to the next weekly meeting and/or revisit this in triage.

@cherrymui spent a good bit of time investigating this and had a lot of difficulty reproducing. (Mentioning that here because I don't think that was recorded in this issue.) The crash rate is also fairly low, so it's difficult to continue to prioritize this against other things that need fixing, especially since the amount of time this bug could take to resolve is unknown and likely high. Again, not trying to make a decision, just trying to rationalize how we got here.

The other aspect to this is that it doesn't seem to have impacted downstream users much, if at all. It's possible that users are just ignoring it, but IIRC other Darwin issues that appeared on the builders have been reported in the past, such as the CPU profile related one.

This makes me wonder if maybe it's our postsubmit Mac machines specifically that are the problem. It also occurs to me that I haven't seen the same kinds of failures on the LUCI infrastructure. Those builders have been up long enough that I would've expected to see at least one such failure by now. They have a few failures from other unique sources, but I haven't seen anything like this yet.

I will say that it's possible the failure got missed on the LUCI builders because watchflakes doesn't run against the LUCI infrastructure yet. At the very least I can do an ad-hoc-but-systematic analysis of all our LUCI darwin builds and see if anything crops up. If not, then that's an interesting data point.

bcmills commented 9 months ago

Previously, the darwin failures in #49692 were resolved by some combination of OS updates and perhaps better ventilation of the builder machines.

Obviously we don't have much control over the ventilation of Amazon's EC2 Mac Minis, though. It's unfortunate that so few Macs support ECC memory. 😒

gopherbot commented 9 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-01-18 23:21 darwin-amd64-13 build@911ff433 go@2540b143 (log)

— watchflakes

cherrymui commented 9 months ago

On the main repo, main branch, watchflakes doesn't find any failure matching this for most recent builds in 45 days. I could run it for longer time period and see.

Update: also no match on LUCI builders in a longer period of time.

gopherbot commented 9 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-01-22 20:52 darwin-amd64-11_0 tools@e2ca5942 go@a95136a8 x/tools/go/analysis/passes/stdmethods.TestAnalyzeEncodingXML (log) fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1026791] runtime stack: runtime.throw({0x12e4d8e?, 0x14b5700?}) /tmp/buildlet/go/src/runtime/panic.go:1047 +0x5d fp=0x7ffeefbff6c8 sp=0x7ffeefbff698 pc=0x103535d runtime.sigpanic() /tmp/buildlet/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0x7ffeefbff728 sp=0x7ffeefbff6c8 pc=0x104c409 runtime.(*mheap).allocNeedsZero(0xc00c100000?, 0x7c?, 0x180?) /tmp/buildlet/go/src/runtime/mheap.go:1021 +0x71 fp=0x7ffeefbff748 sp=0x7ffeefbff728 pc=0x1026791 ... /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:758 +0xd77 fp=0xc00650dc48 sp=0xc00650d9d8 pc=0x1258317 golang.org/x/tools/go/packages.Load(0xc0003c12c0?, {0xc007dd4a70, 0x1, 0x1}) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:249 +0x2c7 fp=0xc00650dcc0 sp=0xc00650dc48 pc=0x12566e7 golang.org/x/tools/go/analysis/analysistest.loadPackages(0x14b03e0, {0x0, 0x0}, {0xc007dd4a70, 0x1, 0x1}) /tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:375 +0x29a fp=0xc00650de50 sp=0xc00650dcc0 pc=0x126361a golang.org/x/tools/go/analysis/analysistest.Run({0x13375a0, 0xc007abb860}, {0x0, 0x0}, 0x14b03e0, {0xc007dd4a70, 0x1, 0x1}) /tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/analysistest/analysistest.go:322 +0xbf fp=0xc00650df18 sp=0xc00650de50 pc=0x1262fbf golang.org/x/tools/go/analysis/passes/stdmethods_test.TestAnalyzeEncodingXML(0x100523d?) /tmp/buildlet/gopath/src/golang.org/x/tools/go/analysis/passes/stdmethods/stdmethods_test.go:20 +0x5f fp=0xc00650df70 sp=0xc00650df18 pc=0x126673f testing.tRunner(0xc007abb860, 0x12ec930)

— watchflakes

gopherbot commented 9 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-02-06 21:16 darwin-amd64-12_0 tools@0d875898 go@f2920803 x/tools/gopls/internal/test/integration/misc.TestRunGovulncheckStd (log) serve.go:441: debug server listening at http://localhost:49266 serve.go:441: debug server listening at http://localhost:49267 #### Start Gopls Test Logs for "TestRunGovulncheckStd/default" [Trace - 20:03:18.284 PM] Sending request 'initialize - (1)'. Params: {"processId":0,"rootUri":"","capabilities":{"workspace":{"workspaceEdit":{"resourceOperations":["rename"]},"didChangeConfiguration":{},"didChangeWatchedFiles":{"dynamicRegistration":true},"configuration":true},"textDocument":{"completion":{"completionItem":{"snippetSupport":true,"tagSupport":{"valueSet":[1]}}},"documentSymbol":{"hierarchicalDocumentSymbolSupport":true},"codeAction":{"codeActionLiteralSupport":{"codeActionKind":{"valueSet":null}}},"publishDiagnostics":{},"semanticTokens":{"requests":{"full":true},"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator"],"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract","async","modification","documentation","defaultLibrary"],"formats":null}},"window":{"workDoneProgress":true}},"initializationOptions":{"codelenses":{"run_govulncheck":true},"completionBudget":"0s","diagnosticsDelay":"10ms","env":{"GO111MODULE":"","GOMODCACHE":"","GOPACKAGESDRIVER":"off","GOPATH":"/tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/gopath","GOPROXY":"file:///tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/proxy","GOSUMDB":"off","GOVULNDB":"file:///tmp/buildlet/tmp/vulndb-test2317938991/ID","_GOPLS_TEST_BINARY_RUN_AS_GOPLS":"true","_GOPLS_TEST_VULNCHECK_GOVERSION":"go1.18"},"verboseWorkDoneProgress":true},"trace":"messages","workspaceFolders":[{"uri":"file:///tmp/buildlet/tmp/gopls-test-76965345/TestRunGovulncheckStd/default/work","name":"work"}]} [Trace - 20:03:18.284 PM] Received response 'initialize - (1)' in 0ms. Result: {"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"save":{}},"completionProvider":{"triggerCharacters":["."]},"hoverProvider":true,"signatureHelpProvider":{"triggerCharacters":["(",","]},"definitionProvider":true,"typeDefinitionProvider":true,"implementationProvider":true,"referencesProvider":true,"documentHighlightProvider":true,"documentSymbolProvider":true,"codeActionProvider":true,"codeLensProvider":{},"documentLinkProvider":{},"workspaceSymbolProvider":true,"documentFormattingProvider":true,"renameProvider":true,"foldingRangeProvider":true,"selectionRangeProvider":true,"executeCommandProvider":{"commands":["gopls.add_dependency","gopls.add_import","gopls.add_telemetry_counters","gopls.apply_fix","gopls.change_signature","gopls.check_upgrades","gopls.diagnose_files","gopls.edit_go_directive","gopls.fetch_vulncheck_result","gopls.gc_details","gopls.generate","gopls.go_get_package","gopls.list_imports","gopls.list_known_packages","gopls.maybe_prompt_for_telemetry","gopls.mem_stats","gopls.regenerate_cgo","gopls.remove_dependency","gopls.reset_go_mod_diagnostics","gopls.run_go_work_command","gopls.run_govulncheck","gopls.run_tests","gopls.start_debugging","gopls.start_profile","gopls.stop_profile","gopls.test","gopls.tidy","gopls.toggle_gc_details","gopls.update_go_sum","gopls.upgrade_dependency","gopls.vendor","gopls.views","gopls.workspace_stats"]},"callHierarchyProvider":true,"semanticTokensProvider":{"legend":{"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","macro","keyword","modifier","comment","string","number","regexp","operator"],"tokenModifiers":["declaration","definition","readonly","static","deprecated","abstract","async","modification","documentation","defaultLibrary"]},"range":true,"full":true},"inlayHintProvider":{},"workspace":{"workspaceFolders":{"supported":true,"changeNotifications":"workspace/didChangeWorkspaceFolders"}}},"serverInfo":{"name":"gopls","version":"{\"GoVersion\":\"go1.21.7\",\"Path\":\"\",\"Main\":{\"Path\":\"\",\"Version\":\"\",\"Sum\":\"\",\"Replace\":null},\"Deps\":null,\"Settings\":null,\"Version\":\"(unknown)\"}"}} ... --- FAIL: TestRunGovulncheckStd (3.56s) --- FAIL: TestRunGovulncheckStd/default (3.53s) vuln_test.go:212: waiting on: Unmeetable: once "completed work for token 680114519661641662" is met, must have: received window/showMessage containing "Found GOSTDLIB" no diagnostics for file "go.mod" err:condition has final verdict Unmeetable state: ... #### diagnostics: go.mod (version 1): #### outstanding work: #### completed work: govulncheck: 1 diagnosing initial workspace load: 1 Setting up workspace: 1 Checking telemetry prompt: 1 diagnosing opened files: 1

— watchflakes

gopherbot commented 9 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-02-13 14:38 darwin-amd64-11_0 tools@1b39a8b6 go@1b541502 x/tools/go/packages (log) runtime: pointer 0xc004212000 to unallocated span span.base()=0xc004212000 span.limit=0xc004213fe0 span.state=0 runtime: found in object at *(0xc015497560+0x58) object=0xc015497560 s.base()=0xc015496000 s.limit=0xc015497fe0 s.spanclass=16 s.elemsize=96 s.state=mSpanInUse *(object+0) = 0xc01540d500 *(object+8) = 0x0 *(object+16) = 0x0 *(object+24) = 0x0 *(object+32) = 0x0 *(object+40) = 0x0 *(object+48) = 0x0 ... fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0x177fe8d?, 0x6?}) /tmp/buildlet/go/src/runtime/panic.go:1021 +0x5c fp=0x70000644cd88 sp=0x70000644cd58 pc=0x152179c runtime.badPointer(0x8f10308, 0xc004212000, 0xc015497560, 0x58) /tmp/buildlet/go/src/runtime/mbitmap.go:286 +0x165 fp=0x70000644cdd8 sp=0x70000644cd88 pc=0x14fdde5 runtime.findObject(0xc02575eb40?, 0xc015497560?, 0x1511665?) /tmp/buildlet/go/src/runtime/mbitmap.go:329 +0xa6 fp=0x70000644ce10 sp=0x70000644cdd8 pc=0x14fdf66 runtime.scanobject(0xc000031268?, 0xc000031268) ... golang.org/x/tools/go/packages.(*loader).loadRecursive(0xc022737ce0?, 0xc000a217d0?) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:822 +0xf fp=0xc000a217a0 sp=0xc000a21768 pc=0x17330cf golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.1(0x0?) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:829 +0x26 fp=0xc000a217c8 sp=0xc000a217a0 pc=0x1733386 golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.gowrap1() /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:831 +0x24 fp=0xc000a217e0 sp=0xc000a217c8 pc=0x1733324 runtime.goexit({}) /tmp/buildlet/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000a217e8 sp=0xc000a217e0 pc=0x155b1c1 created by golang.org/x/tools/go/packages.(*loader).loadRecursive.func1 in goroutine 7427 /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:828 +0x94

— watchflakes

gopherbot commented 7 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-03-27 17:55 darwin-amd64-nocgo go@979b34b3 cmd/compile/internal/ssa [build] (log) panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0xc022ed3] goroutine 34 gp=0xc000104c40 m=0 mp=0xc5076a0 [running]: panic({0xc2ff3e0?, 0xc4f4360?}) runtime/panic.go:778 +0x168 fp=0xc000029570 sp=0xc0000294c0 pc=0xbfcbe08 runtime.panicmem(...) runtime/panic.go:261 runtime.sigpanic() runtime/signal_unix.go:881 +0x378 fp=0xc0000295d0 sp=0xc000029570 pc=0xbfe4ad8 ... cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func2(0xc4fef80) cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:333 +0x19e fp=0xc002a85fa0 sp=0xc002a85ea0 pc=0xc2065de cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3.1(0x5454545454545454?) cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:404 +0x23 fp=0xc002a85fc8 sp=0xc002a85fa0 pc=0xc206403 cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3.gowrap1() cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:406 +0x24 fp=0xc002a85fe0 sp=0xc002a85fc8 pc=0xc2063a4 runtime.goexit({}) runtime/asm_amd64.s:1695 +0x1 fp=0xc002a85fe8 sp=0xc002a85fe0 pc=0xc003dc1 created by cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3 in goroutine 1 cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:403 +0x47

— watchflakes

gopherbot commented 6 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-05-02 22:38 darwin-amd64-11_0 tools@ccdef3cc go@93d8777d x/tools/gopls/internal/test/integration/completion (log) SIGSEGV: segmentation violation PC=0x4b07827 m=5 sigcode=1 addr=0x0 goroutine 0 gp=0xc0001041c0 m=5 mp=0xc000100008 [idle]: runtime.spanOfUnchecked(...) /tmp/buildlet/go/src/runtime/mheap.go:709 runtime.scanobject(0xc00005a168?, 0xc00005a168?) /tmp/buildlet/go/src/runtime/mgcmark.go:1407 +0x47 fp=0x700011677e80 sp=0x700011677df0 pc=0x4b07827 runtime.gcDrain(0xc00005a168, 0x2) /tmp/buildlet/go/src/runtime/mgcmark.go:1242 +0x1f4 fp=0x700011677ee8 sp=0x700011677e80 pc=0x4b071f4 ... r11 0xd0 r12 0x700011677e70 r13 0xc000100530 r14 0xc0001041c0 r15 0x1 rip 0x4b07827 rflags 0x10246 cs 0x2b fs 0x0 gs 0x0

— watchflakes

gopherbot commented 6 months ago

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && goarch == "amd64" && (
    log ~ `fatal error: found pointer to free object` ||
    log ~ `fatal error: found bad pointer in Go heap` ||
    log ~ `fatal error: workbuf is not empty` ||
    log ~ `fatal error: workbuf is empty` ||
    log ~ `schedule: holding locks` ||
    log ~ `fatal error: bad summary data` ||
    log ~ `fatal: morestack on gsignal` ||
    log ~ `runtime\.\(\*mheap\)\.allocNeedsZero` ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && log ~ `^\s*runtime\.sigpanic.*\n\s+.*\n\s*runtime\.sweepone`) ||
    (log ~ `panic: runtime error: invalid memory address or nil pointer dereference` && pkg == "cmd/compile/internal/ssa") ||
    (log ~ `SIGSEGV: segmentation violation` && log ~ `^\s*runtime\.spanOfUnchecked\(`))
2024-05-09 15:16 darwin-amd64-11_0 tools@24f3b32f go@dd6dee48 x/tools/go/packages (log) runtime: pointer 0xc0137b4be0 to unused region of span span.base()=0xc0137b4000 span.limit=0xc0137b6000 span.state=1 runtime: found in object at *(0xc00c0bc000+0xeec8) object=0xc00c0bc000 s.base()=0xc00c09c000 s.limit=0xc00c410000 s.spanclass=0 s.elemsize=3620864 s.state=mSpanInUse *(object+0) = 0xc00f600540 *(object+8) = 0xc00089c660 *(object+16) = 0x0 *(object+24) = 0x0 *(object+32) = 0x0 *(object+40) = 0xfe7b8f8 *(object+48) = 0xc00d9653e0 ... fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?) runtime stack: runtime.throw({0xfda67d6?, 0x6?}) /tmp/buildlet/go/src/runtime/panic.go:1023 +0x48 fp=0x70000bf27d88 sp=0x70000bf27d58 pc=0xfb42928 runtime.badPointer(0x172b25a8, 0xc0137b4be0, 0xc00c0bc000, 0xeec8) /tmp/buildlet/go/src/runtime/mbitmap.go:1247 +0x165 fp=0x70000bf27dd8 sp=0x70000bf27d88 pc=0xfb1ffc5 runtime.findObject(0xc007ad77a0?, 0xc00df2eae0?, 0xfb64a5e?) /tmp/buildlet/go/src/runtime/mbitmap.go:1290 +0xa6 fp=0x70000bf27e10 sp=0x70000bf27dd8 pc=0xfb20146 runtime.scanobject(0xc000037c50?, 0xc000037c50) ... golang.org/x/tools/go/packages.(*loader).loadRecursive(0xc006a2eee8?, 0xc000a787d0?) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:924 +0x3b fp=0xc000a787a0 sp=0xc000a78768 pc=0xfd598fb golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.1(0x0?) /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:931 +0x26 fp=0xc000a787c8 sp=0xc000a787a0 pc=0xfd59b66 golang.org/x/tools/go/packages.(*loader).loadRecursive.func1.gowrap1() /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:933 +0x24 fp=0xc000a787e0 sp=0xc000a787c8 pc=0xfd59b04 runtime.goexit({}) /tmp/buildlet/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc000a787e8 sp=0xc000a787e0 pc=0xfb7d7e1 created by golang.org/x/tools/go/packages.(*loader).loadRecursive.func1 in goroutine 7646 /tmp/buildlet/gopath/src/golang.org/x/tools/go/packages/packages.go:930 +0x94

— watchflakes