rhysh
commented
2 years ago This looks like a problem with linux/riscv64 signal handling, not specific to the cpu profiler: the tests for the os/signal package see similar crashes (though because of the test structure, it's in a subprocess rather than the test runner itself).
$ GOOS=linux GOARCH=riscv64 go test -c -o /tmp/signal.go1.17.3.test os/signal
$ go version /tmp/signal.go1.17.3.test
/tmp/signal.go1.17.3.test: go1.17.3
$ # scp and ssh to the linux/riscv64 host
sipeed@sipeed:~$ nohup ./stress ./signal.go1.17.3.test -test.short >> ./nohup.log &
sipeed@sipeed:~$ tail -n 1 ./nohup.log
3h18m50s: 2809 runs so far, 3 failures (0.11%)One of the failures is fatal error: unexpected signal during runtime execution. Two of them are fatal error: runtime: split stack overflow.
mengzhuo
cherrymui
4a6f656c
gopherbot
What version of Go are you using (
go version)?(Cross-compiling to linux/riscv64)
Does this issue reproduce with the latest release?
Yes, this problem appears with go1.17.3, the latest stable release.
What operating system and processor architecture are you using (
go env)?I only have a working Go installation on my darwin/amd64 machine (below). I'm cross-compiling from there to linux/riscv64, where I see the problem.
go envOutputWhat did you do?
I've got a small RISCV64 computer: Nezha with the Allwinner D1 SOC. I'm running the runtime/pprof tests at version go1.17.3 and seeing several types of "fatal error" failures.
What did you expect to see?
Near-zero failure rate of runtime/pprof's TestCPUProfileMultithreaded test in short mode when using go1.17.3, and zero crashes.
What did you see instead?
Currently about 1.5% failure rate, all of which are crashes ("fatal error"). (On the plus side, I haven't seen any test failures for that test.)
In the last 42 minutes, the stress run has collected:
Here's an example of each:
"fatal error: unexpected signal during runtime execution" / "[signal SIGSEGV: segmentation violation code=0x1 addr=0x2 pc=0x2]"
``` sipeed@sipeed:~$ cat /tmp/go-stress-20211121T161735-4177513427 fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x2 pc=0x2] goroutine 0 [idle]: runtime: unexpected return pc for runtime.sigpanic called from 0x2 stack: frame={sp:0x3fb005b730, fp:0x3fb005b760} stack=[0x3fb0054000,0x3fb005c000) 0x0000003fb005b630: 0x0000000000000000 0x000000000004cb6c"runtime: unexpected return pc for runtime.sigpanic called from 0x0" / "fatal error: unknown caller pc"
``` sipeed@sipeed:~$ cat /tmp/go-stress-20211121T161735-4026159646 runtime: unexpected return pc for runtime.sigpanic called from 0x0 stack: frame={sp:0x3fb405b910, fp:0x3fb405b940} stack=[0x3fb402e800,0x3fb402f000) fatal error: unknown caller pc runtime stack: runtime.throw({0x1aa247, 0x11}) /usr/local/go/src/runtime/panic.go:1198 +0x60 runtime.gentraceback(0x49820, 0x3fb405b8f8, 0x0, 0x3fb40ba820, 0x0, 0x0, 0x7fffffff, 0x3fb403bfc0, 0x0, 0x0) /usr/local/go/src/runtime/traceback.go:274 +0x17c0 runtime.addOneOpenDeferFrame.func1() /usr/local/go/src/runtime/panic.go:751 +0x88 runtime.systemstack() /usr/local/go/src/runtime/asm_riscv64.s:131 +0x58 goroutine 8 [running]: runtime.systemstack_switch() /usr/local/go/src/runtime/asm_riscv64.s:94 +0x8 fp=0x3fb405b7f0 sp=0x3fb405b7e8 pc=0x80768 runtime.addOneOpenDeferFrame(0x3fb40ba820, 0x49820, 0x3fb405b8f8) /usr/local/go/src/runtime/panic.go:750 +0x80 fp=0x3fb405b828 sp=0x3fb405b7f0 pc=0x4b160 panic({0x1871a0, 0x2c5ab0}) /usr/local/go/src/runtime/panic.go:998 +0xcc fp=0x3fb405b8f8 sp=0x3fb405b828 pc=0x4c09c runtime.panicmem() /usr/local/go/src/runtime/panic.go:221 +0x60 fp=0x3fb405b910 sp=0x3fb405b8f8 pc=0x49820 runtime: unexpected return pc for runtime.sigpanic called from 0x0 stack: frame={sp:0x3fb405b910, fp:0x3fb405b940} stack=[0x3fb402e800,0x3fb402f000) runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:735 +0x264 fp=0x3fb405b940 sp=0x3fb405b910 pc=0x66a04 created by runtime/pprof.TestCPUProfileMultithreaded.func1 /usr/local/go/src/runtime/pprof/pprof_test.go:100 +0x64 goroutine 1 [chan receive]: testing.(*T).Run(0x3fb40ba1a0, {0x1acff7, 0x1b}, 0x1b7750) /usr/local/go/src/testing/testing.go:1307 +0x3a8 testing.runTests.func1(0x3fb40ba1a0) /usr/local/go/src/testing/testing.go:1598 +0x84 testing.tRunner(0x3fb40ba1a0, 0x3fb409dd50) /usr/local/go/src/testing/testing.go:1259 +0xf8 testing.runTests(0x3fb400a030, {0x2cb500, 0x1e, 0x1e}, {0x0, 0x0, 0x0}) /usr/local/go/src/testing/testing.go:1596 +0x434 testing.(*M).Run(0x3fb40a4680) /usr/local/go/src/testing/testing.go:1504 +0x574 main.main() _testmain.go:101 +0x174 goroutine 6 [runnable]: runtime/pprof.cpuHog2(0x0) /usr/local/go/src/runtime/pprof/pprof_test.go:71 +0x1c runtime/pprof.cpuHogger(0x1b7940, 0x2fc268, 0x5f5e100) /usr/local/go/src/runtime/pprof/pprof_test.go:40 +0x5c runtime/pprof.TestCPUProfileMultithreaded.func1(0x5f5e100) /usr/local/go/src/runtime/pprof/pprof_test.go:104 +0x8c runtime/pprof.testCPUProfile(0x3fb40ba340, 0x1b79b8, {0x3fb4062f58, 0x2, 0x2}, {0x0, 0x0, 0x0}, 0x1b7748) /usr/local/go/src/runtime/pprof/pprof_test.go:324 +0x108 runtime/pprof.TestCPUProfileMultithreaded(0x3fb40ba340) /usr/local/go/src/runtime/pprof/pprof_test.go:98 +0xd0 testing.tRunner(0x3fb40ba340, 0x1b7750) /usr/local/go/src/testing/testing.go:1259 +0xf8 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x388 goroutine 7 [sleep]: time.Sleep(0x5f5e100) /usr/local/go/src/runtime/time.go:193 +0x16c runtime/pprof.profileWriter({0x1e0400, 0x3fb400e8a0}) /usr/local/go/src/runtime/pprof/pprof.go:799 +0x68 created by runtime/pprof.StartCPUProfile /usr/local/go/src/runtime/pprof/pprof.go:784 +0x190 ERROR: exit status 2 ```"fatal error: runtime: split stack overflow"
``` sipeed@sipeed:~$ cat /tmp/go-stress-20211121T161735-2006060682 runtime: newstack sp=0x3fa80098d0 stack=[0x3fa8062000, 0x3fa8063000] morebuf={pc:0x0 sp:0x3fa80098d0 lr:0x0} sched={pc:0x667b0 sp:0x3fa80098d0 lr:0x0 ctxt:0x0} runtime: gp=0x3fa80bc340, goid=6, gp->status=0x2 runtime: split stack overflow: 0x3fa80098d0 < 0x3fa8062000 fatal error: runtime: split stack overflow runtime stack: runtime.throw({0x1add89, 0x1d}) /usr/local/go/src/runtime/panic.go:1198 +0x60 runtime.newstack() /usr/local/go/src/runtime/stack.go:1032 +0x850 runtime.morestack() /usr/local/go/src/runtime/asm_riscv64.s:202 +0x70 goroutine 6 [running]: runtime: unexpected return pc for runtime.sigpanic called from 0x0 stack: frame={sp:0x3fa80098d0, fp:0x3fa80098d0} stack=[0x3fa8062000,0x3fa8063000) runtime.sigpanic() /usr/local/go/src/runtime/signal_unix.go:716 +0x10 fp=0x3fa80098d0 sp=0x3fa80098d0 pc=0x667b0 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1306 +0x388 goroutine 1 [chan receive]: testing.(*T).Run(0x3fa80bc000, {0x1acff7, 0x1b}, 0x1b7750) /usr/local/go/src/testing/testing.go:1307 +0x3a8 testing.runTests.func1(0x3fa80bc000) /usr/local/go/src/testing/testing.go:1598 +0x84 testing.tRunner(0x3fa80bc000, 0x3fa809dd50) /usr/local/go/src/testing/testing.go:1259 +0xf8 testing.runTests(0x3fa800a030, {0x2cb500, 0x1e, 0x1e}, {0x0, 0x0, 0x0}) /usr/local/go/src/testing/testing.go:1596 +0x434 testing.(*M).Run(0x3fa80a4680) /usr/local/go/src/testing/testing.go:1504 +0x574 main.main() _testmain.go:101 +0x174 goroutine 7 [sleep]: time.Sleep(0x5f5e100) /usr/local/go/src/runtime/time.go:193 +0x16c runtime/pprof.profileWriter({0x1e0400, 0x3fa800e8a0}) /usr/local/go/src/runtime/pprof/pprof.go:799 +0x68 created by runtime/pprof.StartCPUProfile /usr/local/go/src/runtime/pprof/pprof.go:784 +0x190 goroutine 8 [runnable]: runtime/pprof.cpuHog0(...) /usr/local/go/src/runtime/pprof/pprof_test.go:59 runtime/pprof.cpuHog1(0x0) /usr/local/go/src/runtime/pprof/pprof_test.go:54 +0xc runtime/pprof.cpuHogger(0x1b7938, 0x2fc260, 0x5f5e100) /usr/local/go/src/runtime/pprof/pprof_test.go:40 +0x5c runtime/pprof.TestCPUProfileMultithreaded.func1.1(0x5f5e100, 0x3fa805c3c0) /usr/local/go/src/runtime/pprof/pprof_test.go:101 +0x44 created by runtime/pprof.TestCPUProfileMultithreaded.func1 /usr/local/go/src/runtime/pprof/pprof_test.go:100 +0x64 ERROR: exit status 2 ```