CTSRD-CHERI / cheribsd-ports

FreeBSD ports tree adapted for CheriBSD.
https://CheriBSD.org
Other
5 stars 10 forks source link

lang/go*: aarch64 nondeterministically crashes on Morello kernels #9

Open nwf opened 2 years ago

nwf commented 2 years ago

But seemingly not on aarch64 CheriBSD kernels.

Some initial debugging indicates that the crashes happen very slightly after runtime.asyncPreempt does its thing, and, indeed, disabling asynchronous preemption (export GODEBUG=asyncpreemptoff=1) seems to remove the crashes, suggesting something slightly wrong with signal dispatch, signal handling, or the mechanism used to "inject" asyncPreempt in https://github.com/golang/go/blob/4289bd365c38a44db091396873050cab3cfa8098/src/runtime/signal_unix.go#L344-L349

The crashes are various and not consistent in their diagnostics. I've seen SIGBUS, SIGSEGV, and even the occasional SIGPROT. In general, the symptoms are diagnostic reports like the ones below. Good, small reproducers are hard to come by, but in general attempting to run the go compiler itself will very quickly crash if something is amiss. The peano test program (which lang/go-devel installs as /usr/local64/go/test/peano.go) is also known to be enough to trigger the bug on occasion (again, non-deterministically). Triggering appears slightly more likely under qemu than on real hardware, perhaps because qemu is slower by its reported wall-clock and so the runtime makes more attempts at asynchronous preemption.

If you care to try reproducing, you do not need both a Morello instance and an aarch64 instance as I did before discovering the workaround; instead, you can generate executables on Morello by running go build ... with the above workaround in place and then probe the behavior by unsetting the GODEBUG env var before running the resulting binary.

unexpected fault address 0x166
fatal error: fault
[signal SIGBUS: bus error code=0x1 addr=0x166 pc=0x166]

goroutine 1 [running, locked to thread]:
runtime.throw({0x26a2d6?, 0x700?})
        /usr/local64/go/src/runtime/panic.go:992 +0x50 fp=0xa808f750 sp=0xa808f720 pc=0x425f0
runtime: unexpected return pc for runtime.sigpanic called from 0x166
stack: frame={sp:0xa808f750, fp:0xa808f780} stack=[0xa808e000,0xa8090000)
0x00000000a808f650:  0x0000000000043ea0 <runtime.gwrite+0x00000000000000d0>  0x00000000a808f698
0x00000000a808f660:  0x0000000000043e00 <runtime.gwrite+0x0000000000000030>  0x00000000a808f698
0x00000000a808f670:  0x0000000000043ea0 <runtime.gwrite+0x00000000000000d0>  0x0000000000000002
0x00000000a808f680:  0x00000000002d4104  0x0000000000000001
0x00000000a808f690:  0x0000000000000001  0x00000000a808f6d8
0x00000000a808f6a0:  0x0000000000044630 <runtime.printstring+0x0000000000000050>  0x00000000002d4104
0x00000000a808f6b0:  0x0000000000000001  0x0000000000000001
0x00000000a808f6c0:  0x00000000000597ac <runtime.sigpanic+0x000000000000010c>  0x00000000a808f6d8
0x00000000a808f6d0:  0x00000000000427d0 <runtime.fatalthrow+0x0000000000000040>  0x00000000a808f718
0x00000000a808f6e0:  0x00000000000425f0 <runtime.throw+0x0000000000000050>  0x00000000a808f6f8
0x00000000a808f6f0:  0x0000000000000001  0x00000000000427f0 <runtime.fatalthrow.func1+0x0000000000000000>
0x00000000a808f700:  0x00000000a80021a0  0x00000000000425f0 <runtime.throw+0x0000000000000050>
0x00000000a808f710:  0x00000000a808f720  0x00000000a808f748
0x00000000a808f720:  0x00000000000597c4 <runtime.sigpanic+0x0000000000000124>  0x00000000a808f730
0x00000000a808f730:  0x0000000000042610 <runtime.throw.func1+0x0000000000000000>  0x000000000026a2d6
0x00000000a808f740:  0x0000000000000005  0x00000000b243d758
0x00000000a808f750: <0x0000000000000166  0x000000000026a2d6
0x00000000a808f760:  0x0000000000000700  0x0000000000000166
0x00000000a808f770:  0x00000000a80021a0  0x00000000a8062828
0x00000000a808f780: >0x00000000000210e0 <runtime.growWork_faststr+0x0000000000000070>  0x00000000a81168a8
0x00000000a808f790:  0x00000000a8119030  0x00000000a8116958
0x00000000a808f7a0:  0x00000000a8118e00  0x3200000000000000
0x00000000a808f7b0:  0x0000000000000006  0x000000000000002a
0x00000000a808f7c0:  0x000000000001c328 <runtime.makeBucketArray+0x0000000000000158>  0x000000000000002a
0x00000000a808f7d0:  0x00000000a8116958  0x00000000a808f7e8
0x00000000a808f7e0:  0x00000000a81168a0  0x00000000a8062928
0x00000000a808f7f0:  0x00000000000528f0 <runtime.doInit+0x0000000000000070>  0x0000000000403680
0x00000000a808f800:  0x00000000000000d0  0x00000000a81120c0
0x00000000a808f810:  0x0000000000000030  0x0000000000009338
0x00000000a808f820:  0x00000000b243da00  0x00000000807b5a68
0x00000000a808f830:  0x00000000000000d0  0x0000000000429b20
0x00000000a808f840:  0x0000000000000000  0x0000000000000000
0x00000000a808f850:  0x00000000a81120c0  0x00000000a80628c8
0x00000000a808f860:  0x0000000000088da4 <unicode.init+0x00000000000046a4>  0x00000000000000d0
0x00000000a808f870:  0x0000000000247f20  0x0000000000000001
runtime.sigpanic()
        /usr/local64/go/src/runtime/signal_unix.go:815 +0x124 fp=0xa808f780 sp=0xa808f750 pc=0x597c4
runtime: unexpected return pc for runtime.sigpanic called from 0x0
stack: frame={sp:0xac8a4010, fp:0xac8a4040} stack=[0xac700000,0xacb00000)                                                            0x00000000ac8a3f10:  0x00000000ac8a3ff0  0x00000000ac0021a0
0x00000000ac8a3f20:  0x0000000000000000  0x00000000ac8a3fe8
0x00000000ac8a3f30:  0x0000000000037f14 <runtime.panicmem+0x0000000000000054>  0x0000000000400000                                    0x00000000ac8a3f40:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f50:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f60:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f70:  0x0000000000000000  0x00000000ac0021a0                                                                          0x00000000ac8a3f80:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f90:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3fa0:  0x0000000000000000  0x00000000ac0021c0
0x00000000ac8a3fb0:  0x0000000000000000  0x0000000000075700
0x00000000ac8a3fc0:  0x00000000000d1360  0x0000000000000000
0x00000000ac8a3fd0:  0x0000000000000000  0x0000000000000000                                                                          0x00000000ac8a3fe0:  0x0000000000000000  0x00000000ac8a4008
0x00000000ac8a3ff0:  0x000000000004e1fc <runtime.sigpanic+0x00000000000001bc>  0x0000000000075700                                    0x00000000ac8a4000:  0x00000000000d1360  0x0000000000000000
0x00000000ac8a4010: <0x0000000000000000  0x0000000000000000                                                                          0x00000000ac8a4020:  0x0000000000000000  0x00000000ac8a4048
0x00000000ac8a4030:  0x00000000ac0021a0  0x0000000000000000                                                                          0x00000000ac8a4040: >0x0000000000000000  0x00000000ac8a4068
0x00000000ac8a4050:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000                                          0x00000000ac8a4060:  0x0000000000000000  0x00000000ac8a4088
0x00000000ac8a4070:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000                                          0x00000000ac8a4080:  0x0000000000000000  0x00000000ac8a40a8
0x00000000ac8a4090:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000                                          0x00000000ac8a40a0:  0x0000000000000000  0x00000000ac8a40c8                                                                          0x00000000ac8a40b0:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000                                          0x00000000ac8a40c0:  0x0000000000000000  0x00000000ac8a40e8
0x00000000ac8a40d0:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000                                          0x00000000ac8a40e0:  0x0000000000000000  0x00000000ac8a4108
0x00000000ac8a40f0:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a4100:  0x0000000000000000  0x00000000ac8a4128
0x00000000ac8a4110:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a4120:  0x0000000000000000  0x00000000ac8a4148
0x00000000ac8a4130:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x7ccb9?, 0xd08c0?})
        /usr/local64/go/src/runtime/panic.go:992 +0x50
runtime.gentraceback(0xb3df2338?, 0xb3df1fe3?, 0x40?, 0xac0021a0, 0x0, 0x0, 0x7fffffff, 0x80ccd860, 0x200?, 0x0)
        /usr/local64/go/src/runtime/traceback.go:254 +0x1318
runtime.addOneOpenDeferFrame.func1()
        /usr/local64/go/src/runtime/panic.go:599 +0x68
runtime.systemstack()
        /usr/local64/go/src/runtime/asm_arm64.s:237 +0x6c

goroutine 1 [running]:
runtime.systemstack_switch()
        /usr/local64/go/src/runtime/asm_arm64.s:194 +0x8 fp=0xac8a3ef0 sp=0xac8a3ee0 pc=0x5ed08
runtime.addOneOpenDeferFrame(0x400000?, 0x0?, 0x0?)
        /usr/local64/go/src/runtime/panic.go:598 +0x68 fp=0xac8a3f30 sp=0xac8a3ef0 pc=0x38a78
panic({0x75700, 0xd1360})
        /usr/local64/go/src/runtime/panic.go:798 +0xf0 fp=0xac8a3ff0 sp=0xac8a3f30 pc=0x392b0
runtime.panicmem()
        /usr/local64/go/src/runtime/panic.go:220 +0x54 fp=0xac8a4010 sp=0xac8a3ff0 pc=0x37f14
runtime: unexpected return pc for runtime.sigpanic called from 0x0
stack: frame={sp:0xac8a4010, fp:0xac8a4040} stack=[0xac700000,0xacb00000)
0x00000000ac8a3f10:  0x00000000ac8a3ff0  0x00000000ac0021a0
0x00000000ac8a3f20:  0x0000000000000000  0x00000000ac8a3fe8
0x00000000ac8a3f30:  0x0000000000037f14 <runtime.panicmem+0x0000000000000054>  0x0000000000400000
0x00000000ac8a3f40:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f50:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f60:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f70:  0x0000000000000000  0x00000000ac0021a0
0x00000000ac8a3f80:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3f90:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3fa0:  0x0000000000000000  0x00000000ac0021c0
0x00000000ac8a3fb0:  0x0000000000000000  0x0000000000075700
0x00000000ac8a3fc0:  0x00000000000d1360  0x0000000000000000
0x00000000ac8a3fd0:  0x0000000000000000  0x0000000000000000
0x00000000ac8a3fe0:  0x0000000000000000  0x00000000ac8a4008
0x00000000ac8a3ff0:  0x000000000004e1fc <runtime.sigpanic+0x00000000000001bc>  0x0000000000075700
0x00000000ac8a4000:  0x00000000000d1360  0x0000000000000000
0x00000000ac8a4010: <0x0000000000000000  0x0000000000000000
0x00000000ac8a4020:  0x0000000000000000  0x00000000ac8a4048
0x00000000ac8a4030:  0x00000000ac0021a0  0x0000000000000000
0x00000000ac8a4040: >0x0000000000000000  0x00000000ac8a4068
0x00000000ac8a4050:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a4060:  0x0000000000000000  0x00000000ac8a4088
0x00000000ac8a4070:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a4080:  0x0000000000000000  0x00000000ac8a40a8
0x00000000ac8a4090:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a40a0:  0x0000000000000000  0x00000000ac8a40c8
0x00000000ac8a40b0:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a40c0:  0x0000000000000000  0x00000000ac8a40e8
0x00000000ac8a40d0:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a40e0:  0x0000000000000000  0x00000000ac8a4108
0x00000000ac8a40f0:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a4100:  0x0000000000000000  0x00000000ac8a4128
0x00000000ac8a4110:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
0x00000000ac8a4120:  0x0000000000000000  0x00000000ac8a4148
0x00000000ac8a4130:  0x00000000000610e8 <main.count+0x0000000000000028>  0x0000000000000000
runtime.sigpanic()
        /usr/local64/go/src/runtime/signal_unix.go:818 +0x1bc fp=0xac8a4040 sp=0xac8a4010 pc=0x4e1fc
exit status 2