search

tianon / gosu

Simple Go-based setuid+setgid+setgroups+exec
Apache License 2.0
4.68k stars 312 forks source link

Use QEMU and "arch-test" to avoid bad binaries in the future #121

Closed tianon closed 1 year ago

tianon commented 1 year ago

As now noted on https://github.com/tianon/gosu/releases/tag/1.15, our new releases are affected by https://github.com/golang/go/issues/56426, which is broken in Go 1.18+ and not fixed until Go 1.20. :disappointed:

This is an attempt to pre-test for that via QEMU user-mode emulation and actually testing our end binaries (because this reproduces the failure).

Another failure that I think this is also going to run into is that from Go 1.18+ it seems our binaries now often hang in QEMU user-mode emulation (most reliably on ARM), so that's fun.

tianon commented 1 year ago

There's the QEMU bug :sob:

https://github.com/tianon/gosu/actions/runs/3736167285/jobs/6340182931#step:4:457

(it hangs indefinitely in whatever this failure mode is, hence all the timeout invocations)

tianon commented 1 year ago

There we go, there's our MIPS failure! https://github.com/tianon/gosu/actions/runs/3736241126/jobs/6340334082#step:4:10830

Long Silly Backtrace: ```console + timeout 1s /go/bin/gosu-mips64el nobody id trap:5114, a123456=[0,0,0,0,0,0] trap:5114, a123456=[0,0,0,0,0,0] results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0 fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted goroutine 0 [idle]: results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0 fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008 stack: frame={sp:0xc00003fc90, fp:0xc00003fcd8} stack=[0xc000038000,0xc000040000) 0x000000c00003fb90: 0x0000000000000000 0x0000000000000000 0x000000c00003fba0: 0x0000000000000000 0x0000000000000000 0x000000c00003fbb0: 0x0000000000000000 0x0000000000000000 0x000000c00003fbc0: 0x000000000006c3dc 0x0000000000000000 0x000000c00003fbd0: 0x0000000000000000 0x000000c00003fc28 0x000000c00003fbe0: 0x000000000006c258 0x0000000000000021 0x000000c00003fbf0: 0x000000c000002340 0x0000000000000000 0x000000c00003fc00: 0x0000000000000000 0x0000000000000000 0x000000c00003fc10: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fc20: 0x000000000008cf74 0x000000c000000021 0x000000c00003fc30: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fc40: 0x000000c0000024e0 0x0000000000000000 0x000000c00003fc50: 0x0000000000000000 0x0000000000000000 0x000000c00003fc60: 0x0000000000000000 0x0000000000000000 0x000000c00003fc70: 0x0000000000000000 0x000000c0000024e0 0x000000c00003fc80: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fc90: <0x0000004001001008 0x0000000000000021 0x000000c00003fca0: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fcb0: 0x0000000000000000 0x0000000000000000 0x000000c00003fcc0: 0x0000000000000000 0x0000000000000000 0x000000c00003fcd0: 0x0000000000000000 >0x0000000000000000 0x000000c00003fce0: 0x0000000000000000 0x0000000000000000 0x000000c00003fcf0: 0xfffffffa00000021 0x0000000000000000 0x000000c00003fd00: 0x00000000000002a0 0x0000000000000000 0x000000c00003fd10: 0x0000000000000000 0x0000000000000000 0x000000c00003fd20: 0x0000000000000000 0x0000000000000000 0x000000c00003fd30: 0x0000000000000000 0x0000000000000000 0x000000c00003fd40: 0x0000000000000000 0x0000000000000000 0x000000c00003fd50: 0x0000000000000000 0x0000000000000000 0x000000c00003fd60: 0x0000000000000000 0x0000000000000000 0x000000c00003fd70: 0x0000000000000000 0x0000000000000000 0x000000c00003fd80: 0x000000c000038000 0x0000000000008000 0x000000c00003fd90: 0x0000000000000000 0x0000000000000000 0x000000c00003fda0: 0x000000c000045ee8 0x0000000000000004 0x000000c00003fdb0: 0x000000003b9aca00 0x000000000026ec38 0x000000c00003fdc0: 0x0000000000000080 0x0000000000000000 0x000000c00003fdd0: 0x0000000000000001 runtime.throw({0x183f3e, 0x44}) /usr/local/go/src/runtime/panic.go:992 +0x6c runtime.runPerThreadSyscall() /usr/local/go/src/runtime/os_linux.go:874 +0x410 runtime.sighandler(0x21, 0xc00003fcf0, 0xc00003fd70, 0xc0000024e0) /usr/local/go/src/runtime/signal_unix.go:631 +0x770 runtime.sigtrampgo(0x21, 0xc00003fcf0, 0xc00003fd70) /usr/local/go/src/runtime/signal_unix.go:477 +0x228 runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008 stack: frame={sp:0xc00003fc90, fp:0xc00003fcd8} stack=[0xc000038000,0xc000040000) 0x000000c00003fb90: 0x0000000000000000 0x0000000000000000 0x000000c00003fba0: 0x0000000000000000 0x0000000000000000 0x000000c00003fbb0: 0x0000000000000000 0x0000000000000000 0x000000c00003fbc0: 0x000000000006c3dc 0x0000000000000000 0x000000c00003fbd0: 0x0000000000000000 0x000000c00003fc28 0x000000c00003fbe0: 0x000000000006c258 0x0000000000000021 0x000000c00003fbf0: 0x000000c000002340 0x0000000000000000 0x000000c00003fc00: 0x0000000000000000 0x0000000000000000 0x000000c00003fc10: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fc20: 0x000000000008cf74 0x000000c000000021 0x000000c00003fc30: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fc40: 0x000000c0000024e0 0x0000000000000000 0x000000c00003fc50: 0x0000000000000000 0x0000000000000000 0x000000c00003fc60: 0x0000000000000000 0x0000000000000000 0x000000c00003fc70: 0x0000000000000000 0x000000c0000024e0 0x000000c00003fc80: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fc90: <0x0000004001001008 0x0000000000000021 0x000000c00003fca0: 0x000000c00003fcf0 0x000000c00003fd70 0x000000c00003fcb0: 0x0000000000000000 0x0000000000000000 0x000000c00003fcc0: 0x0000000000000000 0x0000000000000000 0x000000c00003fcd0: 0x0000000000000000 >0x0000000000000000 0x000000c00003fce0: 0x0000000000000000 0x0000000000000000 0x000000c00003fcf0: 0xfffffffa00000021 0x0000000000000000 0x000000c00003fd00: 0x00000000000002a0 0x0000000000000000 0x000000c00003fd10: 0x0000000000000000 0x0000000000000000 0x000000c00003fd20: 0x0000000000000000 0x0000000000000000 0x000000c00003fd30: 0x0000000000000000 0x0000000000000000 0x000000c00003fd40: 0x0000000000000000 0x0000000000000000 0x000000c00003fd50: 0x0000000000000000 0x0000000000000000 0x000000c00003fd60: 0x0000000000000000 0x0000000000000000 0x000000c00003fd70: 0x0000000000000000 0x0000000000000000 0x000000c00003fd80: 0x000000c000038000 0x0000000000008000 0x000000c00003fd90: 0x0000000000000000 0x0000000000000000 0x000000c00003fda0: 0x000000c000045ee8 0x0000000000000004 0x000000c00003fdb0: 0x000000003b9aca00 0x000000000026ec38 0x000000c00003fdc0: 0x0000000000000080 0x0000000000000000 0x000000c00003fdd0: 0x0000000000000001 runtime.sigtramp() /usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54 trap:5114, a123456=[0,0,0,0,0,0] results: got {r1=0,r2=5114,errno=0}, want {r1=0,r2=0,errno=0 fatal error: AllThreadsSyscall6 results differ between threads; runtime corrupted goroutine 1 [running, locked to thread]: goroutine running on other thread; stack unavailable goroutine 0 [idle]: runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008 stack: frame={sp:0xc000057c90, fp:0xc000057cd8} stack=[0xc000050000,0xc000058000) 0x000000c000057b90: 0x0000000000000000 0x0000000000000000 0x000000c000057ba0: 0x0000000000000000 0x0000000000000000 0x000000c000057bb0: 0x0000000000000000 0x0000000000000000 0x000000c000057bc0: 0x000000000006c3dc 0x0000000000000000 0x000000c000057bd0: 0x0000000000000000 0x000000c000057c28 0x000000c000057be0: 0x000000000006c258 0x0000000000000021 0x000000c000057bf0: 0x000000c000002b60 0x0000000000000000 0x000000c000057c00: 0x0000000000000000 0x0000000000000000 0x000000c000057c10: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057c20: 0x000000000008cf74 0x000000c000000021 0x000000c000057c30: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057c40: 0x000000c000002d00 0x0000000000000000 0x000000c000057c50: 0x0000000000000000 0x0000000000000000 0x000000c000057c60: 0x0000000000000000 0x0000000000000000 0x000000c000057c70: 0x0000000000000000 0x000000c000002d00 0x000000c000057c80: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057c90: <0x0000004001001008 0x0000000000000021 0x000000c000057ca0: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057cb0: 0x0000000000000000 0x0000000000000000 0x000000c000057cc0: 0x0000000000000000 0x0000000000000000 0x000000c000057cd0: 0x0000000000000000 >0x0000000000000000 0x000000c000057ce0: 0x0000000000000000 0x0000000000000000 0x000000c000057cf0: 0xfffffffa00000021 0x0000000000000000 0x000000c000057d00: 0x00000000000002a0 0x0000000000000000 0x000000c000057d10: 0x0000000000000000 0x0000000000000000 0x000000c000057d20: 0x0000000000000000 0x0000000000000000 0x000000c000057d30: 0x0000000000000000 0x0000000000000000 0x000000c000057d40: 0x0000000000000000 0x0000000000000000 0x000000c000057d50: 0x0000000000000000 0x0000000000000000 0x000000c000057d60: 0x0000000000000000 0x0000000000000000 0x000000c000057d70: 0x0000000000000000 0x0000000000000000 0x000000c000057d80: 0x000000c000050000 0x0000000000008000 0x000000c000057d90: 0x0000000000000000 0x0000000000000000 0x000000c000057da0: 0x0000000000000000 0x000000000000144a 0x000000c000057db0: 0x000000c000002d00 0x000000c000036950 0x000000c000057dc0: 0x0000000000000080 0x0000000000000000 0x000000c000057dd0: 0x0000000000000000 runtime.throw({0x183f3e, 0x44}) /usr/local/go/src/runtime/panic.go:992 +0x6c runtime.runPerThreadSyscall() /usr/local/go/src/runtime/os_linux.go:874 +0x410 runtime.sighandler(0x21, 0xc000057cf0, 0xc000057d70, 0xc000002d00) /usr/local/go/src/runtime/signal_unix.go:631 +0x770 runtime.sigtrampgo(0x21, 0xc000057cf0, 0xc000057d70) /usr/local/go/src/runtime/signal_unix.go:477 +0x228 runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008 stack: frame={sp:0xc000057c90, fp:0xc000057cd8} stack=[0xc000050000,0xc000058000) 0x000000c000057b90: 0x0000000000000000 0x0000000000000000 0x000000c000057ba0: 0x0000000000000000 0x0000000000000000 0x000000c000057bb0: 0x0000000000000000 0x0000000000000000 0x000000c000057bc0: 0x000000000006c3dc 0x0000000000000000 0x000000c000057bd0: 0x0000000000000000 0x000000c000057c28 0x000000c000057be0: 0x000000000006c258 0x0000000000000021 0x000000c000057bf0: 0x000000c000002b60 0x0000000000000000 0x000000c000057c00: 0x0000000000000000 0x0000000000000000 0x000000c000057c10: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057c20: 0x000000000008cf74 0x000000c000000021 0x000000c000057c30: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057c40: 0x000000c000002d00 0x0000000000000000 0x000000c000057c50: 0x0000000000000000 0x0000000000000000 0x000000c000057c60: 0x0000000000000000 0x0000000000000000 0x000000c000057c70: 0x0000000000000000 0x000000c000002d00 0x000000c000057c80: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057c90: <0x0000004001001008 0x0000000000000021 0x000000c000057ca0: 0x000000c000057cf0 0x000000c000057d70 0x000000c000057cb0: 0x0000000000000000 0x0000000000000000 0x000000c000057cc0: 0x0000000000000000 0x0000000000000000 0x000000c000057cd0: 0x0000000000000000 >0x0000000000000000 0x000000c000057ce0: 0x0000000000000000 0x0000000000000000 0x000000c000057cf0: 0xfffffffa00000021 0x0000000000000000 0x000000c000057d00: 0x00000000000002a0 0x0000000000000000 0x000000c000057d10: 0x0000000000000000 0x0000000000000000 0x000000c000057d20: 0x0000000000000000 0x0000000000000000 0x000000c000057d30: 0x0000000000000000 0x0000000000000000 0x000000c000057d40: 0x0000000000000000 0x0000000000000000 0x000000c000057d50: 0x0000000000000000 0x0000000000000000 0x000000c000057d60: 0x0000000000000000 0x0000000000000000 0x000000c000057d70: 0x0000000000000000 0x0000000000000000 0x000000c000057d80: 0x000000c000050000 0x0000000000008000 0x000000c000057d90: 0x0000000000000000 0x0000000000000000 0x000000c000057da0: 0x0000000000000000 0x000000000000144a 0x000000c000057db0: 0x000000c000002d00 0x000000c000036950 0x000000c000057dc0: 0x0000000000000080 0x0000000000000000 0x000000c000057dd0: 0x0000000000000000 runtime.sigtramp() /usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54 goroutine 0 [idle]: runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008 stack: frame={sp:0xc000061c90, fp:0xc000061cd8} stack=[0xc00005a000,0xc000062000) 0x000000c000061b90: 0x0000000000000000 0x0000000000000000 0x000000c000061ba0: 0x0000000000000000 0x0000000000000000 0x000000c000061bb0: 0x0000000000000000 0x0000000000000000 0x000000c000061bc0: 0x000000000006c3dc 0x0000000000000000 0x000000c000061bd0: 0x0000000000000000 0x000000c000061c28 0x000000c000061be0: 0x000000000006c258 0x0000000000000021 0x000000c000061bf0: 0x000000c000002ea0 0x0000000000000000 0x000000c000061c00: 0x0000000000000000 0x0000000000000000 0x000000c000061c10: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061c20: 0x000000000008cf74 0x000000c000000021 0x000000c000061c30: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061c40: 0x000000c000003040 0x0000000000000000 0x000000c000061c50: 0x0000000000000000 0x0000000000000000 0x000000c000061c60: 0x0000000000000000 0x0000000000000000 0x000000c000061c70: 0x0000000000000000 0x000000c000003040 0x000000c000061c80: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061c90: <0x0000004001001008 0x0000000000000021 0x000000c000061ca0: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061cb0: 0x0000000000000000 0x0000000000000000 0x000000c000061cc0: 0x0000000000000000 0x0000000000000000 0x000000c000061cd0: 0x0000000000000000 >0x0000000000000000 0x000000c000061ce0: 0x0000000000000000 0x0000000000000000 0x000000c000061cf0: 0xfffffffa00000021 0x0000000000000000 0x000000c000061d00: 0x00000000000002a0 0x0000000000000000 0x000000c000061d10: 0x0000000000000000 0x0000000000000000 0x000000c000061d20: 0x0000000000000000 0x0000000000000000 0x000000c000061d30: 0x0000000000000000 0x0000000000000000 0x000000c000061d40: 0x0000000000000000 0x0000000000000000 0x000000c000061d50: 0x0000000000000000 0x0000000000000000 0x000000c000061d60: 0x0000000000000000 0x0000000000000000 0x000000c000061d70: 0x0000000000000000 0x0000000000000000 0x000000c000061d80: 0x000000c00005a000 0x0000000000008000 0x000000c000061d90: 0x0000000000000000 0x0000000000000000 0x000000c000061da0: 0x0000000000000000 0x000000000000144a 0x000000c000061db0: 0x000000c000003040 0x000000000029b098 0x000000c000061dc0: 0x0000000000000080 0x0000000000000000 0x000000c000061dd0: 0x0000000000000000 runtime.throw({0x183f3e, 0x44}) /usr/local/go/src/runtime/panic.go:992 +0x6c runtime.runPerThreadSyscall() /usr/local/go/src/runtime/os_linux.go:874 +0x410 runtime.sighandler(0x21, 0xc000061cf0, 0xc000061d70, 0xc000003040) /usr/local/go/src/runtime/signal_unix.go:631 +0x770 runtime.sigtrampgo(0x21, 0xc000061cf0, 0xc000061d70) /usr/local/go/src/runtime/signal_unix.go:477 +0x228 runtime: unexpected return pc for runtime.sigtramp called from 0x4001001008 stack: frame={sp:0xc000061c90, fp:0xc000061cd8} stack=[0xc00005a000,0xc000062000) 0x000000c000061b90: 0x0000000000000000 0x0000000000000000 0x000000c000061ba0: 0x0000000000000000 0x0000000000000000 0x000000c000061bb0: 0x0000000000000000 0x0000000000000000 0x000000c000061bc0: 0x000000000006c3dc 0x0000000000000000 0x000000c000061bd0: 0x0000000000000000 0x000000c000061c28 0x000000c000061be0: 0x000000000006c258 0x0000000000000021 0x000000c000061bf0: 0x000000c000002ea0 0x0000000000000000 0x000000c000061c00: 0x0000000000000000 0x0000000000000000 0x000000c000061c10: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061c20: 0x000000000008cf74 0x000000c000000021 0x000000c000061c30: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061c40: 0x000000c000003040 0x0000000000000000 0x000000c000061c50: 0x0000000000000000 0x0000000000000000 0x000000c000061c60: 0x0000000000000000 0x0000000000000000 0x000000c000061c70: 0x0000000000000000 0x000000c000003040 0x000000c000061c80: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061c90: <0x0000004001001008 0x0000000000000021 0x000000c000061ca0: 0x000000c000061cf0 0x000000c000061d70 0x000000c000061cb0: 0x0000000000000000 0x0000000000000000 0x000000c000061cc0: 0x0000000000000000 0x0000000000000000 0x000000c000061cd0: 0x0000000000000000 >0x0000000000000000 0x000000c000061ce0: 0x0000000000000000 0x0000000000000000 0x000000c000061cf0: 0xfffffffa00000021 0x0000000000000000 0x000000c000061d00: 0x00000000000002a0 0x0000000000000000 0x000000c000061d10: 0x0000000000000000 0x0000000000000000 0x000000c000061d20: 0x0000000000000000 0x0000000000000000 0x000000c000061d30: 0x0000000000000000 0x0000000000000000 0x000000c000061d40: 0x0000000000000000 0x0000000000000000 0x000000c000061d50: 0x0000000000000000 0x0000000000000000 0x000000c000061d60: 0x0000000000000000 0x0000000000000000 0x000000c000061d70: 0x0000000000000000 0x0000000000000000 0x000000c000061d80: 0x000000c00005a000 0x0000000000008000 0x000000c000061d90: 0x0000000000000000 0x0000000000000000 0x000000c000061da0: 0x0000000000000000 0x000000000000144a 0x000000c000061db0: 0x000000c000003040 0x000000000029b098 0x000000c000061dc0: 0x0000000000000080 0x0000000000000000 0x000000c000061dd0: 0x0000000000000000 runtime.sigtramp() /usr/local/go/src/runtime/sys_linux_mips64x.s:435 +0x54 ```
tianon commented 1 year ago

A nice benefit of container-based builds: we can just apply that upstream patch to our Go sources during build and the bug is now fixed. :joy:

yosifkit commented 8 months ago

The mips64le binary is also affected in the Bookworm package since it used Go 1.19 for building. 😢 https://packages.debian.org/bookworm/gosu https://packages.debian.org/bookworm/golang.

(Just to post this somewhere since fixing it in Debian Bookworm would be unlikely since it would need a patch applied to the golang package to fix it but also no one has really noticed 🙈)