runtime: crash on linux/arm64 during os.Setenv

[cbandy]

Go version

go1.22.6 linux/arm64

Output of go env in your module/workspace:

n/a

What did you do?

I was running an application test suite against an arm64 K3s built with go1.22.6:

$ docker run --rm --entrypoint=k3s docker.io/rancher/k3s:v1.31.1-k3s1 --version
k3s version v1.31.1+k3s1 (452dbbc1)
go version go1.22.6

What did you see happen?

kubelet reported that an exec probe failed with SIGSEGV:

Liveness probe failed: SIGSEGV: segmentation violation
…
        goroutine 1 gp=0x40000021c0 m=0 mp=0xffff80951580 [running, locked to thread]:
        runtime.systemstack_switch()
            runtime/asm_arm64.s:200 +0x8 fp=0x400018f980 sp=0x400018f970 pc=0xffff8013c228
        runtime.setenv_c({0x4000166190?, 0x1?}, {0x4000166195, 0x3c})
            runtime/env_posix.go:54 +0xb0 fp=0x400018f9e0 sp=0x400018f980 pc=0xffff800d6d10
        syscall.runtimeSetenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
            runtime/runtime.go:201 +0x2c fp=0x400018fa10 sp=0x400018f9e0 pc=0xffff80139c4c
        syscall.Setenv({0x4000166190, 0x4}, {0x4000166195, 0x3c})
            syscall/env_unix.go:122 +0x310 fp=0x400018fac0 sp=0x400018fa10 pc=0xffff80185870
        os.Setenv({0x4000166190?, 0x3c?}, {0x4000166195?, 0xffffffffffffffff?})
            os/env.go:120 +0x24 fp=0x400018fb00 sp=0x400018fac0 pc=0xffff801b3ea4
…

full message

``` Liveness probe failed: SIGSEGV: segmentation violation PC=0xffff804967e4 m=0 sigcode=1 addr=0x0 goroutine 0 gp=0xffff80950d00 m=0 mp=0xffff80951580 [idle]: runtime: g 0 gp=0xffff80950d00: unknown pc 0xffff804967e4 stack: frame={sp:0xffffe77cbef0, fp:0x0} stack=[0xffffe77ad190,0xffffe77cc270) 0x0000ffffe77cbdf0: 0x0000ffff800dce40 0x0000000000000000 0x0000ffffe77cbe00: 0xffffffffffffefe8 0x0000000000000080 0x0000ffffe77cbe10: 0xffffffffffffff80 0x0000ffff80951580 0x0000ffffe77cbe20: 0x0000004000035220 0x0000ffff3936eb80 0x0000ffffe77cbe30: 0xfffffffffffff800 0x0000ffffe77cbe68 0x0000ffffe77cbe40: 0x0000ffffe77cbe90 0x0000ffff80496388 0x0000ffffe77cbe50: 0x0000ffffe77cbe90 0x0000ffff80496710 0x0000ffffe77cbe60: 0x0000000000000002 0x0000aaaac2860130 0x0000ffffe77cbe70: 0x0000ffffe77cbe90 0x0000ffff80496710 0x0000ffffe77cbe80: 0x0000000000000002 0x0000ffff3936eb80 0x0000ffffe77cbe90: 0x0000ffffe77cbef0 0x0000ffff80496910 0x0000ffffe77cbea0: 0x0000000000000002 0x0000aaaac2860108 0x0000ffffe77cbeb0: 0x0000000000000023 0x0000ffff809b7000 0x0000ffffe77cbec0: 0x0000000000000023 0x0000000000003ff0 0x0000ffffe77cbed0: 0x0000000000000027 0x0000ffff8056d000 0x0000ffffe77cbee0: 0x0000000000000002 0x0000ffffe77cbf68 0x0000ffffe77cbef0: <0x0000ffffe77cbf50 0x0000ffff80496910 0x0000ffffe77cbf00: 0x0000000000000002 0x0000aaaac28600e0 0x0000ffffe77cbf10: 0x000000000000001f 0x0000ffff809b7000 0x0000ffffe77cbf20: 0x000000000000001f 0x0000000000001ff0 0x0000ffffe77cbf30: 0x0000000000000023 0x0000ffff8056d000 0x0000ffffe77cbf40: 0x0000000000000002 0x0000ffffe77cbfb8 0x0000ffffe77cbf50: 0x0000ffffe77cbfb0 0x0000ffff80496910 0x0000ffffe77cbf60: 0x0000000000000002 0x0000aaaac28600b8 0x0000ffffe77cbf70: 0x000000000000001b 0x0000ffff809b7000 0x0000ffffe77cbf80: 0x000000000000001b 0x0000000000000ff0 0x0000ffffe77cbf90: 0x000000000000001f 0x0000ffff8056d000 0x0000ffffe77cbfa0: 0x0000000000000002 0x0000ffffe77cbfd8 0x0000ffffe77cbfb0: 0x0000ffffe77cc010 0x0000ffff80496910 0x0000ffffe77cbfc0: 0x0000000000000002 0x0000aaaac2860090 0x0000ffffe77cbfd0: 0x0000000000000017 0x0000ffff809b7000 0x0000ffffe77cbfe0: 0x0000000000000017 0x00000000000007f0 runtime: g 0 gp=0xffff80950d00: unknown pc 0xffff804967e4 stack: frame={sp:0xffffe77cbef0, fp:0x0} stack=[0xffffe77ad190,0xffffe77cc270) 0x0000ffffe77cbdf0: 0x0000ffff800dce40 0x0000000000000000 0x0000ffffe77cbe00: 0xffffffffffffefe8 0x0000000000000080 0x0000ffffe77cbe10: 0xffffffffffffff80 0x0000ffff80951580 0x0000ffffe77cbe20: 0x0000004000035220 0x0000ffff3936eb80 0x0000ffffe77cbe30: 0xfffffffffffff800 0x0000ffffe77cbe68 0x0000ffffe77cbe40: 0x0000ffffe77cbe90 0x0000ffff80496388 0x0000ffffe77cbe50: 0x0000ffffe77cbe90 0x0000ffff80496710 0x0000ffffe77cbe60: 0x0000000000000002 0x0000aaaac2860130 0x0000ffffe77cbe70: 0x0000ffffe77cbe90 0x0000ffff80496710 0x0000ffffe77cbe80: 0x0000000000000002 0x0000ffff3936eb80 0x0000ffffe77cbe90: 0x0000ffffe77cbef0 0x0000ffff80496910 0x0000ffffe77cbea0: 0x0000000000000002 0x0000aaaac2860108 0x0000ffffe77cbeb0: 0x0000000000000023 0x0000ffff809b7000 0x0000ffffe77cbec0: 0x0000000000000023 0x0000000000003ff0 0x0000ffffe77cbed0: 0x0000000000000027 0x0000ffff8056d000 0x0000ffffe77cbee0: 0x0000000000000002 0x0000ffffe77cbf68 0x0000ffffe77cbef0: <0x0000ffffe77cbf50 0x0000ffff80496910 0x0000ffffe77cbf00: 0x0000000000000002 0x0000aaaac28600e0 0x0000ffffe77cbf10: 0x000000000000001f 0x0000ffff809b7000 0x0000ffffe77cbf20: 0x000000000000001f 0x0000000000001ff0 0x0000ffffe77cbf30: 0x0000000000000023 0x0000ffff8056d000 0x0000ffffe77cbf40: 0x0000000000000002 0x0000ffffe77cbfb8 0x0000ffffe77cbf50: 0x0000ffffe77cbfb0 0x0000ffff80496910 0x0000ffffe77cbf60: 0x0000000000000002 0x0000aaaac28600b8 0x0000ffffe77cbf70: 0x000000000000001b 0x0000ffff809b7000 0x0000ffffe77cbf80: 0x000000000000001b 0x0000000000000ff0 0x0000ffffe77cbf90: 0x000000000000001f 0x0000ffff8056d000 0x0000ffffe77cbfa0: 0x0000000000000002 0x0000ffffe77cbfd8 0x0000ffffe77cbfb0: 0x0000ffffe77cc010 0x0000ffff80496910 0x0000ffffe77cbfc0: 0x0000000000000002 0x0000aaaac2860090 0x0000ffffe77cbfd0: 0x0000000000000017 0x0000ffff809b7000 0x0000ffffe77cbfe0: 0x0000000000000017 0x00000000000007f0 goroutine 1 gp=0x40000021c0 m=0 mp=0xffff80951580 [running, locked to thread]: runtime.systemstack_switch() runtime/asm_arm64.s:200 +0x8 fp=0x400018f980 sp=0x400018f970 pc=0xffff8013c228 runtime.setenv_c({0x4000166190?, 0x1?}, {0x4000166195, 0x3c}) runtime/env_posix.go:54 +0xb0 fp=0x400018f9e0 sp=0x400018f980 pc=0xffff800d6d10 syscall.runtimeSetenv({0x4000166190, 0x4}, {0x4000166195, 0x3c}) runtime/runtime.go:201 +0x2c fp=0x400018fa10 sp=0x400018f9e0 pc=0xffff80139c4c syscall.Setenv({0x4000166190, 0x4}, {0x4000166195, 0x3c}) syscall/env_unix.go:122 +0x310 fp=0x400018fac0 sp=0x400018fa10 pc=0xffff80185870 os.Setenv({0x4000166190?, 0x3c?}, {0x4000166195?, 0xffffffffffffffff?}) os/env.go:120 +0x24 fp=0x400018fb00 sp=0x400018fac0 pc=0xffff801b3ea4 github.com/opencontainers/runc/libcontainer.populateProcessEnvironment({0x400017a200?, 0xffff805b5440?, 0x400004a0f8?}) github.com/opencontainers/runc/libcontainer/init_linux.go:138 +0xe0 fp=0x400018fb80 sp=0x400018fb00 pc=0xffff80401190 github.com/opencontainers/runc/libcontainer.newContainerInit({0x4000028017, 0x5}, 0x400004a0f0, 0x0, 0xffffffffffffffff, 0x4, {0x0, 0x0, 0x0}) github.com/opencontainers/runc/libcontainer/init_linux.go:85 +0xb4 fp=0x400018fbe0 sp=0x400018fb80 pc=0xffff80400da4 github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0x0?) github.com/opencontainers/runc/libcontainer/factory_linux.go:334 +0x3d0 fp=0x400018fdc0 sp=0x400018fbe0 pc=0xffff80400310 main.init.0() github.com/opencontainers/runc/init.go:36 +0x194 fp=0x400018fe10 sp=0x400018fdc0 pc=0xffff8047c7b4 runtime.doInit1(0xffff809399e0) runtime/proc.go:7176 +0xd4 fp=0x400018ff40 sp=0x400018fe10 pc=0xffff80119a44 runtime.doInit(...) runtime/proc.go:7143 runtime.main() runtime/proc.go:253 +0x344 fp=0x400018ffd0 sp=0x400018ff40 pc=0xffff80109bf4 runtime.goexit({}) runtime/asm_arm64.s:1222 +0x4 fp=0x400018ffd0 sp=0x400018ffd0 pc=0xffff8013e754 goroutine 2 gp=0x4000002c40 m=nil [force gc (idle)]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:402 +0xc8 fp=0x4000046f90 sp=0x4000046f70 pc=0xffff80109f68 runtime.goparkunlock(...) runtime/proc.go:408 runtime.forcegchelper() runtime/proc.go:326 +0xb8 fp=0x4000046fd0 sp=0x4000046f90 pc=0xffff80109df8 runtime.goexit({}) runtime/asm_arm64.s:1222 +0x4 fp=0x4000046fd0 sp=0x4000046fd0 pc=0xffff8013e754 created by runtime.init.6 in goroutine 1 runtime/proc.go:314 +0x24 goroutine 3 gp=0x4000003180 m=nil [GC sweep wait]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) runtime/proc.go:402 +0xc8 fp=0x4000047760 sp=0x4000047740 pc=0xffff80109f68 runtime.goparkunlock(...) runtime/proc.go:408 runtime.bgsweep(0x400006e000) runtime/mgcsweep.go:278 +0xa0 fp=0x40000477b0 sp=0x4000047760 pc=0xffff800f54a0 runtime.gcenable.gowrap1() runtime/mgc.go:203 +0x28 fp=0x40000477d0 sp=0x40000477b0 pc=0xffff800e96b8 runtime.goexit({}) runtime/asm_arm64.s:1222 +0x4 fp=0x40000477d0 sp=0x40000477d0 pc=0xffff8013e754 created by runtime.gcenable in goroutine 1 runtime/mgc.go:203 +0x6c goroutine 4 gp=0x4000003340 m=nil [GC scavenge wait]: runtime.gopark(0x400006e000?, 0xffff80555fe8?, 0x1?, 0x0?, 0x4000003340?) runtime/proc.go:402 +0xc8 fp=0x4000047f60 sp=0x4000047f40 pc=0xffff80109f68 runtime.goparkunlock(...) runtime/proc.go:408 runtime.(*scavengerState).park(0xffff80950a20) runtime/mgcscavenge.go:425 +0x5c fp=0x4000047f90 sp=0x4000047f60 pc=0xffff800f2e9c runtime.bgscavenge(0x400006e000) runtime/mgcscavenge.go:653 +0x44 fp=0x4000047fb0 sp=0x4000047f90 pc=0xffff800f33e4 runtime.gcenable.gowrap2() runtime/mgc.go:204 +0x28 fp=0x4000047fd0 sp=0x4000047fb0 pc=0xffff800e9658 runtime.goexit({}) runtime/asm_arm64.s:1222 +0x4 fp=0x4000047fd0 sp=0x4000047fd0 pc=0xffff8013e754 created by runtime.gcenable in goroutine 1 runtime/mgc.go:204 +0xac goroutine 5 gp=0x4000003c00 m=nil [finalizer wait]: runtime.gopark(0x0?, 0x0?, 0xb8?, 0x65?, 0xffff80139b64?) runtime/proc.go:402 +0xc8 fp=0x4000046580 sp=0x4000046560 pc=0xffff80109f68 runtime.runfinq() runtime/mfinal.go:194 +0x108 fp=0x40000467d0 sp=0x4000046580 pc=0xffff800e8788 runtime.goexit({}) runtime/asm_arm64.s:1222 +0x4 fp=0x40000467d0 sp=0x40000467d0 pc=0xffff8013e754 created by runtime.createfing in goroutine 1 runtime/mfinal.go:164 +0x80 r0 0x0 r1 0x50 r2 0x0 r3 0x0 r4 0x2e r5 0xffff394e0010 r6 0xaaaac2860130 r7 0x7fec r8 0x0 r9 0xffff394e7ffc r10 0x2f7273752f3a6e69 r11 0x73752f3a6e696273 r12 0x732f3a6e69622f72 r13 0xffffe77cc220 r14 0x2f r15 0x30 r16 0x80 r17 0x400018f920 r18 0x0 r19 0x2 r20 0xaaaac2860108 r21 0x23 r22 0xffff809b7000 r23 0x23 r24 0x3ff0 r25 0x27 r26 0xffff8056dd20 r27 0x2 r28 0xffff80950d00 r29 0xffffe77cbef0 lr 0xffff80496910 sp 0xffffe77cbef0 pc 0xffff804967e4 fault 0x0 ```

What did you expect to see?

No crash.

Perhaps the issue discussed in #59692 can also happen on arm64?

[gabyhelp]

Related Issues and Documentation

• runtime: misc/cgo/test.TestSetgid panic with `unexpected return pc for runtime.sigpanic` on linux-arm64-packet since 2022-06-04 #53374 (closed)
• runtime: segmentation violation runtime.(*unwinder).next #66151 (closed)
• runtime: linux/arm64 crash in runtime.sigtrampgo #32912 (closed)
• fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x2 addr=0xc002000000 pc=0x46e698] Linux version 5.10.135-yocto-standard #58377 (closed)
• cmd/go: get in docker buildx gets SIGSEGV error #40969 (closed)
• runtime: fatal error: unexpected signal during runtime execution #60783 (closed)
• runtime: frequent TestSegv failures on linux-mips64le-mengzhuo #50605 (closed)
• runtime: `TestSegv/Segv` failure with `unexpected return pc` on `openbsd-386-68` #52043
• runtime #60196 (closed)
• runtime/cgo: SIGSEGV: segmentation violation #6051 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cherrymui]

What is your host machine? Are you running docker as a VM?

It looks like this is crashing in the C setenv function. What are your libc and C toolchain? Thanks.

[mateusz834]

See #63567

[mpx]

As mentioned elsewhere (#63567), libc setenv(3) is not threadsafe - it's a fundamental flaw. Quite a few different libc calls can access the environment in parallel to setenv(3) and cause a segfault (eg, getaddrinfo(3)).

Some options to avoid the libc segfault:

• Avoid libc by disabling CGO (CGO_ENABLE=0 during compilation)
• Ensure your program never modifies the environment once there are multiple goroutines (eg, limit changes to func init).