search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.08k stars 17.68k forks source link

crypto/x509: panic during tls handshake from http client #70374

Open fishy opened 4 hours ago

fishy commented 4 hours ago

Go version

go1.23.2 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/fishy/.cache/go-build'
GOENV='/home/fishy/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/fishy/.gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/fishy/.gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go-1.23'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go-1.23/pkg/tool/linux_amd64'
GOVCS=''
// GOVERSION='go1.23.3' The panic happened from a binary built on go1.23.2
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/fishy/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/fishy/work/ddporkbun/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2780862528=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I have some simple go code to update porkbun dynamic dns: https://github.com/fishy/ddporkbun. The binary is run through an hourly cronjob.

The binary info is (it was built by go1.23.2):

$ go version -m /home/fishy/.gopath/bin/ddporkbun
/home/fishy/.gopath/bin/ddporkbun: go1.23.2
        path    go.yhsif.com/ddporkbun
        mod     go.yhsif.com/ddporkbun  (devel)
        build   -buildmode=exe
        build   -compiler=gc
        build   DefaultGODEBUG=asynctimerchan=1,gotypesalias=0,httplaxcontentlength=1,httpmuxgo121=1,httpservecontentkeepheaders=1,tls10server=1,tls3des=1,tlskyber=0,tlsrsakex=1,tlsunsafeekm=1,winreadlinkvolume=0,winsymlink=0,x509keypairleaf=0,x509negativeserial=1
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=linux
        build   GOAMD64=v1
        build   vcs=git
        build   vcs.revision=6fc89a0b0a453a8dc12405982dcd5a6604f56bcd
        build   vcs.time=2023-09-07T00:00:10Z
        build   vcs.modified=false

In one run it panicked during http tls handshake (note: the call stack here is copied directly from /var/spool/mail/$USERNAME so there are some extra line wrapping and email encoding around line wrapping):

/etc/cron.hourly/porkbun:
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x473301]

goroutine 23 gp=0xc000084a80 m=4 mp=0xc000080008 [running]:
runtime.throw({0x705224?, 0xc000080008?})
    /usr/lib/go-1.23/src/runtime/panic.go:1067 +0x48 fp=0xc000092fe8 sp=0xc000092fb8 pc=0x46aa48
runtime.sigpanic()
    /usr/lib/go-1.23/src/runtime/signal_unix.go:914 +0x26c fp=0xc000093048 sp=0xc000092fe8 pc=0x46c72c
runtime.memmove()
    /usr/lib/go-1.23/src/runtime/memmove_amd64.s:184 +0x141 fp=0xc000093050 sp=0xc000093048 pc=0x473301
runtime.concatstrings(0x0?, {0xc0000930e0?, 0x3, 0xa716d276cf47951c?})
    /usr/lib/go-1.23/src/runtime/string.go:53 +0x178 fp=0xc0000930c0 sp=0xc000093050 pc=0x451098
runtime.concatstring3(0xc000160390?, {0x70790f?, 0x682e6ff35b9cca4f?}, {0x704d1d?, 0xe0b8e411b6cb9703?}, {0x800000c0000d3290?, 0x8cc7020884c87814?})
    /usr/lib/go-1.23/src/runtime/string.go:64 +0x65 fp=0xc000093120 sp=0xc0000930c0 pc=0x451245
crypto/x509.loadSystemRoots()
    /usr/lib/go-1.23/src/crypto/x509/root_unix.go:70 +0x388 fp=0xc000093200 sp=0xc000093120 pc=0x5c0a88
crypto/x509.initSystemRoots()
    /usr/lib/go-1.23/src/crypto/x509/root.go:40 +0x56 fp=0xc000093238 sp=0xc000093200 pc=0x5c05f6
sync.(*Once).doSlow(0x7f694cfc55b8?, 0xa0?)
    /usr/lib/go-1.23/src/sync/once.go:76 +0xb4 fp=0xc000093298 sp=0xc000093238 pc=0x47b834
sync.(*Once).Do(...)
    /usr/lib/go-1.23/src/sync/once.go:67
crypto/x509.systemRootsPool()
    /usr/lib/go-1.23/src/crypto/x509/root.go:31 +0x45 fp=0xc0000932e8 sp=0xc000093298 pc=0x5c04a5
crypto/x509.(*Certificate).Verify(0xc0000de588, {{0xc0000d2090, 0x14}, 0xc0001602d0, 0x0, {0xc1c54e9b8a9d6ba0, 0x35475ba6, 0x953100}, {0x0, 0x0, ...}, ...})
    /usr/lib/go-1.23/src/crypto/x509/verify.go:789 +0x15d fp=0xc000093428 sp=0xc0000932e8 pc=0x5c3efd
crypto/tls.(*Conn).verifyServerCertificate(0xc0000b8a88, {0xc0000c3b00, 0x4, 0x4})
    /usr/lib/go-1.23/src/crypto/tls/handshake_client.go:1128 +0xaa5 fp=0xc0000936b8 sp=0xc000093428 pc=0x5f9b85
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc0000b1520)
    /usr/lib/go-1.23/src/crypto/tls/handshake_client.go:685 +0x271 fp=0xc0000938d0 sp=0xc0000936b8 pc=0x5f7031
crypto/tls.(*clientHandshakeState).handshake(0xc0000b1520)
    /usr/lib/go-1.23/src/crypto/tls/handshake_client.go:593 +0x3a6 fp=0xc000093ae0 sp=0xc0000938d0 pc=0x5f63c6
crypto/tls.(*Conn).clientHandshake(0xc0000b8a88, {0x78ab38, 0xc000108370})
    /usr/lib/go-1.23/src/crypto/tls/handshake_client.go:376 +0x933 fp=0xc000093d30 sp=0xc000093ae0 pc=0x5f4f53
crypto/tls.(*Conn).clientHandshake-fm({0x78ab38?, 0xc000108370?})
    <autogenerated>:1 +0x33 fp=0xc000093d58 sp=0xc000093d30 pc=0x61e653
crypto/tls.(*Conn).handshakeContext(0xc0000b8a88, {0x78ab38, 0xc0001080a0})
    /usr/lib/go-1.23/src/crypto/tls/conn.go:1568 +0x3a6 fp=0xc000093f70 sp=0xc000093d58 pc=0x5efec6
crypto/tls.(*Conn).HandshakeContext(...)
    /usr/lib/go-1.23/src/crypto/tls/conn.go:1508
net/http.(*persistConn).addTLS.func2()
    /usr/lib/go-1.23/src/net/http/transport.go:1651 +0x6e fp=0xc000093fe0 sp=0xc000093f70 pc=0x6721ee
runtime.goexit({})
    /usr/lib/go-1.23/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x472401
created by net/http.(*persistConn).addTLS in goroutine 19
    /usr/lib/go-1.23/src/net/http/transport.go:1647 +0x309

full callstack with other goroutines is attached: callstack.txt

What did you see happen?

Panic during TLS handshake from http client

What did you expect to see?

No panic

gabyhelp commented 4 hours ago

Related Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

cherrymui commented 3 hours ago

The stack trace seems interleaved with some other output?

[signal SIGSEGV: segmentation violation code=3D0x80 addr=3D0x0 pc=3D0x473=
301]

goroutine 23 gp=3D0xc000084a80 m=3D4 mp=3D0xc000080008 [running]:

These seems not valid hex numbers. Could you share a clean stack trace?

Also, it looks like it is panicking in string concatenation, which seems impossible, as strings don't involve nil pointers. Maybe there is a memory corruption? Does your program use cgo or unsafe? Have you tried running under the race detector? Thanks.

fishy commented 2 hours ago

As I noted in the report, the callstack is copied directly from /var/spool/mail/ so there are extra line wrapping and mail encoding around line wrappings. I didn't figure out a good way to copy it out of neomutt.

this only happened once after I've run it for more than a year hourly, so it's possible that it's caused by some memory corruption.

cherrymui commented 2 hours ago

It is not just line wrapping, but the extra characters printed. E.g. it seems 3D appears in a few weird places.

Okay, as this only happened once, I think we'll keep the WaitingForInfo label. If you see more failures, please update and we can investigate. Thanks.

fishy commented 2 hours ago

anyways I got the decoded clean copy out of neomutt, updated the attachment file and the snippet in the original report.