search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
123.57k stars 17.61k forks source link

runtime: spurious crashes related to timers in go 1.23.0 #69625

Open fiber opened 3 weeks ago

fiber commented 3 weeks ago

Go version

go version go1.23.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/gocode/src/go.gtt.net/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3222024183=/tmp/go-build -gno-record-gcc-switches'

What did you do?

recompiled go code that has been running fine for a long time.

What did you see happen?

we are experiencing frequent crashes that follow this pattern:

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad

What did you expect to see?

no crash.

gabyhelp commented 3 weeks ago

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

seankhliao commented 3 weeks ago

are you copying timer values and not pointers?

we need a reproducer for this to be actionable

fiber commented 3 weeks ago

This is a sizeable piece of code. I would not exclude bugs, however it's unlikely that we I am copying timer values, since we're using time.Timer/time.Ticker everywhere.

the below crashes may be related, although I don't understand how. I am rolling back to 1.22.7 to check 

unexpected fault address 0x30000000009
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30000000009 pc=0x466ed2]

goroutine 1014077 gp=0xc0073fd340 m=251 mp=0xc01370b508 [running]:
runtime.throw({0x1d00572?, 0xc01b3a0d20?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc01b3a0af0 sp=0xc01b3a0ac0 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc01b3a0b50 sp=0xc01b3a0af0 pc=0x4808ac
runtime.(*timer).maybeRunChan(0x0?)
        /go/src/runtime/time.go:1244 +0x12 fp=0xc01b3a0b80 sp=0xc01b3a0b50 pc=0x466ed2
runtime.selectgo(0xc01b3a0d20, 0xc01b3a0ce0, 0x22?, 0x1, 0x81ec0a?, 0x1)
        /go/src/runtime/select.go:177 +0x13f fp=0xc01b3a0ca8 sp=0xc01b3a0b80 pc=0x459b1f
io.(*pipe).write(0xc0f0037560, {0xc078532000, 0x1000, 0x1000})
        /go/src/io/pipe.go:86 +0x1e5 fp=0xc01b3a0d80 sp=0xc01b3a0ca8 pc=0x4c12c5
io.(*PipeWriter).Write(0x41f2cb?, {0xc078532000?, 0xc04473bd40?, 0xc0072bc430?})
        /go/src/io/pipe.go:161 +0x1a fp=0xc01b3a0db0 sp=0xc01b3a0d80 pc=0x4c173a
bufio.(*Writer).Flush(0xc065c60040)
        /go/src/bufio/bufio.go:639 +0x55 fp=0xc01b3a0df8 sp=0xc01b3a0db0 pc=0x4d6ef5
bufio.(*Writer).Write(0xc065c60040, {0xc0410de480?, 0x1cb89e0?, 0xc01b3a0fa8?})
        /go/src/bufio/bufio.go:683 +0xcd fp=0xc01b3a0e58 sp=0xc01b3a0df8 pc=0x4d71ad
...

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 72153 gp=0xc004e9bc00 m=518 mp=0xc013509508 [running]:
runtime.throw({0x1d105d2?, 0xc0514c19c9?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc0076ca970 sp=0xc0076ca940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc0076ca9d0 sp=0xc0076ca970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc0514c19c8?}, 0x9b?, {0x80?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc0076ca9d8 sp=0xc0076ca9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262150, {0x1d659d2?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076caa30 sp=0xc0076ca9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076caa68 sp=0xc0076caa30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc004e9bc00?}, 0xc00c9195d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc0076caab8 sp=0xc0076caa68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc000262298, {0xc0514c1950?}, {0x89?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc0076cab10 sp=0xc0076caab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x92?, {0x92?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc0076cab48 sp=0xc0076cab10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x5d7?, {{}, {0x22d52e0?, 0xc049ec4000?}, 0x84?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc0076cab70 sp=0xc0076cab48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22d52e0?, 0xc049ec4000?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc0076caba0 sp=0xc0076cab70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xbc?, 0xe9?}, {0x0, 0x0, 0x0}, {0x22d52e0, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc0076cac70 sp=0xc0076caba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x22b1580, 0xc049ec4000})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc0076cacc0 sp=0xc0076cac70 pc=0x8a7c90

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x8e0055]

goroutine 63842 gp=0xc0025eae00 m=312 mp=0xc012afdc08 [running]:
runtime.throw({0x1d00572?, 0xc01c07046f?})
        /go/src/runtime/panic.go:1067 +0x48 fp=0xc000e1b970 sp=0xc000e1b940 pc=0x47e548
runtime.sigpanic()
        /go/src/runtime/signal_unix.go:914 +0x26c fp=0xc000e1b9d0 sp=0xc000e1b970 pc=0x4808ac
google.golang.org/protobuf/internal/impl.sizeInt64PackedSlice({0xc01c07046e?}, 0x14b?, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_gen.go:2061 +0x15 fp=0xc000e1b9d8 sp=0xc000e1b9d0 pc=0x8e0055
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0150, {0x1d55878?}, {0x88?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1ba30 sp=0xc000e1b9d8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x0?, {0x0?}, {0x0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1ba68 sp=0xc000e1ba30 pc=0x9012af
google.golang.org/protobuf/internal/impl.sizeMessageSliceInfo({0xc0025eae00?}, 0xc0058300d0, {0x49?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:486 +0x50 fp=0xc000e1bab8 sp=0xc000e1ba68 pc=0x8d5370
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointerSlow(0xc0002f0298, {0xc01c070300?}, {0x7f?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:82 +0xd3 fp=0xc000e1bb10 sp=0xc000e1bab8 pc=0x9013d3
google.golang.org/protobuf/internal/impl.(*MessageInfo).sizePointer(0x188?, {0x188?}, {0xb0?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:59 +0x6f fp=0xc000e1bb48 sp=0xc000e1bb10 pc=0x9012af
google.golang.org/protobuf/internal/impl.(*MessageInfo).size(0x400?, {{}, {0x22c3760?, 0xc018dc5930?}, 0xc?})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/encode.go:40 +0x52 fp=0xc000e1bb70 sp=0xc000e1bb48 pc=0x9011d2
google.golang.org/protobuf/internal/impl.(*MessageInfo).size-fm({{}, {0x22c3760?, 0xc018dc5930?}, 0x48?})
        <autogenerated>:1 +0x35 fp=0xc000e1bba0 sp=0xc000e1bb70 pc=0x91fdb5
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x0?, 0xae?, 0x5e?}, {0x0, 0x0, 0x0}, {0x22c3760, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:184 +0x11b fp=0xc000e1bc70 sp=0xc000e1bba0 pc=0x8a815b
google.golang.org/protobuf/proto.Marshal({0x229fb40, 0xc018dc5930})
        /root/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/encode.go:110 +0x50 fp=0xc000e1bcc0 sp=0xc000e1bc70 pc=0x8a7c90
ianlancetaylor commented 3 weeks ago

CC @golang/runtime

A fault address like 0x30000000009 looks like memory corruption. It's not necessarily in code related to timers; that may just be where it is showing up. Look closely at any uses of unsafe or cgo.