search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
122.85k stars 17.52k forks source link

runtime: unexpected signal during runtime execution on Debian #29377

Open cosmtrek opened 5 years ago

cosmtrek commented 5 years ago

What version of Go are you using (go version)?

$ go version
go1.10.1 linux/amd64

Does this issue reproduce with the latest release?

No

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tiger/.cache/"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/tiger/compile_path"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build590147109=/tmp/go-build -gno-record-gcc-switches"

What did you do?

My service started in a docker container and happened to throw panic. I cannot figure it out.

The stack shows as follow:

fatal error: unexpected signal during runtime execution
  [signal SIGSEGV: segmentation violation code=0x1 addr=0x34 pc=0x4341f1]

  runtime stack:
  runtime: unexpected return pc for runtime.schedule called from 0x100000002
  stack: frame={sp:0x7fa5c902ed78, fp:0x7fa5c902edc0} stack=[0x7fa5c882f2b0,0x7fa5c902eeb0)
  00007fa5c902ec78:  0000000000000034  0000000000000001
  00007fa5c902ec88:  000000000249e56c  0000000000000000
  00007fa5c902ec98:  00007fa5c902ecc0  0000000000457b8c <runtime.dopanic.func1+60>
  00007fa5c902eca8:  000000c4200f0600  000000000042c921 <runtime.throw+129>
  00007fa5c902ecb8:  00007fa5c902ed08  00007fa5c902ecf8
  00007fa5c902ecc8:  000000000042c84a <runtime.dopanic+74>  00007fa5c902ecd8
  00007fa5c902ecd8:  0000000000457b50 <runtime.dopanic.func1+0>  000000c4200f0600
  00007fa5c902ece8:  000000000042c921 <runtime.throw+129>  00007fa5c902ed08
  00007fa5c902ecf8:  00007fa5c902ed18  000000000042c921 <runtime.throw+129>
  00007fa5c902ed08:  0000000000000000  000000000000002a
  00007fa5c902ed18:  00007fa5c902ed68  00000000004423be <runtime.sigpanic+654>
  00007fa5c902ed28:  00000000024f5aac  000000000000002a
  00007fa5c902ed38:  0000000000439fd7 <runtime.runqsteal+87>  000000c420050a00
  00007fa5c902ed48:  000000c42004c5e8  000000c4200f0600
  00007fa5c902ed58:  0000000000000000  000b947b2004c000
  00007fa5c902ed68:  00007fa5c902ee10  00000000004341f1 <runtime.schedule+225>
  00007fa5c902ed78: <000000c42004c000  000000c420050a00
  00007fa5c902ed88:  000000c400000000  0000000000000000
  00007fa5c902ed98:  00007fa5c902edd0  0000000000000004
  00007fa5c902eda8:  0000000000000000  0000000400000003
  00007fa5c902edb8: !0000000100000002 >000000c42004c000
  00007fa5c902edc8:  000000c4201f6800  00007fa5c902edf0
  00007fa5c902edd8:  0000000000432bc8 <runtime.wakep+72>  0000000000000000
  00007fa5c902ede8:  000000c42004a000  00000000004331c5 <runtime.execute+261>
  00007fa5c902edf8:  000000c4200f0600  000000c42004c000
  00007fa5c902ee08:  00007fa5c902ed88  00007fa5c902ee58
  00007fa5c902ee18:  000000000043424b <runtime.schedule+315>  000000c42004c000
  00007fa5c902ee28:  0000000000000000  0000000000436300 <runtime.gfput+256>
  00007fa5c902ee38:  00007fa5c902ee58  000000000043d435 <runtime.selparkcommit+133>
  00007fa5c902ee48:  000000c4200f0600  000000c4222fa120
  00007fa5c902ee58:  00007fa5c902ee88  0000000000434576 <runtime.park_m+182>
  00007fa5c902ee68:  000000c4201f2a80  0000000000000000
  00007fa5c902ee78:  00007fa5cc7d0601  000000c4200f0600
  00007fa5c902ee88:  000000c421a4e268  000000000045925b <runtime.mcall+91>
  00007fa5c902ee98:  000000c4201f2a80  0000000000000000
  00007fa5c902eea8:  0100000000000000
  runtime.throw(0x24f5aac, 0x2a)
      /usr/local/go/src/runtime/panic.go:616 +0x81
  runtime.sigpanic()
      /usr/local/go/src/runtime/signal_unix.go:372 +0x28e
  runtime: unexpected return pc for runtime.schedule called from 0x100000002
  stack: frame={sp:0x7fa5c902ed78, fp:0x7fa5c902edc0} stack=[0x7fa5c882f2b0,0x7fa5c902eeb0)
  00007fa5c902ec78:  0000000000000034  0000000000000001
  00007fa5c902ec88:  000000000249e56c  0000000000000000
  00007fa5c902ec98:  00007fa5c902ecc0  0000000000457b8c <runtime.dopanic.func1+60>
  00007fa5c902eca8:  000000c4200f0600  000000000042c921 <runtime.throw+129>
  00007fa5c902ecb8:  00007fa5c902ed08  00007fa5c902ecf8
  00007fa5c902ecc8:  000000000042c84a <runtime.dopanic+74>  00007fa5c902ecd8
  00007fa5c902ecd8:  0000000000457b50 <runtime.dopanic.func1+0>  000000c4200f0600
  00007fa5c902ece8:  000000000042c921 <runtime.throw+129>  00007fa5c902ed08
  00007fa5c902ecf8:  00007fa5c902ed18  000000000042c921 <runtime.throw+129>
  00007fa5c902ed08:  0000000000000000  000000000000002a
  00007fa5c902ed18:  00007fa5c902ed68  00000000004423be <runtime.sigpanic+654>
  00007fa5c902ed28:  00000000024f5aac  000000000000002a
  00007fa5c902ed38:  0000000000439fd7 <runtime.runqsteal+87>  000000c420050a00
  00007fa5c902ed48:  000000c42004c5e8  000000c4200f0600
  00007fa5c902ed58:  0000000000000000  000b947b2004c000
  00007fa5c902ed68:  00007fa5c902ee10  00000000004341f1 <runtime.schedule+225>
  00007fa5c902ed78: <000000c42004c000  000000c420050a00
  00007fa5c902ed88:  000000c400000000  0000000000000000
  00007fa5c902ed98:  00007fa5c902edd0  0000000000000004
  00007fa5c902eda8:  0000000000000000  0000000400000003
  00007fa5c902edb8: !0000000100000002 >000000c42004c000
  00007fa5c902edc8:  000000c4201f6800  00007fa5c902edf0
  00007fa5c902edd8:  0000000000432bc8 <runtime.wakep+72>  0000000000000000
  00007fa5c902ede8:  000000c42004a000  00000000004331c5 <runtime.execute+261>
  00007fa5c902edf8:  000000c4200f0600  000000c42004c000
  00007fa5c902ee08:  00007fa5c902ed88  00007fa5c902ee58
  00007fa5c902ee18:  000000000043424b <runtime.schedule+315>  000000c42004c000
  00007fa5c902ee28:  0000000000000000  0000000000436300 <runtime.gfput+256>
  00007fa5c902ee38:  00007fa5c902ee58  000000000043d435 <runtime.selparkcommit+133>
  00007fa5c902ee48:  000000c4200f0600  000000c4222fa120
  00007fa5c902ee58:  00007fa5c902ee88  0000000000434576 <runtime.park_m+182>
  00007fa5c902ee68:  000000c4201f2a80  0000000000000000
  00007fa5c902ee78:  00007fa5cc7d0601  000000c4200f0600
  00007fa5c902ee88:  000000c421a4e268  000000000045925b <runtime.mcall+91>
  00007fa5c902ee98:  000000c4201f2a80  0000000000000000
  00007fa5c902eea8:  0100000000000000
  runtime.schedule()
      /usr/local/go/src/runtime/proc.go:2547 +0xe1

// some goroutines maybe helpful to show what happened
goroutine 1 [select, 11 minutes]:
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.waitSignal(0xc42239f140, 0x0, 0x0)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:138 +0x1cf
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.Run(0xc400000018, 0x25ed2b8)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:126 +0x66
  main.main()
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/server.go:78 +0x5d6

  goroutine 5 [select]:
  code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics.(*Sender).emitCounterLoop(0xc4200e2100)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics/sender.go:221 +0x167
  created by code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics.(*Sender).runLoops
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/gopkg/metrics/sender.go:213 +0x3f

...

goroutine 2268 [IO wait]:
  internal/poll.runtime_pollWait(0x7fa5cc7846a0, 0x72, 0x0)
      /usr/local/go/src/runtime/netpoll.go:173 +0x57
  internal/poll.(*pollDesc).wait(0xc4201d8998, 0x72, 0xc4222bf200, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
  internal/poll.(*pollDesc).waitRead(0xc4201d8998, 0xffffffffffffff00, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
  internal/poll.(*FD).Accept(0xc4201d8980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
  net.(*netFD).accept(0xc4201d8980, 0xa17e614831, 0x3b8c160, 0x1)
      /usr/local/go/src/net/fd_unix.go:238 +0x42
  net.(*TCPListener).accept(0xc4202be3a8, 0x458530, 0xc42d0abe38, 0xc42d0abe40)
      /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
  net.(*TCPListener).Accept(0xc4202be3a8, 0x25ed4a8, 0xc420da2f00, 0x26f0d00, 0xc440373ad8)
      /usr/local/go/src/net/tcpsock.go:259 +0x49
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.(*RpcServer).AcceptLoop(0xc420da2f00, 0x0, 0x0)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite_server.go:106 +0x4b
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.(*RpcServer).Serve(0xc420da2f00, 0x0, 0x0)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite_server.go:189 +0x118
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.Run.func1(0xc42239f140)
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:124 +0x2d
  created by code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.Run
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/kite.go:123 +0x58

...

goroutine 2277 [IO wait, 11 minutes]:
  internal/poll.runtime_pollWait(0x7fa5cc784910, 0x72, 0x0)
      /usr/local/go/src/runtime/netpoll.go:173 +0x57
  internal/poll.(*pollDesc).wait(0xc420d7f398, 0x72, 0xc422944f00, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
  internal/poll.(*pollDesc).waitRead(0xc420d7f398, 0xffffffffffffff00, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
  internal/poll.(*FD).Accept(0xc420d7f380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
      /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
  net.(*netFD).accept(0xc420d7f380, 0x7fa5cc7d0d90, 0x0, 0x25f12c0)
      /usr/local/go/src/net/fd_unix.go:238 +0x42
  net.(*TCPListener).accept(0xc4201bc128, 0xc4229b1e10, 0x412758, 0x30)
      /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
  net.(*TCPListener).AcceptTCP(0xc4201bc128, 0x2235680, 0xc421a30cc0, 0x210ab60)
      /usr/local/go/src/net/tcpsock.go:246 +0x49
  net/http.tcpKeepAliveListener.Accept(0xc4201bc128, 0xc420044110, 0x210ab60, 0x3b6f970, 0x23954e0)
      /usr/local/go/src/net/http/server.go:3216 +0x2f
  net/http.(*Server).Serve(0xc420106f70, 0x26ecbc0, 0xc4201bc128, 0x0, 0x0)
      /usr/local/go/src/net/http/server.go:2770 +0x1a5
  net/http.(*Server).ListenAndServe(0xc420106f70, 0xc420106f70, 0x1)
      /usr/local/go/src/net/http/server.go:2711 +0xa9
  net/http.ListenAndServe(0xc420eaafb8, 0x6, 0x0, 0x0, 0x1, 0x0)
      /usr/local/go/src/net/http/server.go:2969 +0x7a
  code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.startDebugServer.func2()
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/debug.go:40 +0xb2
  created by code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite.startDebugServer
      /opt/tiger/compile_go_path/src/code.byted.org/video/stream_service/vendor/code.byted.org/kite/kite/debug.go:37 +0x101

I suspect some goroutines waiting for too long to make runtime scheduler abnormal, but I cannot reproduce it.

ianlancetaylor commented 5 years ago

Is this repeatable?

CC @aclements

cosmtrek commented 5 years ago

@ianlancetaylor No, rarely. The difficulty is I could not reproduce it.