search

docker-archive / classicswarm

Swarm Classic: a container clustering system. Not to be confused with Docker Swarm which is at https://github.com/docker/swarmkit
Apache License 2.0
5.76k stars 1.08k forks source link

swarm panic from time to time #1426

Closed agsmorodin closed 8 years ago

agsmorodin commented 8 years ago

I get the following exception that I am experiencing from time to time with latest docker. I get this error almost every day.

docker -v Docker version 1.9.0, build 76d6bc9

and swarm : docker exec -ti e360 /swarm --version swarm version 1.0.0 (087e245).

System details as requested ( I am using docker machine with centos 7.1 image ).

cat /etc/*release*
CentOS Linux release 7.1.1503 (Core) 
Derived from Red Hat Enterprise Linux 7.1 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.1.1503 (Core) 
CentOS Linux release 7.1.1503 (Core) 
cpe:/o:centos:centos:7

uname -a

Linux swarm-master 3.10.0-123.8.1.el7.x86_64 #1 SMP Mon Sep 22 19:06:58 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x45aeb0]

goroutine 102 [running]:
github.com/docker/swarm/cluster.(*Engine).refreshContainer(0xc8202068c0, 0xc82099a800, 0x40, 0x1, 0xffffffffffffff01, 0x0, 0x0)
    /go/src/github.com/docker/swarm/cluster/engine.go:346 +0x44e
github.com/docker/swarm/cluster.(*Engine).handler(0xc8202068c0, 0xc820129f40, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/cluster/engine.go:697 +0x14a
github.com/docker/swarm/cluster.(*Engine).(github.com/docker/swarm/cluster.handler)-fm(0xc820129f40, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/cluster/engine.go:132 +0x52
github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents.func1(0xc8200e1b30, 0x0, 0xc82042c610, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:475 +0x1c8
created by github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:477 +0xa1

goroutine 1 [chan receive, 1897 minutes]:
github.com/docker/swarm/api.(*Server).ListenAndServe(0xc8201400f0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/api/server.go:117 +0x383
github.com/docker/swarm/cli.manage(0xc8201020c0)
    /go/src/github.com/docker/swarm/cli/manage.go:295 +0x1552
github.com/codegangsta/cli.Command.Run(0xd49c80, 0x6, 0xd49c28, 0x1, 0xdd2290, 0x17, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/codegangsta/cli/command.go:113 +0x1082
github.com/codegangsta/cli.(*App).Run(0xc8200cd380, 0xc82000a0b0, 0xb, 0xb, 0x0, 0x0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/codegangsta/cli/app.go:156 +0xbb1
github.com/docker/swarm/cli.Run()
    /go/src/github.com/docker/swarm/cli/cli.go:57 +0x4a4
main.main()
    /go/src/github.com/docker/swarm/main.go:13 +0x14

goroutine 6 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x1187040)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x67
created by github.com/golang/glog.init.1
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x297

goroutine 7 [select]:
github.com/docker/swarm/discovery/file.(*Discovery).Watch.func1(0xc82011fbc0, 0xc82011fb60, 0xc8200100a0, 0xc82012cac0, 0x0)
    /go/src/github.com/docker/swarm/discovery/file/file.go:83 +0x32c
created by github.com/docker/swarm/discovery/file.(*Discovery).Watch
    /go/src/github.com/docker/swarm/discovery/file/file.go:101 +0xb8

goroutine 8 [select, 1897 minutes]:
github.com/docker/swarm/cluster/swarm.(*Cluster).monitorDiscovery(0xc82011fb00, 0xc82011fb60, 0xc82011fbc0)
    /go/src/github.com/docker/swarm/cluster/swarm/cluster.go:264 +0x47a
created by github.com/docker/swarm/cluster/swarm.NewCluster
    /go/src/github.com/docker/swarm/cluster/swarm/cluster.go:78 +0x446

goroutine 189 [select]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc820206540)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 190 [select]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc820206fc0)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 212 [select]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc820206000)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 188 [select]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc820206380)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 211 [select]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc8202068c0)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 210 [select, 1 minutes]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc820206c40)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 191 [select]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc8202061c0)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 213 [select]:
github.com/docker/swarm/cluster.(*Engine).refreshLoop(0xc820206700)
    /go/src/github.com/docker/swarm/cluster/engine.go:403 +0xc53
created by github.com/docker/swarm/cluster.(*Engine).ConnectWithClient
    /go/src/github.com/docker/swarm/cluster/engine.go:148 +0x1fb

goroutine 53 [chan receive, 3 minutes]:
github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents.func1(0xc8200e1dd0, 0x0, 0xc82055f6d0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:468 +0xfe
created by github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:477 +0xa1

goroutine 71 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc8201562c0)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 73 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc8201566e0)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 15 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc8201564d0)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 67 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc820156630)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 65 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc820156420)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 52 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc820156160)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 69 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc820156210)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 75 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc8201560b0)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 13 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc820156370)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 14 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc8201564d0)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 16 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc820156420)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 12 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc820156370)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 70 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc8201562c0)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 68 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc820156210)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 51 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc820156160)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 74 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc8201560b0)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 72 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc8201566e0)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 66 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc820156630)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 9 [IO wait]:
net.runtime_pollWait(0x7f9ce5c5bce8, 0x72, 0xc820012130)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820435fe0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820435fe0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820435f80, 0x0, 0x7f9ce5ca47c0, 0xc8206250e0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820024b68, 0xc820492e40, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820024b68, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
crypto/tls.(*listener).Accept(0xc820430be0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/tls.go:46 +0x60
net/http.(*Server).Serve(0xc82043c300, 0x7f9ce5ca4788, 0xc820430be0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
github.com/docker/swarm/api.(*Server).ListenAndServe.func1(0xc820430aa0, 0x2, 0x2, 0xc8201400f0, 0xc82043c2a0)
    /go/src/github.com/docker/swarm/api/server.go:110 +0x4f1
created by github.com/docker/swarm/api.(*Server).ListenAndServe
    /go/src/github.com/docker/swarm/api/server.go:113 +0x30f

goroutine 49 [select, 1897 minutes]:
net/http.(*persistConn).readLoop(0xc820156580)
    /usr/local/go/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 50 [select, 1897 minutes]:
net/http.(*persistConn).writeLoop(0xc820156580)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 119 [IO wait, 113 minutes]:
net.runtime_pollWait(0x7f9ce5c5c2e8, 0x72, 0xc820012130)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8201b79c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8201b79c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8201b7960, 0xc8205f4000, 0x1000, 0x1000, 0x0, 0x7f9ce5c9b050, 0xc820012130)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200ea160, 0xc8205f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820284e70, 0x7f9ce5c5c578, 0xc8200ea160, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc820153080, 0xee0117, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc820153080, 0xc820386000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x7f9ce5ca8b80, 0xc820153080, 0xc820156268, 0xc820386000, 0x1000, 0x1000, 0x8, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc82057ca60, 0xc820386000, 0x1000, 0x1000, 0xc1, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc820365800)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc820365800, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:328 +0x21a
net/http/internal.readLine(0xc820365800, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/internal/chunked.go:110 +0x4b
net/http/internal.(*chunkedReader).beginChunk(0xc820176000)
    /usr/local/go/src/net/http/internal/chunked.go:47 +0x39
net/http/internal.(*chunkedReader).Read(0xc820176000, 0xc820b02001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/internal/chunked.go:77 +0xb7
net/http.(*body).readLocked(0xc8204c8040, 0xc820b02001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:632 +0x98
net/http.(*body).Read(0xc8204c8040, 0xc820b02001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transfer.go:624 +0x110
net/http.(*bodyEOFSignal).Read(0xc8204c8080, 0xc820b02001, 0x5ff, 0x5ff, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1296 +0x26a
encoding/json.(*Decoder).refill(0xc8205f0d00, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:152 +0x287
encoding/json.(*Decoder).readValue(0xc8205f0d00, 0x1, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:128 +0x41b
encoding/json.(*Decoder).Decode(0xc8205f0d00, 0xa4b620, 0xc820930780, 0x0, 0x0)
    /usr/local/go/src/encoding/json/stream.go:57 +0x159
github.com/samalba/dockerclient.(*DockerClient).MonitorEvents.func1(0xc8205f0d00, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:435 +0x87
github.com/samalba/dockerclient.(*DockerClient).readJSONStream.func1.1(0x7f9ce5c650d0, 0xc8204c8080, 0xc8203d3ef0, 0xc820180060)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:256 +0xce
created by github.com/samalba/dockerclient.(*DockerClient).readJSONStream.func1
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:263 +0x89

goroutine 10833 [select]:
net/http.(*persistConn).writeLoop(0xc8205d4bb0)
    /usr/local/go/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 59 [chan receive]:
github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents.func1(0xc820284330, 0x0, 0xc82044a310, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:468 +0xfe
created by github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:477 +0xa1

goroutine 126 [chan receive, 127 minutes]:
github.com/samalba/dockerclient.(*DockerClient).MonitorEvents.func2(0xc820546d20, 0xc820546d80)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:444 +0x69
created by github.com/samalba/dockerclient.(*DockerClient).MonitorEvents
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:452 +0xbc3

goroutine 125 [select, 127 minutes]:
github.com/samalba/dockerclient.(*DockerClient).readJSONStream.func1(0x7f9ce5c650d0, 0xc8204ee900, 0xc8203cbdc0, 0xc820546d20, 0xc820017aa0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:268 +0x2a6
created by github.com/samalba/dockerclient.(*DockerClient).readJSONStream
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:283 +0x7b

goroutine 63 [chan receive, 127 minutes]:
github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents.func1(0xc8200e1890, 0x0, 0xc82044ada0, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:468 +0xfe
created by github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:477 +0xa1

goroutine 12310 [IO wait]:
net.runtime_pollWait(0x7f9ce5cb0218, 0x72, 0xc820012130)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8203d5100, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8203d5100, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8203d50a0, 0xc820186800, 0x800, 0x800, 0x0, 0x7f9ce5c9b050, 0xc820012130)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820024180, 0xc820186800, 0x800, 0x800, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc82035e510, 0x7f9ce5c5c578, 0xc820024180, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc82013a840, 0xee0117, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc82013a840, 0xc820628000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x7f9ce5ca8b80, 0xc82013a840, 0xc820fd6ec8, 0xc820628000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc82036ab60, 0xc820628000, 0x1000, 0x1000, 0xc82136b2d0, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc820294cc0)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc820294cc0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820fd6e70)
    /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 79 [chan receive, 113 minutes]:
github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents.func1(0xc8200e1350, 0x0, 0xc820448930, 0x0, 0x0, 0x0)
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:468 +0xfe
created by github.com/samalba/dockerclient.(*DockerClient).StartMonitorEvents
    /go/src/github.com/docker/swarm/Godeps/_workspace/src/github.com/samalba/dockerclient/dockerclient.go:477 +0xa1
abronan commented 8 years ago

@agsmorodin Thank you for opening a separate issue. Is there any sequence of actions that triggers this error, something interesting in the Manager logs? Never encountered this issue on long running clusters since we fixed the refresh loop but never tried on CentOS (or RHEL 7.1) with an older kernel (3.10 is the strict minimum required) :confused:

By any chance, is one of the containers in your cluster in the Dead state when this occurs? Printing the Manager logs just before the panic as well as the output of docker ps -a might help. Thanks!

aluzzardi commented 8 years ago

Thanks @agsmorodin - I think #1414 fixes this.

aluzzardi commented 8 years ago

@agsmorodin - would you mind trying with master?

You can run the dockerswarm/swarm:master image which will contain this fix.

aluzzardi commented 8 years ago

Closing this in the meantime - @agsmorodin please re-open if the problem continues.

Thanks!

abronan commented 8 years ago

Fixed by #1465