search

zalando-stups / stups-etcd-cluster

Etcd cluster appliance for the STUPS (AWS) environment
Other
30 stars 9 forks source link

Member fails to recover #43

Open linki opened 7 years ago

linki commented 7 years ago

Our cluster has an unhealthy node which keeps printing the following in the logs. Let me know if you need more information.

Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,639 - Calling paginated ec2:describe_instances with {'Filters': [{'Values': ['etcd-cluster-etcd'], 'Name': 'tag:aws:cloudformation:stack-name'}]}
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,650 - Starting new HTTPS connection (1): ec2.eu-central-1.amazonaws.com
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,738 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,754 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,757 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,759 - My clientURLs list is not empty: ['http://172.31.131.225:2379']
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,759 - My data directory exists=True
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,761 - Started new /bin/etcd process with pid: 444 and args: ['-name', 'i-05982c02c444ea551', '--data-dir', 'data', '-listen-peer-urls', 'http://0.0.0.0:2380', '-initial-advertise-peer-urls', 'http://172.31.131.225:2380', '-listen-client-urls', 'http://0.0.0.0:2379', '-advertise-client-urls', 'http://172.31.131.225:2379', '-initial-cluster', 'i-05982c02c444ea551=http://172.31.131.225:2380,i-06cda1ba0307bcd19=http://172.31.140.202:2380,i-06ed4eb078b00f623=http://172.31.153.93:2380,i-077dfb38fc0039342=http://172.31.159.46:2380,i-0a665d73f1d8c7cfa=http://172.31.131.127:2380', '-initial-cluster-token', 'etcd-cluster-etcd', '-initial-cluster-state', 'existing']
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791434 W | flags: unrecognized environment variable ETCDVERSION=3.0.15
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791768 I | etcdmain: etcd Version: 3.0.15
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791939 I | etcdmain: Git SHA: fc00305
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792099 I | etcdmain: Go Version: go1.6.3
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792262 I | etcdmain: Go OS/Arch: linux/amd64
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792433 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792618 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792891 I | etcdmain: listening for peers on http://0.0.0.0:2380
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.793147 I | etcdmain: listening for client requests on 0.0.0.0:2379
Jun 28 11:58:35 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:35,505 - Starting new HTTP connection (1): 172.31.131.225
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.301983 I | etcdserver: recovered store from snapshot at index 222437884
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302392 I | etcdserver: name = i-05982c02c444ea551
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302576 I | etcdserver: data dir = data
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302734 I | etcdserver: member dir = data/member
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302895 I | etcdserver: heartbeat = 100ms
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303048 I | etcdserver: election = 1000ms
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303180 I | etcdserver: snapshot count = 10000
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303341 I | etcdserver: advertise client URLs = http://172.31.131.225:2379
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: ERROR  2017-06-28 11:58:38,616 - Exception in HouseKeeper main loop
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 377, in _make_request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     httplib_response = conn.getresponse(buffering=True)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: TypeError: getresponse() got an unexpected keyword argument 'buffering'
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 379, in _make_request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     httplib_response = conn.getresponse()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     response.begin()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/http/client.py", line 297, in begin
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     version, status, reason = self._read_status()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/socket.py", line 575, in readinto
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return self._sock.recv_into(b)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: socket.timeout: timed out
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     timeout=timeout
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     _stacktrace=sys.exc_info()[2])
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 247, in increment
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise six.reraise(type(error), error, _stacktrace)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise value
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     body=body, headers=headers)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='172.31.131.225', port=2379): Read timed out. (read timeout=3.1)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 586, in run
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     if self.manager.etcd_pid != 0 and self.is_leader():
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 497, in is_leader
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return self.manager.me.is_leader()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 194, in is_leader
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return not self.api_get('stats/leader') is None
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 163, in api_get
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     response = requests.get(url, timeout=self.API_TIMEOUT)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return request('get', url, params=params, **kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return session.request(method=method, url=url, **kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     resp = self.send(prep, **send_kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     r = adapter.send(request, **kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise ReadTimeout(e, request=request)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: requests.exceptions.ReadTimeout: HTTPConnectionPool(host='172.31.131.225', port=2379): Read timed out. (read timeout=3.1)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.899646 I | etcdserver: restarting member 53733ad8236d11ce in cluster 5cee1e413d02 at commit index 222455528
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.900135 C | raft: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: panic: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: goroutine 1 [running]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: panic(0xd450a0, 0xc835481250)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/usr/local/go/src/runtime/panic.go:481 +0x3e6
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820190620, 0x12375a0, 0x2b, 0xc832070f40, 0x4, 0x4)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).loadState(0xc83639f450, 0x1b334, 0x88b67ac3e24bbdb2, 0xd4266e8, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:942 +0x2a2
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.newRaft(0xc8201078d8, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:225 +0x8ff
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.RestartNode(0xc8201078d8, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:213 +0x45
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.restartNode(0xc82006fb80, 0xc820066000, 0x29, 0xc820107d78, 0x0, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:369 +0x7c7
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc82006fb80, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:348 +0x430e
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc8201a8000, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:374 +0x245f
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:116 +0x2101
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:36 +0x21e
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: main.main()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/main.go:28 +0x14
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:58:38,933 - Process 444 finished with exit code 2
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:58:38,933 - Sleeping 30 seconds before next try...
Jun 28 11:59:08 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:08,944 - Calling paginated ec2:describe_instances with {'Filters': [{'Values': ['etcd-cluster-etcd'], 'Name': 'tag:aws:cloudformation:stack-name'}]}
Jun 28 11:59:08 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:08,959 - Starting new HTTPS connection (1): ec2.eu-central-1.amazonaws.com
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,067 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,071 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,074 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,076 - My clientURLs list is not empty: ['http://172.31.131.225:2379']
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,076 - My data directory exists=True
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,077 - Started new /bin/etcd process with pid: 450 and args: ['-name', 'i-05982c02c444ea551', '--data-dir', 'data', '-listen-peer-urls', 'http://0.0.0.0:2380', '-initial-advertise-peer-urls', 'http://172.31.131.225:2380', '-listen-client-urls', 'http://0.0.0.0:2379', '-advertise-client-urls', 'http://172.31.131.225:2379', '-initial-cluster', 'i-05982c02c444ea551=http://172.31.131.225:2380,i-06cda1ba0307bcd19=http://172.31.140.202:2380,i-06ed4eb078b00f623=http://172.31.153.93:2380,i-077dfb38fc0039342=http://172.31.159.46:2380,i-0a665d73f1d8c7cfa=http://172.31.131.127:2380', '-initial-cluster-token', 'etcd-cluster-etcd', '-initial-cluster-state', 'existing']
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108064 W | flags: unrecognized environment variable ETCDVERSION=3.0.15
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108416 I | etcdmain: etcd Version: 3.0.15
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108593 I | etcdmain: Git SHA: fc00305
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108771 I | etcdmain: Go Version: go1.6.3
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108929 I | etcdmain: Go OS/Arch: linux/amd64
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109088 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109278 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109530 I | etcdmain: listening for peers on http://0.0.0.0:2380
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109792 I | etcdmain: listening for client requests on 0.0.0.0:2379
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.859842 I | etcdserver: recovered store from snapshot at index 222437884
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860265 I | etcdserver: name = i-05982c02c444ea551
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860450 I | etcdserver: data dir = data
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860609 I | etcdserver: member dir = data/member
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860753 I | etcdserver: heartbeat = 100ms
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860895 I | etcdserver: election = 1000ms
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.861047 I | etcdserver: snapshot count = 10000
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.861212 I | etcdserver: advertise client URLs = http://172.31.131.225:2379
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:18.446568 I | etcdserver: restarting member 53733ad8236d11ce in cluster 5cee1e413d02 at commit index 222455528
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:18.447078 C | raft: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: panic: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: goroutine 1 [running]:
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: panic(0xd450a0, 0xc8355fb560)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/usr/local/go/src/runtime/panic.go:481 +0x3e6
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820190620, 0x12375a0, 0x2b, 0xc833422f40, 0x4, 0x4)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).loadState(0xc836564270, 0x1b334, 0x88b67ac3e24bbdb2, 0xd4266e8, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:942 +0x2a2
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.newRaft(0xc8201078d8, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:225 +0x8ff
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.RestartNode(0xc8201078d8, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:213 +0x45
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.restartNode(0xc82006fb80, 0xc820066000, 0x29, 0xc820107d78, 0x0, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:369 +0x7c7
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc82006fb80, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:348 +0x430e
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc8201a8000, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:374 +0x245f
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:116 +0x2101
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main()
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:36 +0x21e
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: main.main()
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/main.go:28 +0x14
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:59:18,481 - Process 450 finished with exit code 2
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:59:18,481 - Sleeping 30 seconds before next try...