search

Quentin-M / etcd-cloud-operator

Deploying and managing production-grade etcd clusters on cloud providers: failure recovery, disaster recovery, backups and resizing.
Apache License 2.0
234 stars 42 forks source link

failed to join the cluster #62

Open jicki opened 3 years ago

jicki commented 3 years ago

error job 

level=error msg="failed to join the cluster" error="failed to add ourselves as a member of the cluster: rpc error: code = Unknown desc = raft: stopped
jicki commented 3 years ago 
time="2021-03-29T08:47:35Z" level=error msg="failed to join the cluster" error="failed to start etcd: listen tcp 172.130.55.215:2380: bind: address already in use"
2021-03-29 08:47:35.461401 W | rafthttp: failed to process raft message (raft: stopped)
2021-03-29 08:47:35.461527 E | rafthttp: failed to find member 3d298e0baea116b7 in cluster dec77a780dc17e55
2021-03-29 08:47:35.462263 E | rafthttp: failed to find member ee032a1060085af5 in cluster dec77a780dc17e55
2021-03-29 08:47:35.483293 E | rafthttp: failed to find member 3d298e0baea116b7 in cluster dec77a780dc17e55
2021-03-29 08:47:35.491297 E | rafthttp: failed to find member ee032a1060085af5 in cluster dec77a780dc17e55
2021-03-29 08:47:35.562004 W | rafthttp: failed to process raft message (raft: stopped)
2021-03-29 08:47:35.581607 W | rafthttp: failed to process raft message (raft: stopped)
2021-03-29 08:47:35.611941 E | rafthttp: failed to find member ee032a1060085af5 in cluster dec77a780dc17e55
2021-03-29 08:47:35.683490 W | rafthttp: failed to process raft message (raft: stopped)
2021-03-29 08:47:35.692528 E | rafthttp: failed to find member 3d298e0baea116b7 in cluster dec77a780dc17e55
2021-03-29 08:47:35.761913 E | rafthttp: failed to find member ee032a1060085af5 in cluster dec77a780dc17e55
2021-03-29 08:47:35.762180 E | rafthttp: failed to find member 3d298e0baea116b7 in cluster dec77a780dc17e55
2021-03-29 08:47:35.762324 W | rafthttp: failed to process raft message (raft: stopped)
2021-03-29 08:47:35.863423 E | rafthttp: failed to find member 3d298e0baea116b7 in cluster dec77a780dc17e55
2021-03-29 08:47:35.865259 E | rafthttp: failed to find member ee032a1060085af5 in cluster dec77a780dc17e55
Quentin-M commented 3 years ago

Hi @jicki,

A bit of a curious error message.

jicki commented 3 years ago

Hi @jicki,

A bit of a curious error message.

  • Are you running a 3 members cluster? On 3 different nodes in an ASG (using the provided Terraform code), otherwise what?
  • What are the other members saying?
  • I do see in your second message that the etcd address is already used locally? What/who is holding this?

yes my running 3 members cluster

pods

kubectl -n demo-etcd get pods
NAME              READY   STATUS    RESTARTS   AGE
etcd-operator-0   1/1     Running   0          2m26s
etcd-operator-1   1/1     Running   0          2m13s
etcd-operator-2   1/1     Running   0          2m1s

pvc

kubectl -n demo-etcd get pvc
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
user-data-etcd-operator-0   Bound    pvc-e3226260-313e-4e6b-8fc4-db853ad88947   10Gi       RWO            nfs-retain     4m44s
user-data-etcd-operator-1   Bound    pvc-5a91311b-db9c-4f5a-bf22-3e2916a13042   10Gi       RWO            nfs-retain     4m31s
user-data-etcd-operator-2   Bound    pvc-a6a494cc-2ed7-42aa-ad5f-c59511ee9189   10Gi       RWO            nfs-retain     4m19s

delete pods

kubectl -n demo-etcd delete pods etcd-operator-1 
pod "etcd-operator-1" deleted

logs etcd-operator-1

kubectl -n demo-etcd logs -f etcd-operator-1 
time="2021-03-30T01:16:25Z" level=info msg="loaded configuration file /etc/eco/eco.yaml"
time="2021-03-30T01:16:25Z" level=info msg="STATUS: Healthy + Not running -> Join"
time="2021-03-30T01:16:25Z" level=info msg="attempting to rejoin cluster under existing identity with local data"
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-30 01:16:25.972790 I | embed: peerTLS: cert = /var/lib/etcd/fixtures/peer/cert.pem, key = /var/lib/etcd/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = 
2021-03-30 01:16:26.006962 I | embed: name = etcd-operator-1
2021-03-30 01:16:26.007001 I | embed: data dir = /var/lib/etcd
2021-03-30 01:16:26.007015 I | embed: member dir = /var/lib/etcd/member
2021-03-30 01:16:26.007024 I | embed: heartbeat = 100ms
2021-03-30 01:16:26.007031 I | embed: election = 1000ms
2021-03-30 01:16:26.007040 I | embed: snapshot count = 100000
2021-03-30 01:16:26.007064 I | embed: advertise client URLs = https://172.130.151.122:2379
2021-03-30 01:16:26.007073 I | embed: initial advertise peer URLs = https://172.130.151.122:2380
2021-03-30 01:16:26.007085 I | embed: initial cluster = 
2021-03-30 01:16:26.007156 W | pkg/fileutil: check file permission: directory "/var/lib/etcd" exist, but the permission is "drwxrwxrwx". The recommended permission is "-rwx------" to prevent possible unprivileged access to the data.
2021-03-30 01:16:26.846878 I | etcdserver: restarting member 573447da2f35d1f in cluster d8e38796de92f4c3 at commit index 4293
raft2021/03/30 01:16:26 INFO: 573447da2f35d1f switched to configuration voters=()
raft2021/03/30 01:16:26 INFO: 573447da2f35d1f became follower at term 2
raft2021/03/30 01:16:26 INFO: newRaft 573447da2f35d1f [peers: [], term: 2, commit: 4293, applied: 0, lastindex: 4293, lastterm: 2]
2021-03-30 01:16:27.002929 W | auth: simple token is not cryptographically signed
2021-03-30 01:16:27.061180 I | etcdserver: starting server... [version: 3.4.13, cluster version: to_be_decided]
raft2021/03/30 01:16:27 INFO: 573447da2f35d1f switched to configuration voters=(2380116157566068054)
2021-03-30 01:16:27.066084 I | etcdserver/membership: added member 2107df08efe4a556 [https://172.130.55.213:2380] to cluster d8e38796de92f4c3
2021-03-30 01:16:27.066110 I | rafthttp: starting peer 2107df08efe4a556...
2021-03-30 01:16:27.066164 I | rafthttp: started HTTP pipelining with peer 2107df08efe4a556
2021-03-30 01:16:27.067419 I | rafthttp: started streaming with peer 2107df08efe4a556 (writer)
2021-03-30 01:16:27.068012 I | rafthttp: started streaming with peer 2107df08efe4a556 (writer)
2021-03-30 01:16:27.069454 I | rafthttp: started peer 2107df08efe4a556
2021-03-30 01:16:27.069523 I | rafthttp: added peer 2107df08efe4a556
2021-03-30 01:16:27.069591 I | rafthttp: started streaming with peer 2107df08efe4a556 (stream Message reader)
2021-03-30 01:16:27.069844 N | etcdserver/membership: set the initial cluster version to 3.4
2021-03-30 01:16:27.069955 I | etcdserver/api: enabled capabilities for version 3.4
raft2021/03/30 01:16:27 INFO: 573447da2f35d1f switched to configuration voters=(2380116157566068054 15333572839742947380)
2021-03-30 01:16:27.070312 I | etcdserver/membership: added member d4cbcb41ca74c034 [https://172.130.61.89:2380] to cluster d8e38796de92f4c3
2021-03-30 01:16:27.070352 I | rafthttp: starting peer d4cbcb41ca74c034...
2021-03-30 01:16:27.070396 I | rafthttp: started HTTP pipelining with peer d4cbcb41ca74c034
2021-03-30 01:16:27.072092 I | rafthttp: started streaming with peer d4cbcb41ca74c034 (writer)
2021-03-30 01:16:27.072171 I | rafthttp: started peer d4cbcb41ca74c034
2021-03-30 01:16:27.072206 I | rafthttp: added peer d4cbcb41ca74c034
2021-03-30 01:16:27.072749 I | rafthttp: started streaming with peer d4cbcb41ca74c034 (writer)
2021-03-30 01:16:27.072794 I | rafthttp: started streaming with peer 2107df08efe4a556 (stream MsgApp v2 reader)
2021-03-30 01:16:27.072846 I | rafthttp: started streaming with peer d4cbcb41ca74c034 (stream MsgApp v2 reader)
raft2021/03/30 01:16:27 INFO: 573447da2f35d1f switched to configuration voters=(392732898906823967 2380116157566068054 15333572839742947380)
2021-03-30 01:16:27.072945 I | etcdserver/membership: added member 573447da2f35d1f [https://172.130.151.86:2380] to cluster d8e38796de92f4c3
2021-03-30 01:16:27.072974 I | rafthttp: started streaming with peer d4cbcb41ca74c034 (stream Message reader)
time="2021-03-30T01:16:27Z" level=info msg="embedded etcd server is now running"
2021-03-30 01:16:27.094719 I | embed: ClientTLS: cert = /opt/etcd/ssl/etcd.pem, key = /opt/etcd/ssl/etcd-key.pem, trusted-ca = /opt/etcd/ssl/ca.pem, client-cert-auth = false, crl-file = 
2021-03-30 01:16:27.094872 I | embed: listening for peers on 172.130.151.122:2380
2021-03-30 01:16:27.095061 I | embed: listening for metrics on http://172.130.151.122:2381
2021-03-30 01:16:27.095111 I | embed: listening for metrics on http://127.0.0.1:2381
2021-03-30 01:16:27.156831 I | rafthttp: peer d4cbcb41ca74c034 became active
2021-03-30 01:16:27.156913 I | rafthttp: established a TCP streaming connection with peer d4cbcb41ca74c034 (stream MsgApp v2 reader)
2021-03-30 01:16:27.224486 I | rafthttp: peer 2107df08efe4a556 became active
2021-03-30 01:16:27.224560 I | rafthttp: established a TCP streaming connection with peer 2107df08efe4a556 (stream MsgApp v2 reader)
2021-03-30 01:16:27.224669 I | rafthttp: established a TCP streaming connection with peer 2107df08efe4a556 (stream Message reader)
2021-03-30 01:16:27.226101 I | rafthttp: established a TCP streaming connection with peer d4cbcb41ca74c034 (stream Message reader)
2021-03-30 01:16:27.269742 I | etcdserver: 573447da2f35d1f initialized peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
raft2021/03/30 01:16:27 INFO: raft.node: 573447da2f35d1f elected leader 2107df08efe4a556 at term 2
2021-03-30 01:16:27.366438 I | etcdserver: published {Name:etcd-operator-1 ClientURLs:[https://172.130.151.122:2379]} to cluster d8e38796de92f4c3
2021-03-30 01:16:27.366496 I | embed: ready to serve client requests
2021-03-30 01:16:27.371519 I | embed: serving client requests on 172.130.151.122:2379
2021-03-30 01:16:39.967624 I | embed: rejected connection from "172.130.61.89:39960" (error "EOF", ServerName "")
time="2021-03-30T01:16:40Z" level=info msg="STATUS: Healthy + Running -> Standby"
raft2021/03/30 01:16:46 INFO: 573447da2f35d1f switched to configuration voters=(2380116157566068054 15333572839742947380)
2021-03-30 01:16:46.337493 I | etcdserver/membership: removed member 573447da2f35d1f from cluster d8e38796de92f4c3
2021-03-30 01:16:46.337779 W | rafthttp: lost the TCP streaming connection with peer d4cbcb41ca74c034 (stream MsgApp v2 reader)
2021-03-30 01:16:46.337868 W | rafthttp: lost the TCP streaming connection with peer 2107df08efe4a556 (stream MsgApp v2 reader)
2021-03-30 01:16:46.338683 W | rafthttp: lost the TCP streaming connection with peer d4cbcb41ca74c034 (stream Message reader)
2021-03-30 01:16:46.340022 W | rafthttp: lost the TCP streaming connection with peer 2107df08efe4a556 (stream Message reader)
2021-03-30 01:16:46.345448 E | rafthttp: failed to write 2107df08efe4a556 on pipeline (unexpected EOF)
2021-03-30 01:16:46.345484 I | rafthttp: peer 2107df08efe4a556 became inactive (message send to peer failed)
2021-03-30 01:16:46.386799 E | rafthttp: failed to dial d4cbcb41ca74c034 on stream MsgApp v2 (the member has been permanently removed from the cluster)
2021-03-30 01:16:46.386830 I | rafthttp: peer d4cbcb41ca74c034 became inactive (message send to peer failed)
2021-03-30 01:16:46.386882 E | etcdserver: the member has been permanently removed from the cluster
2021-03-30 01:16:46.386909 I | etcdserver: the data-dir used by this member must be removed.
2021-03-30 01:16:46.388060 I | rafthttp: stopping peer 2107df08efe4a556...
2021-03-30 01:16:46.388103 I | rafthttp: stopped streaming with peer 2107df08efe4a556 (writer)
2021-03-30 01:16:46.388142 I | rafthttp: stopped streaming with peer 2107df08efe4a556 (writer)
2021-03-30 01:16:46.388190 I | rafthttp: stopped HTTP pipelining with peer 2107df08efe4a556
2021-03-30 01:16:46.388249 I | rafthttp: stopped streaming with peer 2107df08efe4a556 (stream MsgApp v2 reader)
2021-03-30 01:16:46.388283 I | rafthttp: stopped streaming with peer 2107df08efe4a556 (stream Message reader)
2021-03-30 01:16:46.388293 I | rafthttp: stopped peer 2107df08efe4a556
2021-03-30 01:16:46.388301 I | rafthttp: stopping peer d4cbcb41ca74c034...
2021-03-30 01:16:46.388323 I | rafthttp: stopped streaming with peer d4cbcb41ca74c034 (writer)
2021-03-30 01:16:46.388338 I | rafthttp: stopped streaming with peer d4cbcb41ca74c034 (writer)
2021-03-30 01:16:46.388382 I | rafthttp: stopped HTTP pipelining with peer d4cbcb41ca74c034
2021-03-30 01:16:46.388413 I | rafthttp: stopped streaming with peer d4cbcb41ca74c034 (stream MsgApp v2 reader)
2021-03-30 01:16:46.388435 I | rafthttp: stopped streaming with peer d4cbcb41ca74c034 (stream Message reader)
2021-03-30 01:16:46.388450 I | rafthttp: stopped peer d4cbcb41ca74c034
time="2021-03-30T01:16:46Z" level=warning msg="etcd server is stopping"
1.6170670073679156e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-d02ff7bc-6397-4bc4-8374-dd3ba70f7a32/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 01:16:49.865472 I | embed: rejected connection from "172.130.55.213:45440" (error "EOF", ServerName "")
1.6170670123684206e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-d02ff7bc-6397-4bc4-8374-dd3ba70f7a32/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
time="2021-03-30T01:16:55Z" level=info msg="STATUS: Healthy + Not running -> Join"
1.6170670156336913e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-919bb5ff-9235-462c-aede-a662f695ec12/172.130.61.89:2379", "attempt": 0, "error": "rpc error: code = Unknown desc = raft: stopped"}
time="2021-03-30T01:16:55Z" level=error msg="failed to join the cluster" error="failed to add ourselves as a member of the cluster: rpc error: code = Unknown desc = raft: stopped"
1.6170670173688858e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-d02ff7bc-6397-4bc4-8374-dd3ba70f7a32/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
1.6170670223696744e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-d02ff7bc-6397-4bc4-8374-dd3ba70f7a32/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.86:2379: i/o timeout\""}
2021-03-30 01:17:04.865899 I | embed: rejected connection from "172.130.55.213:45690" (error "EOF", ServerName "")
1.6170670273700879e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-d02ff7bc-6397-4bc4-8374-dd3ba70f7a32/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.86:2379: i/o timeout\""}
time="2021-03-30T01:17:10Z" level=info msg="STATUS: Healthy + Not running -> Join"
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-30 01:17:10.592319 I | embed: peerTLS: cert = /var/lib/etcd/fixtures/peer/cert.pem, key = /var/lib/etcd/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = 
1.6170670305989053e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-a0b211bf-6874-4053-aac7-db0ab47dc85d/172.130.61.89:2379", "attempt": 0, "error": "rpc error: code = Unknown desc = raft: stopped"}
1.6170670306201253e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-a0b211bf-6874-4053-aac7-db0ab47dc85d/172.130.61.89:2379", "attempt": 0, "error": "rpc error: code = Unknown desc = raft: stopped"}
time="2021-03-30T01:17:10Z" level=error msg="failed to join the cluster" error="failed to start etcd: listen tcp 172.130.151.122:2380: bind: address already in use"
2021-03-30 01:17:10.645065 E | rafthttp: failed to find member d4cbcb41ca74c034 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.656916 E | rafthttp: failed to find member d4cbcb41ca74c034 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.658378 W | rafthttp: failed to process raft message (raft: stopped)
2021-03-30 01:17:10.671906 E | rafthttp: failed to find member 2107df08efe4a556 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.672779 E | rafthttp: failed to find member 2107df08efe4a556 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.773059 E | rafthttp: failed to find member d4cbcb41ca74c034 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.845863 E | rafthttp: failed to find member d4cbcb41ca74c034 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.853979 E | rafthttp: failed to find member 2107df08efe4a556 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.860586 E | rafthttp: failed to find member 2107df08efe4a556 in cluster d8e38796de92f4c3
2021-03-30 01:17:10.885153 W | rafthttp: failed to process raft message (raft: stopped)
2021-03-30 01:17:10.921513 W | rafthttp: failed to process raft message (raft: stopped)
.....
time="2021-03-30T01:17:25Z" level=error msg="failed to join the cluster" error="failed to add ourselves as a member of the cluster: rpc error: code = Unknown desc = raft: stopped"
1.6170670455224566e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-504e129c-7371-424d-96fd-eb94c6d257d8/172.130.61.89:2379", "attempt": 0, "error": "rpc error: code = Unknown desc = raft: stopped"}

logs etcd-operator-0

time="2021-03-30T01:16:09Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:16:18.473725 W | rafthttp: lost the TCP streaming connection with peer 573447da2f35d1f (stream MsgApp v2 reader)
2021-03-30 01:16:18.476112 W | rafthttp: lost the TCP streaming connection with peer 573447da2f35d1f (stream Message reader)
2021-03-30 01:16:18.548695 E | rafthttp: failed to dial 573447da2f35d1f on stream MsgApp v2 (peer 573447da2f35d1f failed to find local node d4cbcb41ca74c034)
2021-03-30 01:16:18.548713 I | rafthttp: peer 573447da2f35d1f became inactive (message send to peer failed)
2021-03-30 01:16:19.418600 I | embed: rejected connection from "10.9.9.48:24871" (error "read tcp 172.130.61.89:2379->10.9.9.48:24871: read: connection reset by peer", ServerName "")
2021-03-30 01:16:23.869174 W | rafthttp: lost the TCP streaming connection with peer 573447da2f35d1f (stream MsgApp v2 writer)
2021-03-30 01:16:23.869621 W | rafthttp: lost the TCP streaming connection with peer 573447da2f35d1f (stream Message writer)
time="2021-03-30T01:16:24Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:16:27.118410 I | embed: rejected connection from "10.9.9.48:30105" (error "read tcp 172.130.61.89:2379->10.9.9.48:30105: read: connection reset by peer", ServerName "")
2021-03-30 01:16:27.156575 I | rafthttp: peer 573447da2f35d1f became active
2021-03-30 01:16:27.156607 I | rafthttp: established a TCP streaming connection with peer 573447da2f35d1f (stream MsgApp v2 writer)
2021-03-30 01:16:27.224808 I | rafthttp: established a TCP streaming connection with peer 573447da2f35d1f (stream Message writer)
2021-03-30 01:16:27.974920 E | rafthttp: failed to dial 573447da2f35d1f on stream MsgApp v2 (dial tcp 172.130.151.86:2380: i/o timeout)
2021-03-30 01:16:27.974936 I | rafthttp: peer 573447da2f35d1f became inactive (message send to peer failed)
1.6170669882369318e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-cdf76436-0832-4c1a-8044-b771dbef19a2/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 01:16:29.095730 I | embed: rejected connection from "172.130.55.192:15923" (error "EOF", ServerName "")
2021-03-30 01:16:30.422564 I | embed: rejected connection from "10.9.9.48:43183" (error "read tcp 172.130.61.89:2379->10.9.9.48:43183: read: connection reset by peer", ServerName "")
1.6170669932371874e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-cdf76436-0832-4c1a-8044-b771dbef19a2/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 01:16:36.774647 I | embed: rejected connection from "10.9.9.48:51354" (error "EOF", ServerName "")
1.617066998237407e+09   warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-cdf76436-0832-4c1a-8044-b771dbef19a2/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
time="2021-03-30T01:16:39Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:16:42.203827 W | rafthttp: health check for peer 573447da2f35d1f could not connect: dial tcp 172.130.151.86:2380: i/o timeout
2021-03-30 01:16:42.203867 W | rafthttp: health check for peer 573447da2f35d1f could not connect: dial tcp 172.130.151.86:2380: i/o timeout
1.6170670032375584e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-cdf76436-0832-4c1a-8044-b771dbef19a2/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.86:2379: i/o timeout\""}
raft2021/03/30 01:16:46 INFO: d4cbcb41ca74c034 switched to configuration voters=(2380116157566068054 15333572839742947380)
2021-03-30 01:16:46.336242 I | etcdserver/membership: removed member 573447da2f35d1f from cluster d8e38796de92f4c3
2021-03-30 01:16:46.336257 I | rafthttp: stopping peer 573447da2f35d1f...
2021-03-30 01:16:46.336803 I | rafthttp: closed the TCP streaming connection with peer 573447da2f35d1f (stream MsgApp v2 writer)
2021-03-30 01:16:46.336818 I | rafthttp: stopped streaming with peer 573447da2f35d1f (writer)
2021-03-30 01:16:46.341113 I | rafthttp: closed the TCP streaming connection with peer 573447da2f35d1f (stream Message writer)
2021-03-30 01:16:46.341136 I | rafthttp: stopped streaming with peer 573447da2f35d1f (writer)
2021-03-30 01:16:46.341572 I | rafthttp: stopped HTTP pipelining with peer 573447da2f35d1f
2021-03-30 01:16:46.341604 I | rafthttp: stopped streaming with peer 573447da2f35d1f (stream MsgApp v2 reader)
2021-03-30 01:16:46.341622 I | rafthttp: stopped streaming with peer 573447da2f35d1f (stream Message reader)
2021-03-30 01:16:46.341633 I | rafthttp: stopped peer 573447da2f35d1f
2021-03-30 01:16:46.341648 I | rafthttp: removed peer 573447da2f35d1f
2021-03-30 01:16:46.385245 W | rafthttp: rejected the stream from peer 573447da2f35d1f since it was removed
1.6170670082377918e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-cdf76436-0832-4c1a-8044-b771dbef19a2/172.130.151.86:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.86:2379: i/o timeout\""}
time="2021-03-30T01:16:48Z" level=info msg="removing member \"etcd-operator-1\" that's been unhealthy for 30s"
raft2021/03/30 01:16:48 INFO: d4cbcb41ca74c034 switched to configuration voters=(2380116157566068054 15333572839742947380)
1.6170670082805386e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-1ab04957-f7bb-45c1-9d64-9bc17137ef7b/172.130.61.89:2379", "attempt": 0, "error": "rpc error: code = NotFound desc = etcdserver: member not found"}
time="2021-03-30T01:16:54Z" level=info msg="STATUS: Healthy + Running -> Standby"
time="2021-03-30T01:17:09Z" level=info msg="STATUS: Healthy + Running -> Standby"
raft2021/03/30 01:17:10 INFO: d4cbcb41ca74c034 switched to configuration voters=(2380116157566068054 15333572839742947380 17335131455808352196)
2021-03-30 01:17:10.162275 I | etcdserver/membership: added member f092c236ae282fc4 [https://172.130.151.122:2380] to cluster d8e38796de92f4c3
2021-03-30 01:17:10.162298 I | rafthttp: starting peer f092c236ae282fc4...
2021-03-30 01:17:10.162332 I | rafthttp: started HTTP pipelining with peer f092c236ae282fc4
2021-03-30 01:17:10.163998 I | rafthttp: started streaming with peer f092c236ae282fc4 (writer)
2021-03-30 01:17:10.165136 I | rafthttp: started streaming with peer f092c236ae282fc4 (writer)
2021-03-30 01:17:10.166524 I | rafthttp: started peer f092c236ae282fc4
2021-03-30 01:17:10.166680 I | rafthttp: started streaming with peer f092c236ae282fc4 (stream Message reader)
2021-03-30 01:17:10.166793 I | rafthttp: started streaming with peer f092c236ae282fc4 (stream MsgApp v2 reader)
2021-03-30 01:17:10.167203 I | rafthttp: added peer f092c236ae282fc4
time="2021-03-30T01:17:25Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:17:36.774532 I | embed: rejected connection from "10.9.9.48:52470" (error "EOF", ServerName "")
time="2021-03-30T01:17:39Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:17:45.167442 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:17:50.167592 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:17:54Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:17:55.167751 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:00.167902 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:05.168073 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:18:09Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:10.168215 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:15.168379 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:20.168526 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:18:24Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:25.168681 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:30.168826 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:34.845063 I | embed: rejected connection from "172.130.55.213:42352" (error "EOF", ServerName "")
2021-03-30 01:18:35.168982 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:36.774508 I | embed: rejected connection from "10.9.9.48:55210" (error "EOF", ServerName "")
time="2021-03-30T01:18:39Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:40.169201 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:45.169357 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:50.169501 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:18:54Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:55.169648 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:00.169807 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:05.169963 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:19:09Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:10.170139 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:15.170280 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:20.170464 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:19:24Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:25.170597 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:30.170763 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:35.170916 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:36.774644 I | embed: rejected connection from "10.9.9.48:57958" (error "EOF", ServerName "")
time="2021-03-30T01:19:39Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:40.171044 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:45.171195 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:50.171353 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:19:54Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:55.171526 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:00.171683 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:05.171878 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:20:09Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:10.172041 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:15.172221 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:20.172419 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:20:24Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:25.172597 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:30.172790 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:34.847300 I | embed: rejected connection from "172.130.55.213:51870" (error "EOF", ServerName "")
2021-03-30 01:20:35.172987 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:36.774541 I | embed: rejected connection from "10.9.9.48:60700" (error "EOF", ServerName "")
time="2021-03-30T01:20:39Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:40.173187 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:45.173358 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:50.173536 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:20:54Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:55.173721 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:00.173878 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error

logs etcd-operator-2

2021-03-30 01:17:10.193453 I | rafthttp: started streaming with peer f092c236ae282fc4 (stream Message reader)
time="2021-03-30T01:17:19Z" level=info msg="STATUS: Healthy + Running -> Standby"
time="2021-03-30T01:17:34Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:17:39.991120 I | embed: rejected connection from "172.130.61.89:60872" (error "EOF", ServerName "")
2021-03-30 01:17:44.484998 I | embed: rejected connection from "10.9.9.49:49004" (error "EOF", ServerName "")
2021-03-30 01:17:45.193959 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:17:45.194015 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:17:49Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:17:50.194275 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:17:50.194366 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:17:54.992506 I | embed: rejected connection from "172.130.61.89:33328" (error "EOF", ServerName "")
2021-03-30 01:17:55.194655 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:17:55.194734 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:00.194997 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:00.195050 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:18:04Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:05.195339 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:05.195413 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:09.994961 I | embed: rejected connection from "172.130.61.89:34008" (error "EOF", ServerName "")
2021-03-30 01:18:10.175480 I | embed: rejected connection from "172.130.151.122:35742" (error "EOF", ServerName "")
2021-03-30 01:18:10.195724 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:10.195777 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:15.196035 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:15.196115 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:18:19Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:19.880835 I | embed: rejected connection from "172.130.55.213:43020" (error "EOF", ServerName "")
2021-03-30 01:18:20.196317 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:20.196390 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:24.997107 I | embed: rejected connection from "172.130.61.89:34692" (error "EOF", ServerName "")
2021-03-30 01:18:25.184443 I | embed: rejected connection from "172.130.151.122:36076" (error "EOF", ServerName "")
2021-03-30 01:18:25.196638 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:25.196795 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:30.197591 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:30.197666 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:18:34Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:35.197897 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:35.198028 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:40.198271 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:40.198342 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:40.217402 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:18:40.217525 W | etcdserver: not enough started members, rejecting member add {ID:e2c105ebabbe332a RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:18:44.485323 I | embed: rejected connection from "10.9.9.49:53758" (error "EOF", ServerName "")
2021-03-30 01:18:45.198548 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:45.198610 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:18:49Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:18:50.198845 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:50.198933 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:55.199051 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:18:55.199182 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:00.199307 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:00.199332 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:19:04Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:05.199612 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:05.199712 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:10.000760 I | embed: rejected connection from "172.130.61.89:36762" (error "EOF", ServerName "")
2021-03-30 01:19:10.199911 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:10.200053 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:10.225730 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:19:10.225869 W | etcdserver: not enough started members, rejecting member add {ID:a712ae29ab30d7b1 RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:19:15.200289 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:15.200456 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:19:19Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:20.200557 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:20.200601 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:25.008565 I | embed: rejected connection from "172.130.61.89:37442" (error "EOF", ServerName "")
2021-03-30 01:19:25.186679 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:19:25.186853 W | etcdserver: not enough started members, rejecting member add {ID:a197287f4f0cad70 RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:19:25.200880 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:25.200974 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:30.201297 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:30.201372 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:19:34Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:35.201544 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:35.201582 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:40.204538 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:40.204599 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:40.214848 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:19:40.214958 W | etcdserver: not enough started members, rejecting member add {ID:e07fde3d4dd7199d RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:19:44.484952 I | embed: rejected connection from "10.9.9.49:58504" (error "EOF", ServerName "")
2021-03-30 01:19:45.204741 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:45.204821 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:19:49Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:19:50.205079 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:50.205203 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:55.005799 I | embed: rejected connection from "172.130.61.89:38816" (error "EOF", ServerName "")
2021-03-30 01:19:55.198775 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:19:55.198989 W | etcdserver: not enough started members, rejecting member add {ID:2d2ab99a4ae65e85 RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:19:55.205377 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:19:55.205531 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:00.205614 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:00.205806 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:20:04Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:05.205937 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:05.206011 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:10.004395 I | embed: rejected connection from "172.130.61.89:39500" (error "EOF", ServerName "")
2021-03-30 01:20:10.206286 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:10.206348 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:15.206613 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:15.206716 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:20:19Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:19.879359 I | embed: rejected connection from "172.130.55.213:52512" (error "EOF", ServerName "")
2021-03-30 01:20:20.206971 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:20.207045 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:25.207355 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:25.207441 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:30.207697 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:30.207795 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:20:34Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:35.207992 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:35.208031 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:40.208257 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:40.208328 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:44.485327 I | embed: rejected connection from "10.9.9.49:35054" (error "EOF", ServerName "")
2021-03-30 01:20:45.208658 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:45.208759 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:20:49Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:20:50.208996 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:50.209142 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:55.174336 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:20:55.174430 W | etcdserver: not enough started members, rejecting member add {ID:77595d6157d31391 RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:20:55.209301 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:20:55.209411 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:00.209564 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:00.209602 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:21:04Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:21:05.209863 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:05.209972 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:10.158782 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:21:10.158880 W | etcdserver: not enough started members, rejecting member add {ID:641228db1fafcd2d RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:21:10.210252 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:10.210341 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:15.210630 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:15.211016 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
time="2021-03-30T01:21:19Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:21:20.210983 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:20.211444 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:24.974626 I | embed: rejected connection from "172.130.61.89:42936" (error "EOF", ServerName "")
2021-03-30 01:21:25.211277 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:25.211636 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:28.380618 I | embed: rejected connection from "10.9.9.49:35164" (error "read tcp 172.130.55.213:2379->10.9.9.49:35164: read: connection timed out", ServerName "")
2021-03-30 01:21:30.211637 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:30.212044 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:30.428558 I | embed: rejected connection from "10.9.9.49:50978" (error "read tcp 172.130.55.213:2379->10.9.9.49:50978: read: connection timed out", ServerName "")
time="2021-03-30T01:21:34Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 01:21:35.212005 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:35.212414 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:36.572477 I | embed: rejected connection from "172.130.61.64:4238" (error "read tcp 172.130.55.213:2379->172.130.61.64:4238: read: connection timed out", ServerName "")
2021-03-30 01:21:39.969046 I | embed: rejected connection from "172.130.61.89:43620" (error "EOF", ServerName "")
2021-03-30 01:21:40.173880 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)
2021-03-30 01:21:40.173988 W | etcdserver: not enough started members, rejecting member add {ID:631050ed5f222514 RaftAttributes:{PeerURLs:[https://172.130.151.122:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 01:21:40.212297 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error
2021-03-30 01:21:40.212685 W | rafthttp: health check for peer f092c236ae282fc4 could not connect: remote error: tls: internal error

config

eco:
  # The interval between each cluster verification by the operator.
  check-interval: 15s
  # The time after which, an unhealthy member will be removed from the cluster.
  unhealthy-member-ttl: 30s
  # Defines whether the operator will attempt to seed a new cluster from a
  # snapshot after the managed cluster has lost quorum.
  auto-disaster-recovery: true
  # Configuration of the etcd instance.
  etcd:
    # The address that clients should use to connect to the etcd cluster (i.e.
    # load balancer public address - hostname only, no schema or port number).
    advertise-address: 
    # The directory where the etcd data is stored.
    data-dir: /var/lib/etcd
    # The TLS configuration for clients communication.
    client-transport-security:
      auto-tls: false
      cert-file: /opt/etcd/ssl/etcd.pem
      key-file: /opt/etcd/ssl/etcd-key.pem
      trusted-ca-file: /opt/etcd/ssl/ca.pem
      client-cert-auth: false
    # The TLS configuration for peers communications.
    peer-transport-security:
      auto-tls: true
      cert-file: 
      key-file: 
      trusted-ca-file: 
      peer-client-cert-auth: false
    # Defines the maximum amount of data that etcd can store, in bytes, before going into maintenance mode
    backend-quota: 2147483648
    # Defines the auto-compaction policy (set retention to 0 to disable).
    auto-compaction-mode: periodic
    auto-compaction-retention: "0"
    # Defines the initial acl that will be applied to the etcd during provisioning.
  # Configuration of the auto-scaling group provider.
  asg:
    provider: sts
  # Configuration of the snapshot provider.
  snapshot:
    provider: file
    # The interval between each snapshot.
    interval: 30m
    # The time after which a backup has to be deleted.
    ttl: 24h
Quentin-M commented 3 years ago

Hi again,

Thanks for all the details..! I don't really test the Kubernetes integration as it was a contribution from another developer.. not really surprised to see a few issues there.

I see two potential problems here:

jicki commented 3 years ago

Hi again,

Thanks for all the details..! I don't really test the Kubernetes integration as it was a contribution from another developer.. not really surprised to see a few issues there.

  • Seems that etcd-operator-1 starts up 01:16:25, after etcd-operator-0 lost its connection to it around 01:16:18. I assume this was due to the kubectl delete pod etcd-operator-1, correct?
  • The new etcd-operator-1 became ready quickly at 01:16:27, as witnessed by etcd-operator-0at that time - but the operator itself ended up kicking etcd-operator-1 at 01:16:48after it appeared unhealthy for 30s.

I see two potential problems here:

  • I think that unhealthy-member-ttl: 30s might be too short given that the member cleaning job's interval is set to 15s by default.. Actually it does not hurt much to increase that TTL a bit as it only blocks new members (with empty data) from re-joining after another member died.. so making full reconciliation a bit slower. Setting it to 3min sounds fine to me.
  • I think that the etcd server in etcd-operator-1 did not fully stop by 01:17:10 (that's why we see the bind failure) - either it's still in progress or it's stuck - so that prevents etcd-operator-1 from coming back inside the cluster as we would expect. I believe that the "raft: stopped" message actually comes from the fact that the operator is trying to re-join, but it sends the message to any of the three endpoints behind the client.. and it ends up hitting the local etcd server which is stopped but still listening, which is quite insane. At least, we can see etcd-operator-1 is smart enough to understand it needs to reset its data to re-join. I know that the embed etcd server is not the greatest when it comes to clean shutdown. The operator's error watcher caught the etcd message about itself stopping but we don't actually act upon it further.. We may want to change the watcher to wait for a certain grace period (or somehow check if etcd is stopped yet or not) and then issue a force stop with c.Stop(false, false) rather than just doing c.isRunning = false. The idea would be to try fully killing it. On EC2, we have a safeguard for that kind of unclean etcd shutdown.. if the local etcd is not available for more than 15min, we shutdown the instance completely. We could also implement a change to the statefulset to actually do just that.. have a custom health check, and if the local etcd is dead for too long, restart the pod completely. That would be quite ideal, if we are not sure that even with c.Stop(false, false), we would be able to properly stop etcd completely.

Hi
Very strange, if i use 

      volumes:
      - name: data
        emptyDir: {}

Cluster join OK

kubectl -n demo-etcd delete pods etcd-operator-1
pod "etcd-operator-1" deleted


* etcd-operator-2 logs
```bash
2021-03-30 03:38:03.797182 I | embed: rejected connection from "10.9.9.49:38986" (error "EOF", ServerName "")
time="2021-03-30T03:38:08Z" level=info msg="STATUS: Healthy + Running -> Standby"
time="2021-03-30T03:38:23Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 03:38:27.866023 I | embed: rejected connection from "172.130.61.101:45798" (error "EOF", ServerName "")
time="2021-03-30T03:38:38Z" level=info msg="STATUS: Healthy + Running -> Standby"
time="2021-03-30T03:38:53Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 03:39:03.797294 I | embed: rejected connection from "10.9.9.49:40010" (error "EOF", ServerName "")
time="2021-03-30T03:39:09Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 03:39:12.867150 I | embed: rejected connection from "172.130.61.101:46074" (error "EOF", ServerName "")
2021-03-30 03:39:14.881968 I | embed: rejected connection from "172.130.61.64:19741" (error "EOF", ServerName "")
2021-03-30 03:39:16.377053 W | rafthttp: lost the TCP streaming connection with peer 3805fb251c7e3ce8 (stream MsgApp v2 reader)
2021-03-30 03:39:16.377219 W | rafthttp: lost the TCP streaming connection with peer 3805fb251c7e3ce8 (stream Message reader)
2021-03-30 03:39:16.384852 E | rafthttp: failed to dial 3805fb251c7e3ce8 on stream Message (EOF)
2021-03-30 03:39:16.384991 I | rafthttp: peer 3805fb251c7e3ce8 became inactive (message send to peer failed)
2021-03-30 03:39:16.457580 W | rafthttp: lost the TCP streaming connection with peer 3805fb251c7e3ce8 (stream MsgApp v2 writer)
2021-03-30 03:39:16.602742 W | rafthttp: lost the TCP streaming connection with peer 3805fb251c7e3ce8 (stream Message writer)
2021-03-30 03:39:21.828763 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:21.828832 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
time="2021-03-30T03:39:23Z" level=info msg="STATUS: Healthy + Running -> Standby"
2021-03-30 03:39:25.856688 W | etcdserver: read-only range request "key:\"key1130\" " with result "range_response_count:1 size:37" took too long (541.199757ms) to execute
2021-03-30 03:39:27.891204 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:27.891240 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
1.617075569606453e+09   warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-523c0dca-b889-498e-9b7d-a34c1c379f2a/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 03:39:32.866055 W | etcdserver: read-only range request "key:\"key1216\" " with result "range_response_count:1 size:37" took too long (103.197164ms) to execute
2021-03-30 03:39:33.246894 W | etcdserver: read-only range request "key:\"key1219\" " with result "range_response_count:1 size:37" took too long (122.110531ms) to execute
2021-03-30 03:39:33.894591 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:33.894633 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
1.6170755746068401e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-523c0dca-b889-498e-9b7d-a34c1c379f2a/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
time="2021-03-30T03:39:38Z" level=info msg="STATUS: Healthy + Running -> Standby"
1.6170755796071749e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-523c0dca-b889-498e-9b7d-a34c1c379f2a/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 03:39:39.895827 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:39.895880 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:42.871356 I | embed: rejected connection from "172.130.61.101:46400" (error "EOF", ServerName "")
1.6170755846075819e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-523c0dca-b889-498e-9b7d-a34c1c379f2a/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.74:2379: i/o timeout\""}
2021-03-30 03:39:45.217218 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:39:45.218081 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:39:45.897660 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:45.897720 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
1.6170755896080518e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-523c0dca-b889-498e-9b7d-a34c1c379f2a/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.74:2379: i/o timeout\""}
2021-03-30 03:39:50.217569 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:39:50.218354 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:39:51.907783 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:51.907855 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
time="2021-03-30T03:39:53Z" level=info msg="STATUS: Healthy + Running -> Standby"
1.6170755946557484e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-000c5d16-7f6d-4fc0-9833-cbdcc8632cf4/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 03:39:55.168102 W | etcdserver: not healthy for reconfigure, rejecting member add {ID:686b2644b04a5908 RaftAttributes:{PeerURLs:[https://172.130.151.119:2380] IsLearner:false} Attributes:{Name: ClientURLs:[]}}
2021-03-30 03:39:55.217910 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:39:55.218708 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:39:57.873308 I | embed: rejected connection from "172.130.61.101:46536" (error "EOF", ServerName "")
2021-03-30 03:39:57.910001 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:39:57.910036 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
1.6170755996561596e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-000c5d16-7f6d-4fc0-9833-cbdcc8632cf4/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 03:40:00.218252 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:40:00.218985 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:40:03.797380 I | embed: rejected connection from "10.9.9.49:41438" (error "EOF", ServerName "")
2021-03-30 03:40:03.912293 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:40:03.912391 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
1.6170756046566145e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-000c5d16-7f6d-4fc0-9833-cbdcc8632cf4/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
2021-03-30 03:40:05.218613 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:40:05.219334 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
time="2021-03-30T03:40:08Z" level=info msg="STATUS: Healthy + Running -> Standby"
1.6170756096569276e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-000c5d16-7f6d-4fc0-9833-cbdcc8632cf4/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.74:2379: i/o timeout\""}
2021-03-30 03:40:09.914009 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:40:09.914040 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:40:10.218931 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:40:10.219625 W | rafthttp: health check for peer 3805fb251c7e3ce8 could not connect: dial tcp 172.130.151.74:2380: i/o timeout
2021-03-30 03:40:12.870126 I | embed: rejected connection from "172.130.61.101:46668" (error "EOF", ServerName "")
1.6170756146573615e+09  warn    clientv3/retry_interceptor.go:62    retrying of unary invoker failed    {"target": "endpoint://client-000c5d16-7f6d-4fc0-9833-cbdcc8632cf4/172.130.151.74:2379", "attempt": 0, "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.130.151.74:2379: i/o timeout\""}
time="2021-03-30T03:40:14Z" level=info msg="removing member \"etcd-operator-1\" that's been unhealthy for 1m0s"
raft2021/03/30 03:40:14 INFO: 582dc87539c99702 switched to configuration voters=(6353955055095879426 12299822546650219187)
2021-03-30 03:40:14.762882 I | etcdserver/membership: removed member 3805fb251c7e3ce8 from cluster f50188c8af6a2f58
2021-03-30 03:40:14.762921 I | rafthttp: stopping peer 3805fb251c7e3ce8...
2021-03-30 03:40:14.762974 I | rafthttp: stopped streaming with peer 3805fb251c7e3ce8 (writer)
2021-03-30 03:40:14.762999 I | rafthttp: stopped streaming with peer 3805fb251c7e3ce8 (writer)
2021-03-30 03:40:14.770072 I | rafthttp: stopped HTTP pipelining with peer 3805fb251c7e3ce8
2021-03-30 03:40:14.770197 I | rafthttp: stopped streaming with peer 3805fb251c7e3ce8 (stream MsgApp v2 reader)
2021-03-30 03:40:14.770239 I | rafthttp: stopped streaming with peer 3805fb251c7e3ce8 (stream Message reader)
2021-03-30 03:40:14.770267 I | rafthttp: stopped peer 3805fb251c7e3ce8
2021-03-30 03:40:14.770301 I | rafthttp: removed peer 3805fb251c7e3ce8
2021-03-30 03:40:15.956059 W | etcdserver: failed to reach the peerURL(https://172.130.151.74:2380) of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
2021-03-30 03:40:15.956116 W | etcdserver: cannot get the version of member 3805fb251c7e3ce8 (Get https://172.130.151.74:2380/version: dial tcp 172.130.151.74:2380: i/o timeout)
raft2021/03/30 03:40:18 INFO: 582dc87539c99702 switched to configuration voters=(6353955055095879426 12299822546650219187)
time="2021-03-30T03:40:23Z" level=info msg="STATUS: Healthy + Running -> Standby"
raft2021/03/30 03:40:25 INFO: 582dc87539c99702 switched to configuration voters=(500589706128054456 6353955055095879426 12299822546650219187)
2021-03-30 03:40:25.152922 I | etcdserver/membership: added member 6f273af7bfad8b8 [https://172.130.151.119:2380] to cluster f50188c8af6a2f58
2021-03-30 03:40:25.152974 I | rafthttp: starting peer 6f273af7bfad8b8...
2021-03-30 03:40:25.153039 I | rafthttp: started HTTP pipelining with peer 6f273af7bfad8b8
2021-03-30 03:40:25.154215 I | rafthttp: started streaming with peer 6f273af7bfad8b8 (writer)
2021-03-30 03:40:25.159405 I | rafthttp: started streaming with peer 6f273af7bfad8b8 (writer)
2021-03-30 03:40:25.169689 I | rafthttp: started peer 6f273af7bfad8b8
2021-03-30 03:40:25.169809 I | rafthttp: added peer 6f273af7bfad8b8
2021-03-30 03:40:25.169972 I | rafthttp: started streaming with peer 6f273af7bfad8b8 (stream MsgApp v2 reader)
2021-03-30 03:40:25.170480 I | rafthttp: started streaming with peer 6f273af7bfad8b8 (stream Message reader)
2021-03-30 03:40:25.740733 I | rafthttp: peer 6f273af7bfad8b8 became active
2021-03-30 03:40:25.741193 I | rafthttp: established a TCP streaming connection with peer 6f273af7bfad8b8 (stream MsgApp v2 reader)
2021-03-30 03:40:25.741401 I | rafthttp: established a TCP streaming connection with peer 6f273af7bfad8b8 (stream Message reader)
2021-03-30 03:40:25.777996 I | rafthttp: established a TCP streaming connection with peer 6f273af7bfad8b8 (stream Message writer)
2021-03-30 03:40:25.852040 I | rafthttp: established a TCP streaming connection with peer 6f273af7bfad8b8 (stream MsgApp v2 writer)
time="2021-03-30T03:40:38Z" level=info msg="STATUS: Healthy + Running -> Standby"
Quentin-M commented 3 years ago

Hi @jicki,

Yeah, it's just a timing issue. In those new logs with TTL=1min, the unhealthy member gets removed at 3:40:15, and new rejoins at 03:40:25 - which is fine.

Let me give you more context on the unhealthy-member-ttl.