search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.84k stars 9.77k forks source link

etcd quorum formation fails in kubernetes environment #12327

Closed bharath-b23 closed 3 years ago

bharath-b23 commented 4 years ago

Hello,

I am trying to deploy 3 instances of etcd using kubernetes, with TLS enabled. But the instances are not enable to connect with each other to form a quorum, as the requests are getting rejected with below error.

{"level":"warn","ts":"2020-09-22T15:29:35.644Z","caller":"embed/config_logging.go:270","msg":"rejected connection","remote-addr":"10.233.118.32:52394","server-name":"etcd1","ip-addresses":[],"dns-names":["etcd","etcd1","etcd2","etcd3"],"error":"tls: \"10.233.118.32\" does not match any of DNSNames [\"etcd\" \"etcd1\" \"etcd2\" \"etcd3\"] (lookup etcd on 169.254.25.10:53: server misbehaving)"}

etcd3 in the SAN list resolves to 10.233.118.32 IP address and error is reported by etcd1.

Environment Details:

etcd1 conf:

name: etcd1
data-dir: /opt/etcd/data
wal-dir:
snapshot-count: 10000
heartbeat-interval: 100
election-timeout: 1000
listen-peer-urls: https://0.0.0.0:2380
listen-client-urls: https://0.0.0.0:2379
max-snapshots: 5
max-wals: 5
cors:
quota-backend-bytes: 0
backend-batch-limit: 0
backend-bbolt-freelist-type: map
backend-batch-interval: 0
max-txn-ops: 128
max-request-bytes: 1572864
grpc-keepalive-min-time: 5
grpc-keepalive-interval: 2
grpc-keepalive-timeout: 20
initial-advertise-peer-urls: https://etcd1:2380
initial-cluster: etcd1=https://etcd1:2380,etcd2=https://etcd2:2380,etcd3=https://etcd3:2380
initial-cluster-state: new
initial-cluster-token: odim-etcd-cluster
advertise-client-urls: https://etcd1:2379
discovery:
discovery-srv:
discovery-srv-name:
discovery-fallback: 'proxy'
discovery-proxy:
strict-reconfig-check: true
auto-compaction-retention: "1"
auto-compaction-mode: periodic
enable-v2: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: /opt/etcd/conf/etcd.crt
  key-file: /opt/etcd/conf/etcd.key
  client-cert-auth: true
  trusted-ca-file: /opt/etcd/conf/rootCA.crt
  auto-tls: false
peer-transport-security:
  cert-file: /opt/etcd/conf/etcd.crt
  key-file: /opt/etcd/conf/etcd.key
  client-cert-auth: true
  trusted-ca-file: /opt/etcd/conf/rootCA.crt
  peer-auto-tls: false
logger: zap
log-level: debug
enable-pprof: false
metrics: basic
listen-metrics-urls:

etcd1 logs:

{"level":"info","ts":"2020-09-22T15:29:35.131Z","caller":"etcdmain/config.go:304","msg":"loaded server configuration, other configuration command line flags and environment variables will be ignored if provided","path":"/opt/etcd/bin/etcd.conf"}
{"level":"info","ts":"2020-09-22T15:29:35.131Z","caller":"etcdmain/etcd.go:134","msg":"server has been already initialized","data-dir":"/opt/etcd/data","dir-type":"member"}
{"level":"info","ts":"2020-09-22T15:29:35.131Z","caller":"embed/etcd.go:117","msg":"configuring peer listeners","listen-peer-urls":["https://0.0.0.0:2380"]}
{"level":"info","ts":"2020-09-22T15:29:35.131Z","caller":"embed/etcd.go:465","msg":"starting with peer TLS","tls-info":"cert = /opt/etcd/conf/etcd.crt, key = /opt/etcd/conf/etcd.key, trusted-ca = /opt/etcd/conf/rootCA.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2020-09-22T15:29:35.133Z","caller":"embed/etcd.go:127","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:2379"]}
{"level":"info","ts":"2020-09-22T15:29:35.134Z","caller":"embed/etcd.go:299","msg":"starting an etcd server","etcd-version":"3.4.0","git-sha":"898bd1351","go-version":"go1.12.9","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"etcd1","data-dir":"/opt/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/opt/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://etcd1:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://etcd1:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":false,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"1h0m0s","auto-compaction-interval":"1h0m0s","discovery-url":"","discovery-proxy":""}
{"level":"info","ts":"2020-09-22T15:29:35.135Z","caller":"etcdserver/backend.go:79","msg":"opened backend db","path":"/opt/etcd/data/member/snap/db","took":"559.355µs"}
{"level":"info","ts":"2020-09-22T15:29:35.137Z","caller":"etcdserver/raft.go:506","msg":"restarting local member","cluster-id":"efd3ef8b6d84750a","local-member-id":"9bc12ce3a3e2ece1","commit-index":3}
{"level":"info","ts":"2020-09-22T15:29:35.138Z","caller":"raft/raft.go:1530","msg":"9bc12ce3a3e2ece1 switched to configuration voters=()"}
{"level":"info","ts":"2020-09-22T15:29:35.138Z","caller":"raft/raft.go:700","msg":"9bc12ce3a3e2ece1 became follower at term 1409"}
{"level":"info","ts":"2020-09-22T15:29:35.138Z","caller":"raft/raft.go:383","msg":"newRaft 9bc12ce3a3e2ece1 [peers: [], term: 1409, commit: 3, applied: 0, lastindex: 3, lastterm: 1]"}
{"level":"warn","ts":"2020-09-22T15:29:35.193Z","caller":"auth/store.go:1317","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2020-09-22T15:29:35.199Z","caller":"etcdserver/quota.go:98","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2020-09-22T15:29:35.205Z","caller":"etcdserver/server.go:791","msg":"starting etcd server","local-member-id":"9bc12ce3a3e2ece1","local-server-version":"3.4.0","cluster-version":"to_be_decided"}
{"level":"info","ts":"2020-09-22T15:29:35.206Z","caller":"etcdserver/server.go:679","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2020-09-22T15:29:35.207Z","caller":"raft/raft.go:1530","msg":"9bc12ce3a3e2ece1 switched to configuration voters=(1334404034194501063)"}
{"level":"info","ts":"2020-09-22T15:29:35.210Z","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"efd3ef8b6d84750a","local-member-id":"9bc12ce3a3e2ece1","added-peer-id":"1284c1662d02edc7","added-peer-peer-urls":["https://etcd3:2380"]}
{"level":"info","ts":"2020-09-22T15:29:35.210Z","caller":"rafthttp/peer.go:128","msg":"starting remote peer","remote-peer-id":"1284c1662d02edc7"}
{"level":"info","ts":"2020-09-22T15:29:35.211Z","caller":"embed/etcd.go:576","msg":"serving peer traffic","address":"[::]:2380"}
{"level":"info","ts":"2020-09-22T15:29:35.210Z","caller":"rafthttp/pipeline.go:71","msg":"started HTTP pipelining with remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1284c1662d02edc7"}
{"level":"info","ts":"2020-09-22T15:29:35.211Z","caller":"embed/etcd.go:708","msg":"starting with client TLS","tls-info":"cert = /opt/etcd/conf/etcd.crt, key = /opt/etcd/conf/etcd.key, trusted-ca = /opt/etcd/conf/rootCA.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"warn","ts":"2020-09-22T15:29:35.211Z","caller":"grpclog/grpclog.go:55","msg":"Adjusting keepalive ping interval to minimum period of 1s"}
{"level":"info","ts":"2020-09-22T15:29:35.211Z","caller":"embed/etcd.go:241","msg":"now serving peer/client/metrics","local-member-id":"9bc12ce3a3e2ece1","initial-advertise-peer-urls":["https://etcd1:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://etcd1:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2020-09-22T15:29:35.217Z","caller":"rafthttp/stream.go:165","msg":"started stream writer with remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1284c1662d02edc7"}
{"level":"info","ts":"2020-09-22T15:29:35.217Z","caller":"rafthttp/stream.go:165","msg":"started stream writer with remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1284c1662d02edc7"}
{"level":"info","ts":"2020-09-22T15:29:35.219Z","caller":"rafthttp/peer.go:134","msg":"started remote peer","remote-peer-id":"1284c1662d02edc7"}
{"level":"info","ts":"2020-09-22T15:29:35.219Z","caller":"rafthttp/transport.go:327","msg":"added remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1284c1662d02edc7","remote-peer-urls":["https://etcd3:2380"]}
{"level":"info","ts":"2020-09-22T15:29:35.219Z","caller":"rafthttp/stream.go:405","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1284c1662d02edc7"}
{"level":"info","ts":"2020-09-22T15:29:35.219Z","caller":"rafthttp/stream.go:405","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1284c1662d02edc7"}
{"level":"debug","ts":"2020-09-22T15:29:35.220Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"info","ts":"2020-09-22T15:29:35.219Z","caller":"raft/raft.go:1530","msg":"9bc12ce3a3e2ece1 switched to configuration voters=(1334404034194501063 2167718332296818365)"}
{"level":"info","ts":"2020-09-22T15:29:35.220Z","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"efd3ef8b6d84750a","local-member-id":"9bc12ce3a3e2ece1","added-peer-id":"1e15485691a76ebd","added-peer-peer-urls":["https://etcd2:2380"]}
{"level":"info","ts":"2020-09-22T15:29:35.220Z","caller":"rafthttp/peer.go:128","msg":"starting remote peer","remote-peer-id":"1e15485691a76ebd"}
{"level":"info","ts":"2020-09-22T15:29:35.220Z","caller":"rafthttp/pipeline.go:71","msg":"started HTTP pipelining with remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1e15485691a76ebd"}
{"level":"debug","ts":"2020-09-22T15:29:35.219Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"info","ts":"2020-09-22T15:29:35.221Z","caller":"rafthttp/peer.go:134","msg":"started remote peer","remote-peer-id":"1e15485691a76ebd"}
{"level":"info","ts":"2020-09-22T15:29:35.221Z","caller":"rafthttp/transport.go:327","msg":"added remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1e15485691a76ebd","remote-peer-urls":["https://etcd2:2380"]}
{"level":"info","ts":"2020-09-22T15:29:35.221Z","caller":"raft/raft.go:1530","msg":"9bc12ce3a3e2ece1 switched to configuration voters=(1334404034194501063 2167718332296818365 11223301102602743009)"}
{"level":"info","ts":"2020-09-22T15:29:35.221Z","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"efd3ef8b6d84750a","local-member-id":"9bc12ce3a3e2ece1","added-peer-id":"9bc12ce3a3e2ece1","added-peer-peer-urls":["https://etcd1:2380"]}
{"level":"info","ts":"2020-09-22T15:29:35.222Z","caller":"rafthttp/stream.go:405","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1e15485691a76ebd"}
{"level":"debug","ts":"2020-09-22T15:29:35.222Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"info","ts":"2020-09-22T15:29:35.222Z","caller":"rafthttp/stream.go:165","msg":"started stream writer with remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1e15485691a76ebd"}
{"level":"info","ts":"2020-09-22T15:29:35.222Z","caller":"rafthttp/stream.go:405","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1e15485691a76ebd"}
{"level":"debug","ts":"2020-09-22T15:29:35.222Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"info","ts":"2020-09-22T15:29:35.222Z","caller":"rafthttp/stream.go:165","msg":"started stream writer with remote peer","local-member-id":"9bc12ce3a3e2ece1","remote-peer-id":"1e15485691a76ebd"}
{"level":"debug","ts":"2020-09-22T15:29:35.226Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream Message (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.227Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.227Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream Message (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.227Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.227Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream MsgApp v2 (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.227Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream MsgApp v2 (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.227Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.227Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.228Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream Message (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.229Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream MsgApp v2 (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.229Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream MsgApp v2 (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.230Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream Message (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.327Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.327Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.327Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.334Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream Message (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.327Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.339Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream Message (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.339Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream MsgApp v2 (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.339Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream MsgApp v2 (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.427Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.427Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.428Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.428Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.429Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream MsgApp v2 (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.429Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream Message (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.429Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream MsgApp v2 (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.430Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream Message (dial tcp 10.233.50.117:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.527Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.527Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.528Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.528Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.531Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream Message (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.532Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream MsgApp v2 (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.617Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream Message (EOF)"}
{"level":"debug","ts":"2020-09-22T15:29:35.617Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1284c1662d02edc7","error":"failed to dial 1284c1662d02edc7 on stream MsgApp v2 (EOF)"}
{"level":"debug","ts":"2020-09-22T15:29:35.627Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.627Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/message/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.628Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1284c1662d02edc7","address":"https://etcd3:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.628Z","caller":"rafthttp/stream.go:596","msg":"dial stream reader","from":"9bc12ce3a3e2ece1","to":"1e15485691a76ebd","address":"https://etcd2:2380/raft/stream/msgapp/9bc12ce3a3e2ece1"}
{"level":"debug","ts":"2020-09-22T15:29:35.630Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream Message (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"debug","ts":"2020-09-22T15:29:35.633Z","caller":"rafthttp/peer_status.go:82","msg":"peer deactivated again","peer-id":"1e15485691a76ebd","error":"failed to dial 1e15485691a76ebd on stream MsgApp v2 (dial tcp 10.233.19.97:2380: connect: connection refused)"}
{"level":"warn","ts":"2020-09-22T15:29:35.644Z","caller":"embed/config_logging.go:270","msg":"rejected connection","remote-addr":"10.233.118.32:52394","server-name":"etcd1","ip-addresses":[],"dns-names":["etcd","etcd1","etcd2","etcd3"],"error":"tls: \"10.233.118.32\" does not match any of DNSNames [\"etcd\" \"etcd1\" \"etcd2\" \"etcd3\"] (lookup etcd on 169.254.25.10:53: server misbehaving)"}

Please let me know, why connection is getting rejected, and quorum formation is failing.

Thanks in advance.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.