search

siderolabs / cluster-api-control-plane-provider-talos

A control plane provider for CAPI + Talos
Mozilla Public License 2.0
64 stars 20 forks source link

Talos Control Plane Nodes with Capital Letters are Not Correctly Identified #185

Closed ccben87 closed 10 months ago

ccben87 commented 11 months ago

@smira and I previously discussed this issue in Slack where my control plane nodes were not working right because they were named with capital letters. One of the three of my control plane nodes, the one holding the current etcd membership, was receiving requests to remove it from CACPPT. I have since switched to using lowercase which is fine with me but this should be investigated and fixed as this could cause issues for other users. Please see logs below.

CACPPT logs:

2023-10-25T09:35:22Z    INFO    reconcile TalosControlPlane     {"controller": "taloscontrolplane", "controllerGroup": "[controlplane.cluster.x-k8s.io](http://controlplane.cluster.x-k8s.io/)", "controllerKind": "TalosControlPlane", "TalosControlPlane": {"name":"cluster-name-here-cp","namespace":"default"}, "namespace": "default", "name": "cluster-name-here-cp", "reconcileID": "1a846466-423c-43fe-8816-2e413f57ef53", "cluster": "cluster-name-here"}
2023-10-25T09:35:22Z    INFO    controllers.TalosControlPlane   found etcd member that doesn't exist as controlplane machine    {"member": "id:14048463810200033261 hostname:\"NODE-NAME-HERE-CENSORED-CP01\" peer_urls:\"https://xxx.xx.xx.33:2380/\" client_urls:\"https://xxx.xx.xx.33:2379/\""}
2023-10-25T09:35:22Z    INFO    controllers.TalosControlPlane   removing etcd member    {"memberName": "NODE-NAME-HERE-CENSORED-CP01"}
2023-10-25T09:35:22Z    INFO    controllers.TalosControlPlane   verifying etcd health on all nodes      {"node": "cluster-name-here-cp-8ng4t", "node": "cluster-name-here-cp-q5r89", "node": "cluster-name-here-cp-zkn6c"}
2023-10-25T09:35:22Z    INFO    controllers.TalosControlPlane   attempting to set control plane status
2023-10-25T09:35:22Z    INFO    controllers.TalosControlPlane   ready replicas  {"count": 3}
2023-10-25T09:35:22Z    INFO    controllers.TalosControlPlane   successfully updated control plane status       {"namespace": "default", "talosControlPlane": "cluster-name-here-cp", "cluster": "cluster-name-here"}

Logs from my 3 control plane nodes:

72.28.20.41: {"level":"info","ts":"2023-10-25T08:56:01.499213Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"57b0aa28df78c2d0","local-member-id":"a67c2e0271de613b","added-peer-id":"c2f62b6746e757ed","added-peer-peer-urls":["https://xxx.xx.xx.33:2380/"]}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.499229Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.499249Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.500836Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.500875Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed","remote-peer-urls":["https://xxx.xx.xx.33:2380/"]}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.500998Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"57b0aa28df78c2d0","local-member-id":"a67c2e0271de613b","cluster-version":"3.5"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501029Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501134Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a67c2e0271de613b switched to configuration voters=(14048463810200033261) learners=(2249582023990197245)"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501154Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"57b0aa28df78c2d0","local-member-id":"a67c2e0271de613b","added-peer-id":"1f381eee990a3bfd","added-peer-peer-urls":["https://xxx.xx.xx.55:2380/"]}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501199Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501225Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501446Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /system/secrets/etcd/server.crt, key = /system/secrets/etcd/server.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501818Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501894Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501938Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501974Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.501981Z","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"a67c2e0271de613b","initial-advertise-peer-urls":["https://xxx.xx.xx.41:2380/"],"listen-peer-urls":["https://0.0.0.0:2380/"],"advertise-client-urls":["https://xxx.xx.xx.41:2379/"],"listen-client-urls":["https://0.0.0.0:2379/"],"listen-metrics-urls":[]}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.502355Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.50247Z","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"[::]:2380"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.502484Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"[::]:2380"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.502708Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.503464Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.503517Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd","remote-peer-urls":["https://xxx.xx.xx.55:2380/"]}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.503967Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.504128Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.504197Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a67c2e0271de613b switched to configuration voters=(2249582023990197245 14048463810200033261)"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.504289Z","caller":"membership/cluster.go:535","msg":"promote member","cluster-id":"57b0aa28df78c2d0","local-member-id":"a67c2e0271de613b"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.505452Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a67c2e0271de613b switched to configuration voters=(2249582023990197245 14048463810200033261) learners=(11996514095443370299)"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.505521Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"57b0aa28df78c2d0","local-member-id":"a67c2e0271de613b","added-peer-id":"a67c2e0271de613b","added-peer-peer-urls":["https://xxx.xx.xx.41:2380/"]}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.505664Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a67c2e0271de613b switched to configuration voters=(2249582023990197245 11996514095443370299 14048463810200033261)"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.505713Z","caller":"membership/cluster.go:535","msg":"promote member","cluster-id":"57b0aa28df78c2d0","local-member-id":"a67c2e0271de613b"}
xxx.xx.xx.41: {"level":"warn","ts":"2023-10-25T08:56:01.507595Z","caller":"etcdserver/server.go:1127","msg":"server error","error":"the member has been permanently removed from the cluster"}
xxx.xx.xx.41: {"level":"warn","ts":"2023-10-25T08:56:01.507664Z","caller":"etcdserver/server.go:1128","msg":"data-dir used by this member must be removed"}
xxx.xx.xx.41: {"level":"warn","ts":"2023-10-25T08:56:01.507771Z","caller":"etcdserver/server.go:2073","msg":"stopped publish because server is stopped","local-member-id":"a67c2e0271de613b","local-member-attributes":"{Name:NODE-NAME-HERE-CENSORED-CP03 ClientURLs:[https://xxx.xx.xx.41:2379/]}","publish-timeout":"7s","error":"etcdserver: server stopped"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508086Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508129Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508159Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508205Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508241Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508265Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"a67c2e0271de613b","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508284Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"1f381eee990a3bfd"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508294Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.50832Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508366Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508415Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508475Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508544Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"a67c2e0271de613b","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.41: {"level":"info","ts":"2023-10-25T08:56:01.508569Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"c2f62b6746e757ed"}

xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.10014Z","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"bdbbde998","go-version":"go1.19.9","go-os":"linux","go-arch":"amd64","max-cpu-set":2,"max-cpu-available":2,"member-initialized":true,"name":"NODE-NAME-HERE-CENSORED-CP02","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://xxx.xx.xx.55:2380/"],"listen-peer-urls":["https://0.0.0.0:2380/"],"advertise-client-urls":["https://xxx.xx.xx.55:2379/"],"listen-client-urls":["https://0.0.0.0:2379/"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":true,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.101281Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"943.906µs"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.114481Z","caller":"etcdserver/server.go:530","msg":"No snapshot found. Recovering WAL from scratch!"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.127621Z","caller":"etcdserver/raft.go:530","msg":"restarting local member","cluster-id":"57b0aa28df78c2d0","local-member-id":"1f381eee990a3bfd","commit-index":769}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.127872Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1f381eee990a3bfd switched to configuration voters=()"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.128075Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1f381eee990a3bfd became follower at term 2"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.128103Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 1f381eee990a3bfd [peers: [], term: 2, commit: 769, applied: 0, lastindex: 769, lastterm: 2]"}
xxx.xx.xx.55: {"level":"warn","ts":"2023-10-25T09:00:41.12937Z","caller":"auth/store.go:1238","msg":"simple token is not cryptographically signed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.131918Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":642}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.133785Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.135826Z","caller":"etcdserver/corrupt.go:95","msg":"starting initial corruption check","local-member-id":"1f381eee990a3bfd","timeout":"7s"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136378Z","caller":"etcdserver/corrupt.go:165","msg":"initial corruption checking passed; no corruption","local-member-id":"1f381eee990a3bfd"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.13641Z","caller":"etcdserver/server.go:854","msg":"starting etcd server","local-member-id":"1f381eee990a3bfd","local-server-version":"3.5.9","cluster-version":"to_be_decided"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136584Z","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136697Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136736Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136742Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136885Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1f381eee990a3bfd switched to configuration voters=(14048463810200033261)"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136953Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"57b0aa28df78c2d0","local-member-id":"1f381eee990a3bfd","added-peer-id":"c2f62b6746e757ed","added-peer-peer-urls":["https://xxx.xx.xx.33:2380/"]}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.136973Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.139316Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /system/secrets/etcd/server.crt, key = /system/secrets/etcd/server.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.139977Z","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"[::]:2380"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.139996Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"[::]:2380"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.140172Z","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"1f381eee990a3bfd","initial-advertise-peer-urls":["https://xxx.xx.xx.55:2380/"],"listen-peer-urls":["https://0.0.0.0:2380/"],"advertise-client-urls":["https://xxx.xx.xx.55:2379/"],"listen-client-urls":["https://0.0.0.0:2379/"],"listen-metrics-urls":[]}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.140474Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.141421Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.141943Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142173Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142193Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed","remote-peer-urls":["https://xxx.xx.xx.33:2380/"]}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142246Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.14242Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142586Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"57b0aa28df78c2d0","local-member-id":"1f381eee990a3bfd","cluster-version":"3.5"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142611Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142711Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1f381eee990a3bfd switched to configuration voters=(14048463810200033261) learners=(2249582023990197245)"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142759Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"57b0aa28df78c2d0","local-member-id":"1f381eee990a3bfd","added-peer-id":"1f381eee990a3bfd","added-peer-peer-urls":["https://xxx.xx.xx.55:2380/"]}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142854Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1f381eee990a3bfd switched to configuration voters=(2249582023990197245 14048463810200033261)"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.142888Z","caller":"membership/cluster.go:535","msg":"promote member","cluster-id":"57b0aa28df78c2d0","local-member-id":"1f381eee990a3bfd"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.143885Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1f381eee990a3bfd switched to configuration voters=(2249582023990197245 14048463810200033261) learners=(11996514095443370299)"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.144498Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"57b0aa28df78c2d0","local-member-id":"1f381eee990a3bfd","added-peer-id":"a67c2e0271de613b","added-peer-peer-urls":["https://xxx.xx.xx.41:2380/"]}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.144583Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.144637Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"warn","ts":"2023-10-25T09:00:41.146164Z","caller":"etcdserver/server.go:1127","msg":"server error","error":"the member has been permanently removed from the cluster"}
xxx.xx.xx.55: {"level":"warn","ts":"2023-10-25T09:00:41.146268Z","caller":"etcdserver/server.go:1128","msg":"data-dir used by this member must be removed"}
xxx.xx.xx.55: {"level":"warn","ts":"2023-10-25T09:00:41.146403Z","caller":"etcdserver/server.go:2083","msg":"failed to publish local member to cluster through raft","local-member-id":"1f381eee990a3bfd","local-member-attributes":"{Name:NODE-NAME-HERE-CENSORED-CP02 ClientURLs:[https://xxx.xx.xx.55:2379/]}","request-path":"/0/members/1f381eee990a3bfd/attributes","publish-timeout":"7s","error":"etcdserver: request cancelled"}
xxx.xx.xx.55: {"level":"warn","ts":"2023-10-25T09:00:41.146472Z","caller":"etcdserver/server.go:2083","msg":"failed to publish local member to cluster through raft","local-member-id":"1f381eee990a3bfd","local-member-attributes":"{Name:NODE-NAME-HERE-CENSORED-CP02 ClientURLs:[https://xxx.xx.xx.55:2379/]}","request-path":"/0/members/1f381eee990a3bfd/attributes","publish-timeout":"7s","error":"etcdserver: request cancelled"}
xxx.xx.xx.55: {"level":"warn","ts":"2023-10-25T09:00:41.146503Z","caller":"etcdserver/server.go:2083","msg":"failed to publish local member to cluster through raft","local-member-id":"1f381eee990a3bfd","local-member-attributes":"{Name:NODE-NAME-HERE-CENSORED-CP02 ClientURLs:[https://xxx.xx.xx.55:2379/]}","request-path":"/0/members/1f381eee990a3bfd/attributes","publish-timeout":"7s","error":"etcdserver: request cancelled"}
xxx.xx.xx.55: {"level":"warn","ts":"2023-10-25T09:00:41.146567Z","caller":"etcdserver/server.go:2073","msg":"stopped publish because server is stopped","local-member-id":"1f381eee990a3bfd","local-member-attributes":"{Name:NODE-NAME-HERE-CENSORED-CP02 ClientURLs:[https://xxx.xx.xx.55:2379/]}","publish-timeout":"7s","error":"etcdserver: server stopped"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.147215Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.147349Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.148365Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.148404Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b","remote-peer-urls":["https://xxx.xx.xx.41:2380/"]}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.14881Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1f381eee990a3bfd switched to configuration voters=(2249582023990197245 11996514095443370299 14048463810200033261)"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.148906Z","caller":"membership/cluster.go:535","msg":"promote member","cluster-id":"57b0aa28df78c2d0","local-member-id":"1f381eee990a3bfd"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.149845Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.150261Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.150776Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.15092Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.150987Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151031Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151485Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151636Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1f381eee990a3bfd","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151683Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"c2f62b6746e757ed"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151712Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.15173Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.15178Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.15182Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151881Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151919Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1f381eee990a3bfd","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.55: {"level":"info","ts":"2023-10-25T09:00:41.151936Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"a67c2e0271de613b"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:46.947307Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"xxx.xx.xx.55:54590","server-name":"","error":"EOF"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:46.947416Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"xxx.xx.xx.55:54592","server-name":"","error":"read tcp xxx.xx.xx.33:2380->xxx.xx.xx.55:54592: read: connection reset by peer"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:51.185171Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"a67c2e0271de613b"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:51.186797Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"a67c2e0271de613b"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:51.188557Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"xxx.xx.xx.41:47642","server-name":"","error":"EOF"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:51.188738Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"xxx.xx.xx.41:47644","server-name":"","error":"EOF"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:52.0997Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"1f381eee990a3bfd"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:52.100955Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"1f381eee990a3bfd"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:52.10341Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"xxx.xx.xx.55:51298","server-name":"","error":"read tcp xxx.xx.xx.33:2380->xxx.xx.xx.55:51298: read: connection reset by peer"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:52.104122Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"xxx.xx.xx.55:51294","server-name":"","error":"read tcp xxx.xx.xx.33:2380->xxx.xx.xx.55:51294: read: connection reset by peer"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:53.482754Z","caller":"membership/cluster.go:657","msg":"rejecting member remove; started member will be less than quorum","number-of-started-member":0,"quorum":1,"cluster-id":"57b0aa28df78c2d0","local-member-id":"c2f62b6746e757ed"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:53.482794Z","caller":"etcdserver/server.go:1798","msg":"rejecting member remove request; not enough healthy members","local-member-id":"c2f62b6746e757ed","requested-member-remove-id":"c2f62b6746e757ed","error":"etcdserver: re-configuration failed due to not enough started members"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:53.537287Z","caller":"membership/cluster.go:657","msg":"rejecting member remove; started member will be less than quorum","number-of-started-member":0,"quorum":1,"cluster-id":"57b0aa28df78c2d0","local-member-id":"c2f62b6746e757ed"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:53.537329Z","caller":"etcdserver/server.go:1798","msg":"rejecting member remove request; not enough healthy members","local-member-id":"c2f62b6746e757ed","requested-member-remove-id":"c2f62b6746e757ed","error":"etcdserver: re-configuration failed due to not enough started members"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:56.344192Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"a67c2e0271de613b"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:56.344356Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"a67c2e0271de613b"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:56.347675Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"a67c2e0271de613b"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:56.348366Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"a67c2e0271de613b"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:57.266062Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"1f381eee990a3bfd"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:57.26709Z","caller":"rafthttp/http.go:394","msg":"rejected stream from remote peer because it was removed","local-member-id":"c2f62b6746e757ed","remote-peer-id-stream-handler":"c2f62b6746e757ed","remote-peer-id-from":"1f381eee990a3bfd"}
xxx.xx.xx.33: {"level":"warn","ts":"2023-10-25T09:03:57.268184Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"xxx.xx.xx.55:51324","server-name":"","error":"EOF"}
smira commented 11 months ago

thanks for reporting it, the core bug here is an attempt to match Node nodename (which is always lowercase) to the hostname (in etcd members) leading to inconsitencies.