search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
5.75k stars 463 forks source link

etcd failure to join the cluster #5350

Closed smira closed 2 years ago

smira commented 2 years ago

Bug Report

Observed in sfyra when scaling up the control plane from 1->3.

2 out of 3 nodes are members of the cluster, but the 3rd node is stuck.

Description

Logs

172.25.0.5: [talos] service[etcd](Waiting): Waiting for service "cri" to be "up", time sync, network
172.25.0.5: [talos] service[etcd](Waiting): Waiting for service "cri" to be "up", time sync
172.25.0.5: [talos] service[etcd](Waiting): Waiting for time sync
172.25.0.5: [talos] service[etcd](Preparing): Running pre state
172.25.0.5: [talos] etcd is waiting to join the cluster, if this node is the first node in the cluster, please run `talosctl bootstrap` against one of the following IPs:
172.25.0.5: [talos] retrying error: error adding member: etcdserver: too many learner members in cluster
172.25.0.5: [talos] service[etcd](Preparing): Creating service runner
172.25.0.5: [talos] retrying error: etcdserver: can only promote a learner member which is in sync with leader
172.25.0.5: [talos] service[etcd](Running): Started task etcd (PID 2108) for container etcd
172.25.0.5: [talos] service[etcd](Running): Health check failed: error building etcd client: context deadline exceeded
172.25.0.5: [talos] service[etcd](Waiting): Runner Containerd(etcd) exited without error, going to restart it
172.25.0.5: [talos] service[etcd](Running): Started task etcd (PID 2337) for container etcd
172.25.0.5: [talos] retrying error: etcdserver: no leader
172.25.0.5: [talos] retrying error: etcdserver: request timed out
172.25.0.5: machined Unknown [/machine.MachineService/EtcdMemberList] 5.000970608s unary error building etcd client: context deadline exceeded (:authority=localhost;content-type=application/grpc;proxyfrom=172.25.0.4,172.25.0.5,172.25.0.6;talos-role=os:admin;user-agent=grpc-go/1.44.1-dev)
172.25.0.5: [talos] retrying error: etcdserver: member not found

^^ etcd failed (with exit code 1), and as it was restarted, it hasn't joined fully, but instead it started a "fresh" cluster of sorts.

etcd logs:

``` 172.25.0.5: {"level":"info","ts":1649962007.5504527,"caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CIPHER_SUITES","variable-value":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:46:47.550Z","caller":"etcdmain/etcd.go:72","msg":"Running: ","args":["/usr/local/bin/etcd","--advertise-client-urls=https://172.25.0.5:2379","--auto-tls=false","--cert-file=/system/secrets/etcd/peer.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--initial-advertise-peer-urls=https://172.25.0.5:2380","--initial-cluster=pxe-1=https://172.25.0.4:2380,pxe-3=https://172.25.0.6:2380,pxe-2=https://172.25.0.5:2380","--initial-cluster-state=existing","--key-file=/system/secrets/etcd/peer.key","--listen-client-urls=https://0.0.0.0:2379","--listen-peer-urls=https://0.0.0.0:2380","--name=pxe-2","--peer-auto-tls=false","--peer-cert-file=/system/secrets/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/system/secrets/etcd/peer.key","--peer-trusted-ca-file=/system/secrets/etcd/ca.crt","--trusted-ca-file=/system/secrets/etcd/ca.crt"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:46:47.550Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://0.0.0.0:2380"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:46:47.550Z","caller":"embed/etcd.go:479","msg":"starting with peer TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:46:47.556Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:2379"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:46:47.556Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.2","git-sha":"99018a77b","go-version":"go1.16.3","go-os":"linux","go-arch":"amd64","max-cpu-set":2,"max-cpu-available":2,"member-initialized":false,"name":"pxe-2","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://172.25.0.5:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://172.25.0.5:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"pxe-1=https://172.25.0.4:2380,pxe-2=https://172.25.0.5:2380,pxe-3=https://172.25.0.6:2380","initial-cluster-state":"existing","initial-cluster-token":"etcd-cluster","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:46:55.458Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"7.901092048s"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:02.466Z","caller":"etcdserver/cluster_util.go:428","msg":"failed to convert response","address":"https://172.25.0.4:2380/downgrade/enabled","remote-member-id":"4721d2496fb67522","error":"strconv.ParseBool: parsing \"failed linearized read: context deadline exceeded\\n\": invalid syntax"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:02.466Z","caller":"etcdserver/cluster_util.go:381","msg":"failed to get downgrade enabled status","remote-member-id":"4721d2496fb67522","error":"strconv.ParseBool: parsing \"failed linearized read: context deadline exceeded\\n\": invalid syntax"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.912Z","caller":"wal/wal.go:802","msg":"slow fdatasync","took":"4.596541139s","expected-duration":"1s"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.931Z","caller":"etcdserver/raft.go:448","msg":"starting local member","local-member-id":"b84dd5702e3dba73","cluster-id":"f9d7b3fd6c64c2f0"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.931Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 switched to configuration voters=()"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.931Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 became follower at term 0"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.931Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft b84dd5702e3dba73 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.933Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.936Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.937Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.939Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.939Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522","remote-peer-urls":["https://172.25.0.4:2380"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.939Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.939Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184","remote-peer-urls":["https://172.25.0.6:2380"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.939Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.939Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.939Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.940Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.940Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.940Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.940Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522","remote-peer-urls":["https://172.25.0.4:2380"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.940Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.940Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.941Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.943Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.945Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.945Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.945Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184","remote-peer-urls":["https://172.25.0.6:2380"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.945Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.945Z","caller":"etcdserver/server.go:847","msg":"starting etcd server","local-member-id":"b84dd5702e3dba73","local-server-version":"3.5.2","cluster-version":"to_be_decided"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.946Z","caller":"etcdserver/server.go:748","msg":"starting initial election tick advance","election-ticks":10} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.946Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.950Z","caller":"embed/etcd.go:688","msg":"starting with client TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.952Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"b84dd5702e3dba73","initial-advertise-peer-urls":["https://172.25.0.5:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://172.25.0.5:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.952Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"[::]:2380"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.952Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"[::]:2380"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.952Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 [term: 0] received a MsgHeartbeat message with higher term from 4982323e3f55d184 [term: 2]"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.952Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 became follower at term 2"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.952Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: b84dd5702e3dba73 elected leader 4982323e3f55d184 at term 2"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.952Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"b84dd5702e3dba73","to":"4721d2496fb67522","stream-type":"stream Message"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.953Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.953Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.954Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"b84dd5702e3dba73","to":"4721d2496fb67522","stream-type":"stream MsgApp v2"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.954Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.954Z","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"4721d2496fb67522","error":"failed to dial 4721d2496fb67522 on stream MsgApp v2 (the member has been permanently removed from the cluster)"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.955Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"b84dd5702e3dba73","to":"4982323e3f55d184","stream-type":"stream Message"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.955Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.955Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.955Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"b84dd5702e3dba73","to":"4982323e3f55d184","stream-type":"stream MsgApp v2"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.955Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.957Z","caller":"etcdserver/server.go:1146","msg":"server error","error":"the member has been permanently removed from the cluster"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.957Z","caller":"etcdserver/server.go:1147","msg":"data-dir used by this member must be removed"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.957Z","caller":"etcdserver/server.go:2042","msg":"stopped publish because server is stopped","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","publish-timeout":"7s","error":"etcdserver: server stopped"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.957Z","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"4982323e3f55d184","error":"failed to dial 4982323e3f55d184 on stream Message (the member has been permanently removed from the cluster)"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.964Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 [term: 2] received a MsgApp message with higher term from 4721d2496fb67522 [term: 3]"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.964Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 became follower at term 3"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.964Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: b84dd5702e3dba73 changed leader from 4982323e3f55d184 to 4721d2496fb67522 at term 3"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.966Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.966Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.966Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.968Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.968Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.977Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.977Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.981Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.982Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.982Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.982Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"4721d2496fb67522"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.982Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.982Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.982Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:12.983Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.983Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.983Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.983Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.983Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"b84dd5702e3dba73","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.983Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"4982323e3f55d184"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.987Z","caller":"etcdmain/main.go:47","msg":"notifying init daemon"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:12.987Z","caller":"etcdmain/main.go:53","msg":"successfully notified init daemon"} 172.25.0.5: {"level":"info","ts":1649962038.1264303,"caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CIPHER_SUITES","variable-value":"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.126Z","caller":"etcdmain/etcd.go:72","msg":"Running: ","args":["/usr/local/bin/etcd","--advertise-client-urls=https://172.25.0.5:2379","--auto-tls=false","--cert-file=/system/secrets/etcd/peer.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--initial-advertise-peer-urls=https://172.25.0.5:2380","--initial-cluster=pxe-1=https://172.25.0.4:2380,pxe-3=https://172.25.0.6:2380,pxe-2=https://172.25.0.5:2380","--initial-cluster-state=existing","--key-file=/system/secrets/etcd/peer.key","--listen-client-urls=https://0.0.0.0:2379","--listen-peer-urls=https://0.0.0.0:2380","--name=pxe-2","--peer-auto-tls=false","--peer-cert-file=/system/secrets/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/system/secrets/etcd/peer.key","--peer-trusted-ca-file=/system/secrets/etcd/ca.crt","--trusted-ca-file=/system/secrets/etcd/ca.crt"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.126Z","caller":"etcdmain/etcd.go:115","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.126Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://0.0.0.0:2380"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.126Z","caller":"embed/etcd.go:479","msg":"starting with peer TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.132Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://0.0.0.0:2379"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.132Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.2","git-sha":"99018a77b","go-version":"go1.16.3","go-os":"linux","go-arch":"amd64","max-cpu-set":2,"max-cpu-available":2,"member-initialized":true,"name":"pxe-2","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://172.25.0.5:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://172.25.0.5:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.133Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"99.96µs"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.133Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.133Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"f9d7b3fd6c64c2f0","local-member-id":"b84dd5702e3dba73","commit-index":0} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.134Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 switched to configuration voters=()"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.134Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"b84dd5702e3dba73 became follower at term 0"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.134Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft b84dd5702e3dba73 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:18.137Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.154Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.154Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.170Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"b84dd5702e3dba73","timeout":"7s"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.170Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"b84dd5702e3dba73"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.170Z","caller":"etcdserver/server.go:847","msg":"starting etcd server","local-member-id":"b84dd5702e3dba73","local-server-version":"3.5.2","cluster-version":"to_be_decided"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.171Z","caller":"etcdserver/server.go:748","msg":"starting initial election tick advance","election-ticks":10} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.174Z","caller":"embed/etcd.go:688","msg":"starting with client TLS","tls-info":"cert = /system/secrets/etcd/peer.crt, key = /system/secrets/etcd/peer.key, client-cert=, client-key=, trusted-ca = /system/secrets/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.174Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"[::]:2380"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.174Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"[::]:2380"} 172.25.0.5: {"level":"info","ts":"2022-04-14T18:47:18.174Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"b84dd5702e3dba73","initial-advertise-peer-urls":["https://172.25.0.5:2380"],"listen-peer-urls":["https://0.0.0.0:2380"],"advertise-client-urls":["https://172.25.0.5:2379"],"listen-client-urls":["https://0.0.0.0:2379"],"listen-metrics-urls":[]} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:25.172Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:32.172Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:39.172Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:46.173Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:47:53.174Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:00.174Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:07.174Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:14.175Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:21.175Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:28.176Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:35.177Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:42.178Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:49.179Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:48:56.180Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:03.180Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:10.180Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:17.181Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:24.181Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:31.182Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:38.183Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:45.184Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:52.185Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:49:59.186Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:50:06.187Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:50:13.188Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:50:20.188Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:50:27.188Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:50:34.189Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} 172.25.0.5: {"level":"warn","ts":"2022-04-14T18:50:41.190Z","caller":"etcdserver/server.go:2052","msg":"failed to publish local member to cluster through raft","local-member-id":"b84dd5702e3dba73","local-member-attributes":"{Name:pxe-2 ClientURLs:[https://172.25.0.5:2379]}","request-path":"/0/members/b84dd5702e3dba73/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} ```

Environment

smira commented 2 years ago

was a problem in CACPPT, fixed with https://github.com/siderolabs/cluster-api-control-plane-provider-talos/pull/120