search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.09k stars 1.11k forks source link

CKS HA controller nodes not work property #7537

Closed VanDuy91 closed 1 year ago

VanDuy91 commented 1 year ago
ISSUE TYPE
COMPONENT NAME
Kubernetes
CLOUDSTACK VERSION
4.18.0.0
CONFIGURATION
OS / ENVIRONMENT

Ubuntu 22.04

SUMMARY

Create kubernetes cluster with 2 controller nodes. When stop one of them, I can't access to the cluster.

STEPS TO REPRODUCE
Create new cluster with 2 controller node (with isolated network)
after the cluster turn to active status, stop one of two controller nodes
Testing: get node of the cluster (using kubectl)
EXPECTED RESULTS
Still can continue to access the cluster
ACTUAL RESULTS
Cannot access the cluster with result: 
Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be pr
ocessing the request (get nodes)
Unable to connect to the server: EOF
I also created a cluster with three controller nodes and upon attempting to stop two of them. I found that the result was consistent. 
This means that the cluster can still be accessed with at least two controller nodes while HA enabled?!
boring-cyborg[bot] commented 1 year ago

Thanks for opening your first issue here! Be sure to follow the issue template!

kiranchavala commented 1 year ago

Hi @VanDuy91

When creating kubernetes cluster, did you select HA enabled? If yes, then its working as per the expected behaviour

Screenshot 2023-05-22 at 4 00 06 PM
VanDuy91 commented 1 year ago

Hi @VanDuy91

When creating kubernetes cluster, did you select HA enabled? If yes, then its working as per the expected behaviour

Screenshot 2023-05-22 at 4 00 06 PM

I opted for the HA (High Availability) enabled option and proceeded to shut down all controller nodes, leaving only one running. However, even after doing so, I'm still unable to access the cluster. (to access the cluster, I must keep at least two controller nodes running.)

kiranchavala commented 1 year ago

Thansks @VanDuy91

I am able to reproduce the issue

kiranchavala commented 1 year ago

@weizhouapache @Pearl1594 could you please look into it this issue

weizhouapache commented 1 year ago

added to 4.18.1.0 milestone

zap51 commented 1 year ago

@weizhouapache I tested this scenario with 3 control plane nodes. Guess etcd is refusing to work with a single node, perhaps issue with the quorum https://etcd.io/docs/v3.3/faq/#why-an-odd-number-of-cluster-members?

  1. Started with 3 nodes, cluster works as expected.
  2. Stopped one node, cluster works with 2 nodes as expected.
  3. Stopped 2 nodes, cluster stops working with single node.
  4. Start one more control plane node totalling 2, cluster starts to work.

I also took a look at the VR's HAProxy under these conditions, active health checks are enabled on k8s api server 6443, backends are correctly marked down without any issues. Logged into master node where there was only single node active saw that k8s-api-server static pod is continuously crashing. Also looks etcd static pod is stuck in election storm

# crictl logs 100c1133f3192
WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 
{"level":"info","ts":"2023-07-29T06:48:38.292Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--advertise-client-urls=https://10.1.1.4:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--initial-advertise-peer-urls=https://10.1.1.4:2380","--initial-cluster=cks-control-1898b776268=https://10.1.1.4:2380","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://10.1.1.4:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://10.1.1.4:2380","--name=cks-control-1898b776268","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]}
{"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}
{"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://10.1.1.4:2380"]}
{"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"embed/etcd.go:479","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://10.1.1.4:2379","https://127.0.0.1:2379"]}
{"level":"info","ts":"2023-07-29T06:48:38.294Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.3","git-sha":"0452feec7","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"cks-control-1898b776268","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://10.1.1.4:2380"],"listen-peer-urls":["https://10.1.1.4:2380"],"advertise-client-urls":["https://10.1.1.4:2379"],"listen-client-urls":["https://10.1.1.4:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-07-29T06:48:38.301Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"7.349991ms"}
{"level":"info","ts":"2023-07-29T06:48:39.062Z","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":1600160,"snapshot-size":"13 kB"}
{"level":"info","ts":"2023-07-29T06:48:39.062Z","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":13955072,"backend-size":"14 MB","backend-size-in-use-bytes":4677632,"backend-size-in-use":"4.7 MB"}
{"level":"info","ts":"2023-07-29T06:48:39.291Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","commit-index":1606465}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d switched to configuration voters=(139226378462478669 394890042043021191 17681339031669231524)"}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became follower at term 11"}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 1eea1a74f9ec14d [peers: [1eea1a74f9ec14d,57aee66a9503787,f560bc2b67e4cfa4], term: 11, commit: 1606465, applied: 1600160, lastindex: 1606469, lastterm: 11]"}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","recovered-remote-peer-id":"57aee66a9503787","recovered-remote-peer-urls":["https://10.1.1.151:2380"]}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","recovered-remote-peer-id":"f560bc2b67e4cfa4","recovered-remote-peer-urls":["https://10.1.1.183:2380"]}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","recovered-remote-peer-id":"1eea1a74f9ec14d","recovered-remote-peer-urls":["https://10.1.1.4:2380"]}
{"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2023-07-29T06:48:39.295Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-07-29T06:48:39.297Z","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1422383}
{"level":"info","ts":"2023-07-29T06:48:39.308Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1425171}
{"level":"info","ts":"2023-07-29T06:48:39.312Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-07-29T06:48:39.315Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"57aee66a9503787"}
{"level":"info","ts":"2023-07-29T06:48:39.315Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"}
{"level":"info","ts":"2023-07-29T06:48:39.316Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"}
{"level":"info","ts":"2023-07-29T06:48:39.317Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"}
{"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"57aee66a9503787"}
{"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"}
{"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"}
{"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787","remote-peer-urls":["https://10.1.1.151:2380"]}
{"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"f560bc2b67e4cfa4"}
{"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"}
{"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"}
{"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"}
{"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"f560bc2b67e4cfa4"}
{"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4","remote-peer-urls":["https://10.1.1.183:2380"]}
{"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"1eea1a74f9ec14d","timeout":"7s"}
{"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"}
{"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"}
{"level":"warn","ts":"2023-07-29T06:48:40.977Z","caller":"etcdserver/corrupt.go:289","msg":"failed hash kv request","local-member-id":"1eea1a74f9ec14d","requested-revision":1425171,"remote-peer-endpoint":"https://10.1.1.151:2380","error":"Get \"https://10.1.1.151:2380/members/hashkv\": dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:42.978Z","caller":"etcdserver/corrupt.go:289","msg":"failed hash kv request","local-member-id":"1eea1a74f9ec14d","requested-revision":1425171,"remote-peer-endpoint":"https://10.1.1.183:2380","error":"Get \"https://10.1.1.183:2380/members/hashkv\": dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:48:42.978Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"1eea1a74f9ec14d"}
{"level":"info","ts":"2023-07-29T06:48:42.978Z","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"1eea1a74f9ec14d","local-server-version":"3.5.3","cluster-id":"eddf204d5d97ae54","cluster-version":"3.5"}
{"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:688","msg":"starting with client TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"1eea1a74f9ec14d","initial-advertise-peer-urls":["https://10.1.1.4:2380"],"listen-peer-urls":["https://10.1.1.4:2380"],"advertise-client-urls":["https://10.1.1.4:2379"],"listen-client-urls":["https://10.1.1.4:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"]}
{"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:763","msg":"serving metrics","address":"http://127.0.0.1:2381"}
{"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"10.1.1.4:2380"}
{"level":"info","ts":"2023-07-29T06:48:42.992Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"10.1.1.4:2380"}
{"level":"warn","ts":"2023-07-29T06:48:44.318Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:44.318Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:44.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:44.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:46.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:46.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:46.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:46.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:46.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:48:49.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:49.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:49.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:48:49.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:48:49.983Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:48:51.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:48:51.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:53.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:48:54.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:54.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:54.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:54.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:48:56.984Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:58.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:48:58.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:48:58.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:48:58.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:48:58.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:48:59.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:59.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:59.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:48:59.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:01.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:49:01.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:03.984Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2023-07-29T06:49:04.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:04.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:04.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:04.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:09.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:09.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:09.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:09.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:10.985Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2023-07-29T06:49:11.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:49:11.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:14.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:14.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:14.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:14.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:17.986Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:19.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:19.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:19.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:19.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:49:20.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:20.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:20.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:20.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:20.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:21.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:49:21.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:24.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:24.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:24.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:24.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:24.987Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:29.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:29.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:29.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:29.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:31.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:49:31.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"warn","ts":"2023-07-29T06:49:31.987Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:34.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:34.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:34.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:34.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:49:34.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:34.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:34.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:34.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:34.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:38.988Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2023-07-29T06:49:39.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:39.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:39.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:39.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:41.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:49:41.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:44.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:44.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:44.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:44.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:49:45.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:45.989Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:49.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:49.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:49.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:49.330Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:51.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:49:51.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"warn","ts":"2023-07-29T06:49:52.990Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:54.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:54.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:54.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:49:54.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:58.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:49:58.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:49:58.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:49:58.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:49:58.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:49:59.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:59.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:59.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:59.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:49:59.990Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:01.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:50:01.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:03.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:04.330Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:04.330Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:04.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:04.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:06.991Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:09.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:09.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:09.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:09.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:11.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:50:11.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:13.992Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2023-07-29T06:50:14.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:14.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:14.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:14.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:16.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:19.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:19.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:19.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:19.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:20.993Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2023-07-29T06:50:21.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:50:21.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:23.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:24.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:24.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:24.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:24.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:27.994Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:29.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:29.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:29.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:29.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:31.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:50:31.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:34.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:34.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:34.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:34.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:34.995Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:50:36.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:39.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:39.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:39.338Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:39.338Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:41.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:50:41.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:41.996Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:44.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:44.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:44.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:44.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:46.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:48.996Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"warn","ts":"2023-07-29T06:50:49.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:49.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:49.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:50:49.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:51.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:50:51.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:54.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:54.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:54.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:54.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:55.997Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:50:59.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:59.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:59.341Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:50:59.341Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:51:01.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:51:01.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"warn","ts":"2023-07-29T06:51:02.998Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:51:04.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:51:04.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:51:04.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"warn","ts":"2023-07-29T06:51:04.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"}
{"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:51:09.338Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:51:09.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:51:09.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:51:09.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:51:09.998Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:51:11.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"}
{"level":"warn","ts":"2023-07-29T06:51:11.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
{"level":"info","ts":"2023-07-29T06:51:12.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:51:14.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:51:14.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:51:14.343Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"warn","ts":"2023-07-29T06:51:14.343Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"}
{"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}
{"level":"warn","ts":"2023-07-29T06:51:16.999Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"}
{"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"}
{"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"}
{"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"}
{"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"}
{"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"}

and from inside the etcd container


sh-5.1# etcdctl --cluster=true endpoint health
{"level":"warn","ts":"2023-07-29T06:55:21.108Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000432380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: failed to fetch endpoints from etcd cluster member list: context deadline exceeded

Could be the reason k8s-api-server not responding. @DaanHoogland @rohityadavcloud FYI

Cheers, Jayanth

weizhouapache commented 1 year ago

@zap51 I think you are right. It is an issue caused by etcd.

To achieve HA, we should

I think this is not a bug. cc @VanDuy91 @kiranchavala

VanDuy91 commented 1 year ago

I've got it. I appreciate your help.