search

openebs / mayastor

Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes & Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.
Apache License 2.0
670 stars 101 forks source link

Error: etcdserver: member not found, after upgrading helm chart etcd replicas 1 to 3 #1659

Open todeb opened 1 month ago

todeb commented 1 month ago

Describe the bug After upgrading chart with mayastor.etcd.replicaCount=3 from 1.

etcd-0 member crashloopbackoff with log:

etcd 09:39:00.03 Welcome to the Bitnami etcd container
etcd 09:39:00.03 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 09:39:00.04 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 09:39:00.04
etcd 09:39:00.05 INFO  ==> ** Starting etcd setup **
etcd 09:39:00.07 INFO  ==> Validating settings in ETCD_* env vars..
etcd 09:39:00.07 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 09:39:00.09 INFO  ==> Initializing etcd
etcd 09:39:00.09 INFO  ==> Generating etcd config file using env variables
etcd 09:39:00.15 INFO  ==> Detected data from previous deployments
etcd 09:39:00.49 INFO  ==> Updating member in existing cluster
{"level":"warn","ts":"2024-05-16T09:39:00.595Z","logger":"etcd-client","caller":"v3@v3.5.6/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00037a8c0/openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found

other new deployed etcd members logs:

Defaulted container "etcd" out of: etcd, volume-permissions (init)
etcd 09:26:48.36
etcd 09:26:48.36 Welcome to the Bitnami etcd container
etcd 09:26:48.37 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 09:26:48.37 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 09:26:48.37
etcd 09:26:48.37 INFO  ==> ** Starting etcd setup **
etcd 09:26:48.39 INFO  ==> Validating settings in ETCD_* env vars..
etcd 09:26:48.39 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 09:26:48.40 INFO  ==> Initializing etcd
etcd 09:26:48.40 INFO  ==> Generating etcd config file using env variables
etcd 09:26:48.45 INFO  ==> There is no data from previous deployments
etcd 09:26:48.45 INFO  ==> Bootstrapping a new cluster
etcd 09:26:48.48 INFO  ==> Obtaining cluster member ID
etcd 09:26:48.50 INFO  ==> Starting etcd in background
etcd 09:26:51.69 INFO  ==> Stored member ID: 6f5be5a5cdec82d9
etcd 09:26:51.70 INFO  ==> Stopping etcd
Terminated
etcd 09:26:52.72 INFO  ==> ** etcd setup finished! **

etcd 09:26:52.74 INFO  ==> ** Starting etcd **
{"level":"info","ts":"2024-05-16T09:26:52.780Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379,http://openebs-etcd.openebs.svc.cluster.local:2379"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"100"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-05-16T09:26:52.781Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"openebs-etcd-0=http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380,openebs-etcd-1=http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380,openebs-etcd-2=http://openebs-etcd-2.openebs-etcd-headless.openebs.svc.cluster.local:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"openebs-etcd-1"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"8589934592"}
{"level":"warn","ts":"2024-05-16T09:26:52.782Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=openebs-etcd-headless.openebs.svc.cluster.local"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2024-05-16T09:26:52.783Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2024-05-16T09:26:52.784Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"info","ts":"2024-05-16T09:26:52.784Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"warn","ts":"2024-05-16T09:26:52.785Z","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"member_id","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2024-05-16T09:26:52.785Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"info","ts":"2024-05-16T09:26:52.785Z","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.785Z","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2024-05-16T09:26:52.786Z","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.6","git-sha":"cecbe35ce","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":10,"max-cpu-available":10,"member-initialized":true,"name":"openebs-etcd-1","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379","http://openebs-etcd.openebs.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":8589934592,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"revision","auto-compaction-retention":"100ns","auto-compaction-interval":"100ns","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2024-05-16T09:26:52.789Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/bitnami/etcd/data/member/snap/db","took":"229.366µs"}
{"level":"info","ts":"2024-05-16T09:26:52.789Z","caller":"etcdserver/server.go:530","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2024-05-16T09:26:52.795Z","caller":"etcdserver/raft.go:529","msg":"restarting local member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","commit-index":8}
{"level":"info","ts":"2024-05-16T09:26:52.798Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=()"}
{"level":"info","ts":"2024-05-16T09:26:52.798Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 became follower at term 3"}
{"level":"info","ts":"2024-05-16T09:26:52.798Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6f5be5a5cdec82d9 [peers: [], term: 3, commit: 8, applied: 0, lastindex: 8, lastterm: 3]"}
{"level":"info","ts":"2024-05-16T09:26:52.827Z","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2024-05-16T09:26:52.828Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":8589934592,"quota-size":"8.6 GB"}
{"level":"info","ts":"2024-05-16T09:26:52.828Z","caller":"etcdserver/server.go:854","msg":"starting etcd server","local-member-id":"6f5be5a5cdec82d9","local-server-version":"3.5.6","cluster-version":"to_be_decided"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/bitnami/etcd/data/member/snap","suffix":"snap","max":5,"interval":"30s"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/bitnami/etcd/data/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=(8024259661377864409)"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","added-peer-id":"6f5be5a5cdec82d9","added-peer-peer-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=(8024259661377864409 8763204770994443348)"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","added-peer-id":"799d284f4d21d454","added-peer-peer-urls":["http://openebs-etcd-2.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.829Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454","remote-peer-urls":["http://openebs-etcd-2.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 switched to configuration voters=(8024259661377864409 8763204770994443348 16092638608179729187)"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","added-peer-id":"df54898d3b4a1323","added-peer-peer-urls":["http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.831Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.832Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.834Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.834Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"6f5be5a5cdec82d9","initial-advertise-peer-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379","http://openebs-etcd.openebs.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2024-05-16T09:26:52.835Z","caller":"embed/etcd.go:586","msg":"serving peer traffic","address":"[::]:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.835Z","caller":"embed/etcd.go:558","msg":"cmux::serve","address":"[::]:2380"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323","remote-peer-urls":["http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380"]}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"f2b3b9f758cdc80","local-member-id":"6f5be5a5cdec82d9","cluster-version":"3.5"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2024-05-16T09:26:52.836Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6f5be5a5cdec82d9","to":"799d284f4d21d454","stream-type":"stream Message"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"df54898d3b4a1323"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6f5be5a5cdec82d9 elected leader 799d284f4d21d454 at term 3"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"6f5be5a5cdec82d9","to":"799d284f4d21d454","stream-type":"stream MsgApp v2"}
{"level":"info","ts":"2024-05-16T09:26:52.837Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"info","ts":"2024-05-16T09:26:52.839Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454"}
{"level":"warn","ts":"2024-05-16T09:26:52.841Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"embed/serve.go:100","msg":"ready to serve client requests"}
{"level":"warn","ts":"2024-05-16T09:26:52.842Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"etcdserver/server.go:2054","msg":"published local member to cluster through raft","local-member-id":"6f5be5a5cdec82d9","local-member-attributes":"{Name:openebs-etcd-1 ClientURLs:[http://openebs-etcd-1.openebs-etcd-headless.openebs.svc.cluster.local:2379 http://openebs-etcd.openebs.svc.cluster.local:2379]}","request-path":"/0/members/6f5be5a5cdec82d9/attributes","cluster-id":"f2b3b9f758cdc80","publish-timeout":"7s"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2024-05-16T09:26:52.842Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
{"level":"info","ts":"2024-05-16T09:26:52.844Z","caller":"embed/serve.go:146","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"[::]:2379"}
{"level":"warn","ts":"2024-05-16T09:26:52.845Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:52.845Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:52.948Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:52.949Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.045Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.045Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.149Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.154Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.251Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.251Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.347Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.348Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.450Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.458Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.546Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.547Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.648Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.649Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.757Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.761Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.851Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.851Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.947Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:53.948Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.049Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.050Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 [term 3] received MsgTimeoutNow from 799d284f4d21d454 and starts an election to get leadership."}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 is starting a new election at term 3"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 became candidate at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 received MsgVoteResp from 6f5be5a5cdec82d9 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 [logterm: 3, index: 9] sent MsgVote request to 799d284f4d21d454 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 [logterm: 3, index: 9] sent MsgVote request to df54898d3b4a1323 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.057Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6f5be5a5cdec82d9 lost leader 799d284f4d21d454 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 received MsgVoteResp from 799d284f4d21d454 at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 has received 2 MsgVoteResp votes and 0 vote rejections"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6f5be5a5cdec82d9 became leader at term 4"}
{"level":"info","ts":"2024-05-16T09:26:54.063Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6f5be5a5cdec82d9 elected leader 6f5be5a5cdec82d9 at term 4"}
{"level":"error","ts":"2024-05-16T09:26:54.067Z","caller":"rafthttp/util.go:99","msg":"request sent was ignored due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-cluster-id":"f2b3b9f758cdc80","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.checkPostResponse\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/util.go:99\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).post\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:163\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).handle\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:100"}
{"level":"error","ts":"2024-05-16T09:26:54.077Z","caller":"rafthttp/util.go:99","msg":"request sent was ignored due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-cluster-id":"f2b3b9f758cdc80","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.checkPostResponse\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/util.go:99\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).post\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:163\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).handle\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:100"}
{"level":"error","ts":"2024-05-16T09:26:54.103Z","caller":"rafthttp/util.go:99","msg":"request sent was ignored due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-cluster-id":"f2b3b9f758cdc80","stacktrace":"go.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.checkPostResponse\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/util.go:99\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).post\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:163\ngo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp.(*pipeline).handle\n\tgo.etcd.io/etcd/server/v3/etcdserver/api/rafthttp/pipeline.go:100"}
{"level":"warn","ts":"2024-05-16T09:26:54.146Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.150Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"df54898d3b4a1323","remote-peer-cluster-id":"cc6a2769e8595d1d","local-member-id":"6f5be5a5cdec82d9","local-member-cluster-id":"f2b3b9f758cdc80","error":"cluster ID mismatch"}
{"level":"warn","ts":"2024-05-16T09:26:54.162Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454","error":"EOF"}
{"level":"warn","ts":"2024-05-16T09:26:54.166Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"6f5be5a5cdec82d9","remote-peer-id":"799d284f4d21d454","error":"EOF"}
...
{"level":"warn","ts":"2024-05-16T09:45:46.015Z","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:46.015Z","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:48.208Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"df54898d3b4a1323","rtt":"3.654257ms","error":"dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:48.224Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"df54898d3b4a1323","rtt":"8.09826ms","error":"dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:50.022Z","caller":"etcdserver/cluster_util.go:288","msg":"failed to reach the peer URL","address":"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}
{"level":"warn","ts":"2024-05-16T09:45:50.022Z","caller":"etcdserver/cluster_util.go:155","msg":"failed to get version","remote-member-id":"df54898d3b4a1323","error":"Get \"http://openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2380/version\": dial tcp 10.42.4.22:2380: connect: connection refused"}

Current helm values:

release:
  version: "4.0.1"
mayastor:
  enabled: true
  etcd:
    replicaCount: 3

Previous helm values:

release:
  version: "4.0.1"
mayastor:
  enabled: true
  etcd:
    replicaCount: 1

To Reproduce

Expected behavior all etcd members are healthy

Screenshots If applicable, add screenshots to help explain your problem.

OS info (please complete the following information): openebs.io/version: 2.6.1

Additional context Add any other context about the problem here.

tiagolobocastro commented 1 month ago

I don't think this will work automatically. Can you provide some points @datacore-tilangovan ?

todeb commented 1 month ago

did also a test scalling to 2 replicas, in that case etcd-0 was healthy but etcd-1 show errors:

k logs -n openebs openebs-etcd-1
Defaulted container "etcd" out of: etcd, volume-permissions (init)
etcd 10:38:13.72
etcd 10:38:13.72 Welcome to the Bitnami etcd container
etcd 10:38:13.72 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 10:38:13.72 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 10:38:13.72
etcd 10:38:13.72 INFO  ==> ** Starting etcd setup **
etcd 10:38:13.76 INFO  ==> Validating settings in ETCD_* env vars..
etcd 10:38:13.77 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 10:38:13.78 INFO  ==> Initializing etcd
etcd 10:38:13.78 INFO  ==> Generating etcd config file using env variables
etcd 10:38:13.81 INFO  ==> Detected data from previous deployments
etcd 10:38:14.00 INFO  ==> Updating member in existing cluster
{"level":"warn","ts":"2024-05-16T10:38:14.099Z","logger":"etcd-client","caller":"v3@v3.5.6/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000e6a80/openebs-etcd-0.openebs-etcd-headless.openebs.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
Error: etcdserver: member not found

Also after 1st case when I rollback from 3 to 1 replica for etcd. etcd-0 lost its all data. Fortunately I made a snapshot of etcd before and restored it.

In 2nd case when scalling from 1 to 2 and rollback from 2 to 1. etcd-0 was healthy entire time so data was kept.

tiagolobocastro commented 1 month ago

I haven't had the time yet, but I wonder if the etcd-operator would help here? https://github.com/openebs/mayastor/issues/1630