[bitnami/etcd] - New Deployments Not Clustering in v6.0.0

disfluxly commented 3 years ago

Which chart: bitnami/etcd v6.0.0

Describe the bug With a new cluster, specifying more than 1 etcd replica will result in each replica becoming its own etcd cluster.

To Reproduce Steps to reproduce the behavior:

Using the default* values, specify the replicaCount to be 3.
After they have finished spinning up, exec into each etcd pod and run etcdctl member list. Each pod will only show itself.

*Only changes made to default values was turning off rbac and updating persistent store sizes / storage class.

Expected behavior etcd pods should cluster together.

Version of Helm and Kubernetes:

Output of helm version:

version.BuildInfo{Version:"v3.5.1", GitCommit:"32c22239423b3b4ba6706d450bd044baffdcf9e6", GitTreeState:"dirty", GoVersion:"go1.15.7"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Additional context Here's a the environment set by a describe pod:

Environment:
      BITNAMI_DEBUG:                     false
      MY_POD_IP:                          (v1:status.podIP)
      MY_POD_NAME:                       etcd-0 (v1:metadata.name)
      ETCDCTL_API:                       3
      ETCD_ON_K8S:                       yes
      ETCD_START_FROM_SNAPSHOT:          no
      ETCD_DISASTER_RECOVERY:            yes
      ETCD_NAME:                         $(MY_POD_NAME)
      ETCD_DATA_DIR:                     /bitnami/etcd/data
      ETCD_LOG_LEVEL:                    info
      ALLOW_NONE_AUTHENTICATION:         yes
      ETCD_ADVERTISE_CLIENT_URLS:        http://$(MY_POD_NAME).etcd-headless.observability.svc.cluster.local:2379
      ETCD_LISTEN_CLIENT_URLS:           http://0.0.0.0:2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS:  http://$(MY_POD_NAME).etcd-headless.observability.svc.cluster.local:2380
      ETCD_LISTEN_PEER_URLS:             http://0.0.0.0:2380
      ETCD_INITIAL_CLUSTER_TOKEN:        etcd-cluster-k8s
      ETCD_INITIAL_CLUSTER_STATE:        new
      ETCD_INITIAL_CLUSTER:              etcd-0=http://etcd-0.etcd-headless.observability.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.observability.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.observability.svc.cluster.local:2380
      ETCD_CLUSTER_DOMAIN:               etcd-headless.observability.svc.cluster.local

Here's what the environment variables look like within the pod:

ETCD_MAX_WALS=3
HOSTNAME=etcd-0
ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
ETCD_MAX_SNAPSHOTS=3
ETCD_SERVICE_HOST=172.20.170.213
ETCD_PORT_2380_TCP_ADDR=172.20.170.213
ETCD_QUOTA_BACKEND_BYTES=5368709120
ETCD_PORT_2379_TCP=tcp://172.20.170.213:2379
PWD=/opt/bitnami/etcd
ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
ETCD_PORT_2380_TCP_PROTO=tcp
ETCD_ON_K8S=yes
ETCDCTL_API=3
ETCD_AUTO_COMPACTION_RETENTION=30m
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
ETCD_PORT_2379_TCP_PORT=2379
ETCD_PORT_2379_TCP_ADDR=172.20.170.213
ETCD_LOG_LEVEL=info
ETCD_HEARTBEAT_INTERVAL=150
ETCD_PORT_2380_TCP_PORT=2380
ETCD_NAME=etcd-0
ETCD_PORT_2380_TCP=tcp://172.20.170.213:2380
ETCD_CLUSTER_DOMAIN=etcd-headless.observability.svc.cluster.local
ETCD_DISASTER_RECOVERY=yes
ETCD_SNAPSHOT_COUNT=10000
ETCD_START_FROM_SNAPSHOT=no
MY_POD_NAME=etcd-0
BITNAMI_APP_NAME=etcd
ETCD_DATA_DIR=/bitnami/etcd/data
ETCD_SERVICE_PORT_CLIENT=2379
ETCD_MAX_TXN_OPS=128
ETCD_SERVICE_PORT_PEER=2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER=etcd-0=http://etcd-0.etcd-headless.observability.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.observability.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.observability.svc.cluster.local:2380
PATH=/opt/bitnami/common/bin:/opt/bitnami/etcd/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ETCD_PORT_2379_TCP_PROTO=tcp
ETCD_ADVERTISE_CLIENT_URLS=http://etcd-0.etcd-headless.observability.svc.cluster.local:2379
ETCD_PORT=tcp://172.20.170.213:2379
ETCD_SERVICE_PORT=2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-0.etcd-headless.observability.svc.cluster.local:2380

This problem only seemed to occur on new clusters. Upgrading from the the v5 helm chart to v6 on an existing cluster did not break the etcd cluster. Thus deploying out a 1 replica cluster, then subsequently scaling to a 3 replica cluster does make it so that the clustering comes together. There seems to be some problem with the ETCD_INITIAL_CLUSTER_STATE

Note: I'm running this on EKS with EFS Volumes for the persistent storage.

tl;dr workaround:

Deploy etcd with only 1 replica
Update number of replicas to X (whatever you want) in helm, and helm upgrade.

juan131 commented 3 years ago

Hi @disfluxly

I wasn't able to reproduce the issue. I installed the chart with 3 replicas and the cluster was successfully created. Could you please share the the etcd container logs during the 1st boot?

I attach my logs:

installation:

$ helm install etcd bitnami/etcd --set replicaCount=3
...
(installation notes are displayed)
...
$ helm list
NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
etcd    default     1           2021-03-15 09:37:28.851219 +0100 CET    deployed    etcd-6.0.0  3.4.15
$ kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
etcd-0   1/1     Running   0          2m45s
etcd-1   1/1     Running   0          2m45s
etcd-2   1/1     Running   0          2m45s

create etcd client and check members:

$ kubectl run etcd-client --restart='Never' --image docker.io/bitnami/etcd:3.4.15-debian-10-r9 --env ROOT_PASSWORD=$(kubectl get secret --namespace default etcd -o jsonpath="{.data.etcd-root-password}" | base64 --decode) --env ETCDCTL_ENDPOINTS="etcd.default.svc.cluster.local:2379" --namespace default --command -- sleep infinity
$ kubectl exec -it etcd-client -- bash
I have no name!@etcd-client:/opt/bitnami/etcd$ etcdctl member list
45a18acb10aa275e, started, etcd-2, http://etcd-2.etcd-headless.default.svc.cluster.local:2380, http://etcd-2.etcd-headless.default.svc.cluster.local:2379, false
936ce633ac273d75, started, etcd-1, http://etcd-1.etcd-headless.default.svc.cluster.local:2380, http://etcd-1.etcd-headless.default.svc.cluster.local:2379, false
df3b2df95cd5fd29, started, etcd-0, http://etcd-0.etcd-headless.default.svc.cluster.local:2380, http://etcd-0.etcd-headless.default.svc.cluster.local:2379, false

brumhard commented 3 years ago

For me it seems to just not work everytime but at times it does. I had runs where it just ran through successfully and created a cluster and now my last 4 runs all failed. Mostly I get the log message rafthttp: request cluster ID mismatch or rafthttp: request sent was ignored (cluster ID mismatch:... which seems to match the desribed error that each node creates it's own cluster. In these cases one node starts successfully and the others fail with the mentioned log message.

juan131 commented 3 years ago

Thanks for the feedback @brumhard @disfluxly !

I'm reviewing what can cause this and I think it could be related with the function below:

https://github.com/bitnami/bitnami-docker-etcd/blob/master/3/debian-10/rootfs/opt/bitnami/scripts/libetcd.sh#L108

Even we're using podManagementPolicy=Parallel to create every initial replica simultaneously, there could be some differences and some replicas could be initialized with a small delay. Therefore, not every replica FQDN will be added to the DNS entries at the same time.

That could be problematic since this etcdctl_get_endpoints function could return less endpoints that the actual required ones. This function is used in the recalculate_initial_cluster function and that could be causing some of the replicas to be initialized as an independent cluster, see:

https://github.com/bitnami/bitnami-docker-etcd/blob/master/3/debian-10/rootfs/opt/bitnami/scripts/libetcd.sh#L289

I think that including the change below in the run.sh should fix the issue:

diff --git a/3/debian-10/rootfs/opt/bitnami/scripts/etcd/run.sh b/3/debian-10/rootfs/opt/bitnami/scripts/etcd/run.sh
index 6d86029..c0fda67 100755
--- a/3/debian-10/rootfs/opt/bitnami/scripts/etcd/run.sh
+++ b/3/debian-10/rootfs/opt/bitnami/scripts/etcd/run.sh
@@ -21,7 +21,7 @@ set -o pipefail
 EXEC="$(command -v etcd)"

 ! is_empty_value "$ETCD_ROOT_PASSWORD" && unset ETCD_ROOT_PASSWORD
-if ! is_empty_value "$ETCD_INITIAL_CLUSTER"; then
+if ! is_empty_value "$ETCD_INITIAL_CLUSTER" && [[ "$ETCD_INITIAL_CLUSTER_STATE" != "new" ]]; then
     ETCD_INITIAL_CLUSTER="$(recalculate_initial_cluster)"
     export ETCD_INITIAL_CLUSTER
 fi

I just created a custom image including the changes above and I published it in DockerHub for you to try. Could you please give it a try? You just need to install the Bitnami etcd chart using the command below:

$ helm install etcd bitnami/etcd \
  --set replicaCount=3 \
  --set image.repository=juanariza131/etcd \
  --set image.tag=3.4.15-debian-10-r8

brumhard commented 3 years ago

Hi @juan131 thanks for the quick reply. I will give it a try.

brumhard commented 3 years ago

Seems to be working for me 👍

disfluxly commented 3 years ago

@juan131 - I think you're absolutely right. A delay in pods coming up can especially happen when using something like fargate for worker nodes as not all pods could hit scheduling at the same time.

Here's some logs from my 3 pod ETCD cluster:

etcd-0:

2021-03-15 14:49:21.773710 E | rafthttp: request cluster ID mismatch (got 62690dd5c1e46bad want 991b2451488b357d)
2021-03-15 14:49:21.872008 E | rafthttp: request cluster ID mismatch (got 62690dd5c1e46bad want 991b2451488b357d)
2021-03-15 14:49:21.874201 E | rafthttp: request cluster ID mismatch (got 62690dd5c1e46bad want 991b2451488b357d)
2021-03-15 14:49:21.971761 E | rafthttp: request cluster ID mismatch (got 62690dd5c1e46bad want 991b2451488b357d)
... repeats to infinity

etcd-1

2021-03-15 14:49:59.774815 E | rafthttp: request cluster ID mismatch (got 62690dd5c1e46bad want 9f06c4da3a5aab99)
2021-03-15 14:49:59.874677 E | rafthttp: request cluster ID mismatch (got 62690dd5c1e46bad want 9f06c4da3a5aab99)
2021-03-15 14:49:59.874826 E | rafthttp: request cluster ID mismatch (got 62690dd5c1e46bad want 9f06c4da3a5aab99)
... repeats to infinity

etcd-2

2021-03-15 14:50:22.276203 E | rafthttp: request sent was ignored (cluster ID mismatch: peer[8d0f8a30d3b67b68]=9f06c4da3a5aab99, local=62690dd5c1e46bad)
2021-03-15 14:50:22.276580 E | rafthttp: request sent was ignored (cluster ID mismatch: peer[8d0f8a30d3b67b68]=9f06c4da3a5aab99, local=62690dd5c1e46bad)
2021-03-15 14:50:22.372586 E | rafthttp: request sent was ignored (cluster ID mismatch: peer[f9ff647346dd46d]=991b2451488b357d, local=62690dd5c1e46bad)
2021-03-15 14:50:22.374270 E | rafthttp: request sent was ignored (cluster ID mismatch: peer[f9ff647346dd46d]=991b2451488b357d, local=62690dd5c1e46bad)
2021-03-15 14:50:22.376183 E | rafthttp: request sent was ignored (cluster ID mismatch: peer[8d0f8a30d3b67b68]=9f06c4da3a5aab99, local=62690dd5c1e46bad)
2021-03-15 14:50:22.376911 E | rafthttp: request sent was ignored (cluster ID mismatch: peer[8d0f8a30d3b67b68]=9f06c4da3a5aab99, local=62690dd5c1e46bad)

Results of etcdctl member list in each:

etcd-0

I have no name!@etcd-0:/opt/bitnami/etcd$ etcdctl member list
f9ff647346dd46d, started, etcd-0, http://etcd-0.etcd-headless.etcd.svc.cluster.local:2380, http://etcd-0.etcd-headless.etcd.svc.cluster.local:2379, false

etcd-1

I have no name!@etcd-1:/opt/bitnami/etcd$ etcdctl member list
8d0f8a30d3b67b68, started, etcd-1, http://etcd-1.etcd-headless.etcd.svc.cluster.local:2380, http://etcd-1.etcd-headless.etcd.svc.cluster.local:2379, false

etcd-2

{"level":"warn","ts":"2021-03-15T14:52:41.178Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-abd09271-6b08-4069-ab2f-2ad1d0b4a2c2/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Error: context deadline exceeded

etcd-2 never hits ready status, hence the error.

Note that this behavior is inconsistent. In the past I've seen 3 completely separate clusters spin up, with each etcd coming ready. This appears to be more of a mismatch on expected cluster to connect to. To me this identifies what you said before, essentially a race condition on fqdn.

Unfortunately I'm limited on images I can pull, so I'll rely on @brumhard to test (thanks btw!)

juan131 commented 3 years ago

@brumhard @disfluxly thanks for your insights!

We'll release a revision of the etcd image including a variant of the solution I shared before during the next day. I'll notify you as soon as the new image is released and included in the etcd chart by default.

Thanks again for helping us to improve the chart!

juan131 commented 3 years ago

@brumhard @disfluxly the new chart version using the new revision of the etcd image was just released, see:

https://github.com/bitnami/charts/commit/182114e06decb91fa3eb07ee0cf14b4cf74da724

Could you please give it a try?

Find the changes that we added to the image in the commit below:

https://github.com/bitnami/bitnami-docker-etcd/commit/63b56323c7d734a9ceb36efce86f8e5601025bf4

brumhard commented 3 years ago

@juan131 I've just checked the deployment and it looks like it's working now. Thank you very much for the fast help. Much appreciated 👍

brumhard commented 3 years ago

Wait nvm on my one cluster it just works, on the other one it still fails with the rafthttp: request sent was ignored (cluster ID mismatch error messages even with the new chart.

The following desribes my sts:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
  creationTimestamp: "2021-03-16T14:42:20Z"
  generation: 1
  labels:
    app.kubernetes.io/instance: redacted-etcd
    app.kubernetes.io/managed-by: pulumi
    app.kubernetes.io/name: etcd
    helm.sh/chart: etcd-6.1.2
  name: redacted-etcd
  namespace: redacted-prd
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: redacted-etcd
      app.kubernetes.io/name: etcd
  serviceName: redacted-etcd-headless
  template:
    metadata:
      annotations:
        prometheus.io/port: "2379"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: redacted-etcd
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: etcd
        helm.sh/chart: etcd-6.1.2
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: redacted-etcd
                  app.kubernetes.io/name: etcd
              namespaces:
              - redacted-prd
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: ETCDCTL_API
          value: "3"
        - name: ETCD_ON_K8S
          value: "yes"
        - name: ETCD_START_FROM_SNAPSHOT
          value: "no"
        - name: ETCD_DISASTER_RECOVERY
          value: "no"
        - name: ETCD_NAME
          value: $(MY_POD_NAME)
        - name: ETCD_DATA_DIR
          value: /bitnami/etcd/data
        - name: ETCD_LOG_LEVEL
          value: info
        - name: ALLOW_NONE_AUTHENTICATION
          value: "no"
        - name: ETCD_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: etcd-root-password
              name: redacted-etcd-root-password
        - name: ETCD_ADVERTISE_CLIENT_URLS
          value: https://$(MY_POD_NAME).redacted-etcd-headless.redacted-prd.svc.cluster.local:2379
        - name: ETCD_LISTEN_CLIENT_URLS
          value: https://0.0.0.0:2379
        - name: ETCD_INITIAL_ADVERTISE_PEER_URLS
          value: https://$(MY_POD_NAME).redacted-etcd-headless.redacted-prd.svc.cluster.local:2380
        - name: ETCD_LISTEN_PEER_URLS
          value: https://0.0.0.0:2380
        - name: ETCD_AUTO_COMPACTION_RETENTION
          value: 6h
        - name: ETCD_INITIAL_CLUSTER_TOKEN
          value: etcd-cluster-k8s
        - name: ETCD_INITIAL_CLUSTER_STATE
          value: new
        - name: ETCD_INITIAL_CLUSTER
          value: redacted-etcd-0=https://redacted-etcd-0.redacted-etcd-headless.redacted-prd.svc.cluster.local:2380,redacted-etcd-1=https://redacted-etcd-1.redacted-etcd-headless.redacted-prd.svc.cluster.local:2380,redacted-etcd-2=https://redacted-etcd-2.redacted-etcd-headless.redacted-prd.svc.cluster.local:2380
        - name: ETCD_CLUSTER_DOMAIN
          value: redacted-etcd-headless.redacted-prd.svc.cluster.local
        - name: ETCD_CERT_FILE
          value: /opt/bitnami/etcd/certs/client/tls.crt
        - name: ETCD_KEY_FILE
          value: /opt/bitnami/etcd/certs/client/tls.key
        - name: ETCD_CLIENT_CERT_AUTH
          value: "true"
        - name: ETCD_TRUSTED_CA_FILE
          value: /opt/bitnami/etcd/certs/client/ca.crt
        - name: ETCD_PEER_CERT_FILE
          value: /opt/bitnami/etcd/certs/peer/tls.crt
        - name: ETCD_PEER_KEY_FILE
          value: /opt/bitnami/etcd/certs/peer/tls.key
        - name: ETCD_PEER_CLIENT_CERT_AUTH
          value: "true"
        - name: ETCD_PEER_TRUSTED_CA_FILE
          value: /opt/bitnami/etcd/certs/peer/ca.crt
        image: docker.io/bitnami/etcd:3.4.15-debian-10-r14
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /opt/bitnami/scripts/etcd/prestop.sh
        livenessProbe:
          exec:
            command:
            - /opt/bitnami/scripts/etcd/healthcheck.sh
          failureThreshold: 5
          initialDelaySeconds: 60
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        name: etcd
        ports:
        - containerPort: 2379
          name: client
          protocol: TCP
        - containerPort: 2380
          name: peer
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /opt/bitnami/scripts/etcd/healthcheck.sh
          failureThreshold: 5
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /bitnami/etcd
          name: data
        - mountPath: /opt/bitnami/etcd/certs/client/
          name: etcd-client-certs
          readOnly: true
        - mountPath: /opt/bitnami/etcd/certs/peer/
          name: etcd-peer-certs
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      terminationGracePeriodSeconds: 30
      volumes:
      - name: etcd-client-certs
        secret:
          defaultMode: 256
          secretName: redacted-etcd-client-cert
      - name: etcd-peer-certs
        secret:
          defaultMode: 256
          secretName: redacted-etcd-peer-cert
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      volumeMode: Filesystem
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 3
  currentRevision: redacted-etcd-5778b8cb47
  observedGeneration: 1
  readyReplicas: 1
  replicas: 3
  updateRevision: redacted-etcd-5778b8cb47
  updatedReplicas: 3

juan131 commented 3 years ago

Hi @brumhard

Did you find the error on a clean environment? I mean, did you remove the PVCs from previous releases before installing the chart? I installed several chart releases and I found no issues on my side.

brumhard commented 3 years ago

Yes I did a full delete and install, but sth's odd with that cluster anyways so it could as well be another issue.

disfluxly commented 3 years ago

I'll be able to test this out in my setup either tomorrow or Friday. @juan131 I'll let you know the results! Thanks for the quick update

juan131 commented 3 years ago

Thanks both! You might also find interesting the conversation at https://github.com/bitnami/charts/issues/3190

haroonb commented 3 years ago

I have tried to upgrade to upgrade our old etcd deployment from helmv2 to helmv3. As far as I could tell this is the preferred method on switching charts, please correct me if I'm wrong. This worked fine with the mongodb chart.

Steps to reproduce the issue:

Install etcd using helmv2 chart

41                     /snap/bin/helm install etcd bitnami/etcd
42                     -f /tmp/etcdvalues.yaml
43                     --version 4.9.1
44                     --set statefulset.replicaCount=3
45                     --set image.tag=3.4.10-debian-10-r1
46                     --set disasterRecovery.pvc.existingClaim=etcd-snapshot-volume
47                     --set disasterRecovery.pvc.size=16Gi
48                     --set persistence.size=16Gi
49                     --set podAntiAffinity=soft
50                     --set disasterRecovery.pvc.storageClassName=nfs

wait for cluster to be running, write data and wait for a few snapshots
/snap/bin/helm delete etcd

Install etcd using helmv3 chart

10                 /snap/bin/helm install etcd bitnami/etcd
9                 --version 6.1.2
8                 -f /tmp/etcdvalues.yaml
7                 --set replicaCount=3
6                 --set image.tag=3.4.15-debian-10-r14
5                 --set metrics.enabled=true
4                 --set auth.rbac.enabled=false
3                 --set disasterRecovery.pvc.existingClaim=etcd-snapshot-volume
2                 --set persistence.enabled=true
1                 --set persistence.storageClass=rook-ceph-block
82                  --set persistence.size={{ etcd_pvsize }}Gi
1                 --set podAntiAffinity=soft

Describe the results you received:

I have a cluster recognise it's members.

Describe the results you expected:

I expected the cluster to use the old data and just come back up.

@perftest-m1:~$ kubectl get po | grep etcd
etcd-0                                    0/1     CrashLoopBackOff   17         70m
etcd-1                                    0/1     CrashLoopBackOff   17         70m
etcd-2                                    1/1     Running            0          70m

USER-SUPPLIED VALUES:
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/instance
            operator: In
            values:
            - etcd
        topologyKey: kubernetes.io/hostname
      weight: 100
args: []
auth:
  client:
    caFilename: ""
    certFilename: cert.pem
    certKeyFilename: key.pem
    enableAuthentication: false
    existingSecret: ""
    secureTransport: false
    useAutoTLS: false
  peer:
    caFilename: ""
    certFilename: cert.pem
    certKeyFilename: key.pem
    enableAuthentication: false
    existingSecret: ""
    secureTransport: false
    useAutoTLS: false
  rbac:
    allowNoneAuthentication: true
    enabled: false
    existingSecret: ""
    rootPassword: ""
clusterDomain: cluster.local
command: []
commonAnnotations: {}
commonLabels: {}
containerPorts:
  client: 2379
  peer: 2380
containerSecurityContext:
  enabled: true
  runAsNonRoot: true
  runAsUser: 1001
customLivenessProbe: {}
customReadinessProbe: {}
customStartupProbe: {}
disasterRecovery:
  cronjob:
    historyLimit: 1
    podAnnotations: {}
    resources:
      limits: {}
      requests: {}
    schedule: '*/30 * * * *'
    snapshotHistoryLimit: 1
  enabled: false
  pvc:
    existingClaim: etcd-snapshot-volume
    size: 2Gi
    storageClassName: nfs
extraDeploy: []
extraEnvVars: []
extraEnvVarsCM: ""
extraEnvVarsSecret: ""
extraVolumeMounts: []
extraVolumes: []
hostAliases: []
image:
  debug: false
  pullPolicy: IfNotPresent
  pullSecrets: []
  registry: docker.io
  repository: bitnami/etcd
  tag: 3.4.15-debian-10-r14
initContainers: {}
initialClusterState: ""
kubeVersion: ""
lifecycleHooks: {}
livenessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 60
  periodSeconds: 30
  successThreshold: 1
  timeoutSeconds: 5
metrics:
  enabled: true
  podAnnotations:
    prometheus.io/port: "2379"
    prometheus.io/scrape: "true"
  podMonitor:
    additionalLabels: {}
    enabled: false
    interval: 30s
    scheme: http
    tlsConfig: {}
nodeAffinityPreset:
  key: ""
  type: ""
  values: []
nodeSelector: {}
pdb:
  create: false
  minAvailable: 1
persistence:
  accessModes:
  - ReadWriteOnce
  annotations: {}
  enabled: true
  selector: {}
  size: 2Gi
  storageClass: rook-ceph-block
podAffinityPreset: ""
podAnnotations:
  backup.velero.io/backup-volumes: data
podAntiAffinity: soft
podAntiAffinityPreset: soft
podLabels: {}
podManagementPolicy: Parallel
podSecurityContext:
  enabled: true
  fsGroup: 1001
priorityClassName: ""
readinessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
removeMemberOnContainerTermination: true
replicaCount: 3
resources:
  limits: {}
  requests: {}
service:
  annotations: {}
  clientPortNameOverride: ""
  externalIPs: []
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  nodePorts:
    clientPort: ""
    peerPort: ""
  peerPort: 2380
  peerPortNameOverride: ""
  port: 2379
  type: ClusterIP
sidecars: {}
startFromSnapshot:
  enabled: false
  existingClaim: ""
  snapshotFilename: ""
startupProbe:
  enabled: false
  failureThreshold: 60
  initialDelaySeconds: 0
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
tolerations: []
updateStrategy:
  type: RollingUpdate
volumePermissions:
  enabled: false
  image:
    pullPolicy: Always
    pullSecrets: []
    registry: docker.io
    repository: bitnami/bitnami-shell
    tag: "10"
  resources:
    limits: {}
    requests: {}

perftest-m1:~$ kubectl logs etcd-0
etcd 14:40:32.28
etcd 14:40:32.28 Welcome to the Bitnami etcd container
etcd 14:40:32.29 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 14:40:32.29 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 14:40:32.29
etcd 14:40:32.29 INFO  ==> ** Starting etcd setup **
etcd 14:40:32.30 INFO  ==> Validating settings in ETCD_* env vars..
etcd 14:40:32.30 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 14:40:32.31 INFO  ==> Initializing etcd
etcd 14:40:32.31 INFO  ==> Detected data from previous deployments
etcd 14:40:47.40 WARN  ==> Cluster not responding!
etcd 14:40:47.40 WARN  ==> Disaster recovery is disabled, the cluster will try to recover on it's own
etcd 14:40:47.41 INFO  ==> ** etcd setup finished! **

etcd 14:40:47.42 INFO  ==> ** Starting etcd **
2021-03-17 14:40:47.434920 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd-0.etcd-headless.default.svc.cluster.local:2379
2021-03-17 14:40:47.434981 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false
2021-03-17 14:40:47.434991 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
2021-03-17 14:40:47.435016 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-0.etcd-headless.default.svc.cluster.local:2380
2021-03-17 14:40:47.435027 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
2021-03-17 14:40:47.435031 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2021-03-17 14:40:47.435034 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
2021-03-17 14:40:47.435043 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
2021-03-17 14:40:47.435055 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
2021-03-17 14:40:47.435061 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=info
2021-03-17 14:40:47.435069 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-0
2021-03-17 14:40:47.435076 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false
2021-03-17 14:40:47.435101 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE=
2021-03-17 14:40:47.435112 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.107.248.12
2021-03-17 14:40:47.435116 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_ADDR=10.107.248.12
2021-03-17 14:40:47.435120 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.107.248.12:2379
2021-03-17 14:40:47.435132 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes
2021-03-17 14:40:47.435137 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PROTO=tcp
2021-03-17 14:40:47.435140 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots
2021-03-17 14:40:47.435145 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin
2021-03-17 14:40:47.435149 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd
2021-03-17 14:40:47.435154 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
2021-03-17 14:40:47.435176 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.107.248.12
2021-03-17 14:40:47.435183 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PORT=2380
2021-03-17 14:40:47.435189 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD=
2021-03-17 14:40:47.435192 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP=tcp://10.107.248.12:2380
2021-03-17 14:40:47.435196 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=etcd-headless.default.svc.cluster.local
2021-03-17 14:40:47.435198 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no
2021-03-17 14:40:47.435201 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE=
2021-03-17 14:40:47.435208 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd
2021-03-17 14:40:47.435210 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=no
2021-03-17 14:40:47.435216 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_CLIENT=2379
2021-03-17 14:40:47.435222 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot
2021-03-17 14:40:47.435233 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_PEER=2380
2021-03-17 14:40:47.435239 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd
2021-03-17 14:40:47.435244 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
2021-03-17 14:40:47.435248 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE=
2021-03-17 14:40:47.435250 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.107.248.12:2379
2021-03-17 14:40:47.435252 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
2021-03-17 14:40:47.435254 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs
2021-03-17 14:40:47.435256 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-17 14:40:47.435280 I | etcdmain: etcd Version: 3.4.15
2021-03-17 14:40:47.435285 I | etcdmain: Git SHA: aa7126864
2021-03-17 14:40:47.435287 I | etcdmain: Go Version: go1.12.17
2021-03-17 14:40:47.435299 I | etcdmain: Go OS/Arch: linux/amd64
2021-03-17 14:40:47.435302 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
2021-03-17 14:40:47.435342 W | etcdmain: found invalid file/dir member_id under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
2021-03-17 14:40:47.435349 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-17 14:40:47.435470 I | embed: name = etcd-0
2021-03-17 14:40:47.435478 I | embed: data dir = /bitnami/etcd/data
2021-03-17 14:40:47.435480 I | embed: member dir = /bitnami/etcd/data/member
2021-03-17 14:40:47.435483 I | embed: heartbeat = 100ms
2021-03-17 14:40:47.435485 I | embed: election = 1000ms
2021-03-17 14:40:47.435488 I | embed: snapshot count = 100000
2021-03-17 14:40:47.435493 I | embed: advertise client URLs = http://etcd-0.etcd-headless.default.svc.cluster.local:2379
2021-03-17 14:40:47.435497 I | embed: initial advertise peer URLs = http://etcd-0.etcd-headless.default.svc.cluster.local:2380
2021-03-17 14:40:47.435501 I | embed: initial cluster =
2021-03-17 14:40:47.437768 I | etcdserver: restarting member df3b2df95cd5fd29 in cluster 9e98e654be1e22d7 at commit index 33
raft2021/03/17 14:40:47 INFO: df3b2df95cd5fd29 switched to configuration voters=()
raft2021/03/17 14:40:47 INFO: df3b2df95cd5fd29 became follower at term 13
raft2021/03/17 14:40:47 INFO: newRaft df3b2df95cd5fd29 [peers: [], term: 13, commit: 33, applied: 0, lastindex: 34, lastterm: 13]
2021-03-17 14:40:47.440170 W | auth: simple token is not cryptographically signed
2021-03-17 14:40:47.444589 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
raft2021/03/17 14:40:47 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030)
2021-03-17 14:40:47.445079 I | etcdserver/membership: added member 45a18acb10aa275e [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
2021-03-17 14:40:47.445106 I | rafthttp: starting peer 45a18acb10aa275e...
2021-03-17 14:40:47.445143 I | rafthttp: started HTTP pipelining with peer 45a18acb10aa275e
2021-03-17 14:40:47.445354 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
2021-03-17 14:40:47.446050 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
2021-03-17 14:40:47.446768 I | rafthttp: started peer 45a18acb10aa275e
2021-03-17 14:40:47.446800 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream Message reader)
2021-03-17 14:40:47.446809 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
2021-03-17 14:40:47.446895 I | rafthttp: added peer 45a18acb10aa275e
raft2021/03/17 14:40:47 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805)
2021-03-17 14:40:47.447145 I | etcdserver/membership: added member 936ce633ac273d75 [http://etcd-1.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
2021-03-17 14:40:47.447177 I | rafthttp: starting peer 936ce633ac273d75...
2021-03-17 14:40:47.447210 I | rafthttp: started HTTP pipelining with peer 936ce633ac273d75
2021-03-17 14:40:47.447656 I | embed: listening for peers on [::]:2380
2021-03-17 14:40:47.448724 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
2021-03-17 14:40:47.448861 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
2021-03-17 14:40:47.449770 E | etcdserver: the member has been permanently removed from the cluster
2021-03-17 14:40:47.449788 I | etcdserver: the data-dir used by this member must be removed.
2021-03-17 14:40:47.449817 I | etcdserver: aborting publish because server is stopped
2021-03-17 14:40:47.450805 I | rafthttp: started peer 936ce633ac273d75
2021-03-17 14:40:47.450827 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
2021-03-17 14:40:47.450854 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream Message reader)
2021-03-17 14:40:47.450916 I | rafthttp: added peer 936ce633ac273d75
raft2021/03/17 14:40:47 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805 16085501043111492905)
2021-03-17 14:40:47.451082 I | etcdserver/membership: added member df3b2df95cd5fd29 [http://etcd-0.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
2021-03-17 14:40:47.451188 N | etcdserver/membership: set the initial cluster version to 3.0
2021-03-17 14:40:47.451242 I | etcdserver/api: enabled capabilities for version 3.0
2021-03-17 14:40:47.451282 N | etcdserver/membership: updated the cluster version from 3.0 to 3.4
2021-03-17 14:40:47.451311 I | etcdserver/api: enabled capabilities for version 3.4
raft2021/03/17 14:40:47 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805)
2021-03-17 14:40:47.451835 I | etcdserver/membership: removed member df3b2df95cd5fd29 from cluster 9e98e654be1e22d7
2021-03-17 14:40:47.451866 I | rafthttp: stopping peer 45a18acb10aa275e...
2021-03-17 14:40:47.451883 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
2021-03-17 14:40:47.451892 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
2021-03-17 14:40:47.451912 I | rafthttp: stopped HTTP pipelining with peer 45a18acb10aa275e
2021-03-17 14:40:47.451937 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
2021-03-17 14:40:47.451954 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream Message reader)
2021-03-17 14:40:47.451965 I | rafthttp: stopped peer 45a18acb10aa275e
2021-03-17 14:40:47.451970 I | rafthttp: stopping peer 936ce633ac273d75...
2021-03-17 14:40:47.451982 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
2021-03-17 14:40:47.451989 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
2021-03-17 14:40:47.452022 I | rafthttp: stopped HTTP pipelining with peer 936ce633ac273d75
2021-03-17 14:40:47.452041 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
2021-03-17 14:40:47.452053 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream Message reader)
2021-03-17 14:40:47.452057 I | rafthttp: stopped peer 936ce633ac273d75

perftest-m1:~$ kubectl logs etcd-2
etcd 13:39:56.29
etcd 13:39:56.29 Welcome to the Bitnami etcd container
etcd 13:39:56.29 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd 13:39:56.30 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd 13:39:56.30
etcd 13:39:56.30 INFO  ==> ** Starting etcd setup **
etcd 13:39:56.31 INFO  ==> Validating settings in ETCD_* env vars..
etcd 13:39:56.31 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 13:39:56.32 INFO  ==> Initializing etcd
etcd 13:39:56.33 INFO  ==> Detected data from previous deployments
etcd 13:40:16.42 WARN  ==> Cluster not responding!
etcd 13:40:16.42 WARN  ==> Disaster recovery is disabled, the cluster will try to recover on it's own
etcd 13:40:16.43 INFO  ==> ** etcd setup finished! **

etcd 13:40:16.44 INFO  ==> ** Starting etcd **
2021-03-17 13:40:16.456572 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd-2.etcd-headless.default.svc.cluster.local:2379
2021-03-17 13:40:16.456631 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false
2021-03-17 13:40:16.456639 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
2021-03-17 13:40:16.456661 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
2021-03-17 13:40:16.456670 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
2021-03-17 13:40:16.456674 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new
2021-03-17 13:40:16.456679 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
2021-03-17 13:40:16.456686 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
2021-03-17 13:40:16.456695 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
2021-03-17 13:40:16.456699 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=info
2021-03-17 13:40:16.456709 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-2
2021-03-17 13:40:16.456714 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false
2021-03-17 13:40:16.456743 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE=
2021-03-17 13:40:16.456758 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.107.248.12
2021-03-17 13:40:16.456762 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_ADDR=10.107.248.12
2021-03-17 13:40:16.456766 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.107.248.12:2379
2021-03-17 13:40:16.456779 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes
2021-03-17 13:40:16.456783 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PROTO=tcp
2021-03-17 13:40:16.456787 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots
2021-03-17 13:40:16.456793 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin
2021-03-17 13:40:16.456799 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd
2021-03-17 13:40:16.456806 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
2021-03-17 13:40:16.456809 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.107.248.12
2021-03-17 13:40:16.456816 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PORT=2380
2021-03-17 13:40:16.456824 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD=
2021-03-17 13:40:16.456827 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP=tcp://10.107.248.12:2380
2021-03-17 13:40:16.456833 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=etcd-headless.default.svc.cluster.local
2021-03-17 13:40:16.456835 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no
2021-03-17 13:40:16.456839 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE=
2021-03-17 13:40:16.456848 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd
2021-03-17 13:40:16.456851 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=no
2021-03-17 13:40:16.456859 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_CLIENT=2379
2021-03-17 13:40:16.456867 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot
2021-03-17 13:40:16.456875 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_PEER=2380
2021-03-17 13:40:16.456878 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd
2021-03-17 13:40:16.456884 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
2021-03-17 13:40:16.456891 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE=
2021-03-17 13:40:16.456893 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.107.248.12:2379
2021-03-17 13:40:16.456895 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
2021-03-17 13:40:16.456898 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs
2021-03-17 13:40:16.456900 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-17 13:40:16.456928 I | etcdmain: etcd Version: 3.4.15
2021-03-17 13:40:16.456933 I | etcdmain: Git SHA: aa7126864
2021-03-17 13:40:16.456936 I | etcdmain: Go Version: go1.12.17
2021-03-17 13:40:16.456938 I | etcdmain: Go OS/Arch: linux/amd64
2021-03-17 13:40:16.456941 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
2021-03-17 13:40:16.457000 W | etcdmain: found invalid file/dir member_id under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
2021-03-17 13:40:16.457007 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-17 13:40:16.457166 I | embed: name = etcd-2
2021-03-17 13:40:16.457176 I | embed: data dir = /bitnami/etcd/data
2021-03-17 13:40:16.457179 I | embed: member dir = /bitnami/etcd/data/member
2021-03-17 13:40:16.457181 I | embed: heartbeat = 100ms
2021-03-17 13:40:16.457183 I | embed: election = 1000ms
2021-03-17 13:40:16.457185 I | embed: snapshot count = 100000
2021-03-17 13:40:16.457193 I | embed: advertise client URLs = http://etcd-2.etcd-headless.default.svc.cluster.local:2379
2021-03-17 13:40:16.457199 I | embed: initial advertise peer URLs = http://etcd-2.etcd-headless.default.svc.cluster.local:2380
2021-03-17 13:40:16.457204 I | embed: initial cluster =
2021-03-17 13:40:16.465254 I | etcdserver: restarting member 45a18acb10aa275e in cluster 9e98e654be1e22d7 at commit index 36
raft2021/03/17 13:40:16 INFO: 45a18acb10aa275e switched to configuration voters=()
raft2021/03/17 13:40:16 INFO: 45a18acb10aa275e became follower at term 14
raft2021/03/17 13:40:16 INFO: newRaft 45a18acb10aa275e [peers: [], term: 14, commit: 36, applied: 0, lastindex: 36, lastterm: 14]
2021-03-17 13:40:16.467499 W | auth: simple token is not cryptographically signed
2021-03-17 13:40:16.471870 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
raft2021/03/17 13:40:16 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030)
2021-03-17 13:40:16.472382 I | etcdserver/membership: added member 45a18acb10aa275e [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
raft2021/03/17 13:40:16 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030 10623118730666130805)
2021-03-17 13:40:16.472495 I | etcdserver/membership: added member 936ce633ac273d75 [http://etcd-1.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
2021-03-17 13:40:16.472514 I | rafthttp: starting peer 936ce633ac273d75...
2021-03-17 13:40:16.472539 I | rafthttp: started HTTP pipelining with peer 936ce633ac273d75
2021-03-17 13:40:16.473057 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
2021-03-17 13:40:16.473128 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
2021-03-17 13:40:16.473746 I | rafthttp: started peer 936ce633ac273d75
2021-03-17 13:40:16.473786 I | rafthttp: added peer 936ce633ac273d75
2021-03-17 13:40:16.474221 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream Message reader)
raft2021/03/17 13:40:16 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030 10623118730666130805 16085501043111492905)
2021-03-17 13:40:16.474238 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
2021-03-17 13:40:16.474301 I | etcdserver/membership: added member df3b2df95cd5fd29 [http://etcd-0.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
2021-03-17 13:40:16.474317 I | rafthttp: starting peer df3b2df95cd5fd29...
2021-03-17 13:40:16.474338 I | rafthttp: started HTTP pipelining with peer df3b2df95cd5fd29
2021-03-17 13:40:16.475814 I | embed: listening for peers on [::]:2380
2021-03-17 13:40:16.475832 I | rafthttp: started peer df3b2df95cd5fd29
2021-03-17 13:40:16.475861 I | rafthttp: added peer df3b2df95cd5fd29
2021-03-17 13:40:16.475890 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
2021-03-17 13:40:16.475999 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream Message reader)
2021-03-17 13:40:16.476021 N | etcdserver/membership: set the initial cluster version to 3.0
2021-03-17 13:40:16.476062 I | etcdserver/api: enabled capabilities for version 3.0
2021-03-17 13:40:16.476159 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
2021-03-17 13:40:16.476171 N | etcdserver/membership: updated the cluster version from 3.0 to 3.4
2021-03-17 13:40:16.476202 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
2021-03-17 13:40:16.476246 I | etcdserver/api: enabled capabilities for version 3.4
raft2021/03/17 13:40:16 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030 10623118730666130805)
2021-03-17 13:40:16.477102 I | etcdserver/membership: removed member df3b2df95cd5fd29 from cluster 9e98e654be1e22d7
2021-03-17 13:40:16.477155 I | rafthttp: stopping peer df3b2df95cd5fd29...
2021-03-17 13:40:16.477208 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (writer)
2021-03-17 13:40:16.477252 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (writer)
2021-03-17 13:40:16.477284 I | rafthttp: stopped HTTP pipelining with peer df3b2df95cd5fd29
2021-03-17 13:40:16.477368 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
2021-03-17 13:40:16.477423 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (stream Message reader)
2021-03-17 13:40:16.477450 I | rafthttp: stopped peer df3b2df95cd5fd29
2021-03-17 13:40:16.477530 I | rafthttp: removed peer df3b2df95cd5fd29
raft2021/03/17 13:40:16 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030)
2021-03-17 13:40:16.477706 I | etcdserver/membership: removed member 936ce633ac273d75 from cluster 9e98e654be1e22d7
2021-03-17 13:40:16.477756 I | rafthttp: stopping peer 936ce633ac273d75...
2021-03-17 13:40:16.477809 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
2021-03-17 13:40:16.477856 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
2021-03-17 13:40:16.477918 I | rafthttp: stopped HTTP pipelining with peer 936ce633ac273d75
2021-03-17 13:40:16.477954 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
2021-03-17 13:40:16.478012 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream Message reader)
2021-03-17 13:40:16.478061 I | rafthttp: stopped peer 936ce633ac273d75
2021-03-17 13:40:16.478106 I | rafthttp: removed peer 936ce633ac273d75
raft2021/03/17 13:40:17 INFO: 45a18acb10aa275e is starting a new election at term 14
raft2021/03/17 13:40:17 INFO: 45a18acb10aa275e became candidate at term 15
raft2021/03/17 13:40:17 INFO: 45a18acb10aa275e received MsgVoteResp from 45a18acb10aa275e at term 15
raft2021/03/17 13:40:17 INFO: 45a18acb10aa275e became leader at term 15
raft2021/03/17 13:40:17 INFO: raft.node: 45a18acb10aa275e elected leader 45a18acb10aa275e at term 15
2021-03-17 13:40:17.970650 I | etcdserver: published {Name:etcd-2 ClientURLs:[http://etcd-2.etcd-headless.default.svc.cluster.local:2379]} to cluster 9e98e654be1e22d7
2021-03-17 13:40:17.970685 I | embed: ready to serve client requests
2021-03-17 13:40:17.971439 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
2021-03-17 13:40:18.032905 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:18.033236 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:18.034979 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:18.035111 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:19.635811 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:40:19.635964 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:40:19.638747 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:40:19.638833 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:40:33.879174 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:33.879353 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:33.880416 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:33.880461 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:40:35.730536 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:40:35.730652 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:40:35.733671 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:40:35.733777 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:41:00.439669 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:41:00.439754 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:41:00.441231 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:41:00.441249 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:41:07.685121 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:41:07.685337 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:41:07.686756 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:41:07.686778 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:41:45.479736 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:41:45.479932 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:41:49.695483 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:41:49.695803 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:42:42.442452 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:42:42.442607 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:42:42.444130 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:42:42.444230 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:42:47.689695 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:42:47.689807 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:42:47.691718 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:42:47.691965 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:44:23.474679 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:44:23.474851 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:44:23.696657 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:44:23.697429 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:44:23.698314 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:44:23.699013 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:47:27.494528 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:47:27.494790 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:47:27.496125 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:47:27.496350 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:47:28.885733 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:47:28.885845 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:47:28.887493 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:47:28.887510 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:52:46.705292 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:52:46.705613 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:52:46.706807 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:52:46.706930 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:52:52.462399 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:52:52.462503 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:52:52.464209 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:52:52.464318 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:58:07.741378 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:58:07.741591 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:58:07.742957 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:58:07.743042 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 13:58:10.490865 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 13:58:10.490991 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:03:28.696264 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:03:28.696497 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:03:28.697695 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:03:28.698604 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:03:30.443516 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:03:30.443565 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:03:30.445382 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:03:30.445820 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:08:51.484912 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:08:51.485203 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:08:51.486502 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:08:51.486586 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:08:56.669844 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:08:56.669926 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:08:56.671420 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:08:56.671526 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:14:12.513691 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:14:12.513744 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:14:12.515403 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:14:12.515556 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:14:14.671902 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:14:14.672011 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:14:14.673808 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:14:14.673918 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:19:28.474162 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:19:28.474399 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:19:28.475983 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:19:28.476226 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:19:32.643861 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:19:32.643918 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:24:51.455581 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:24:51.455698 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:24:51.457804 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:24:51.457835 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:24:55.687877 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:24:55.688535 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:30:09.457385 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:30:09.457893 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:30:11.660139 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:30:11.660164 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:35:26.449106 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:35:26.449372 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:35:38.703341 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:35:38.703393 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:35:38.704905 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:35:38.704930 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:40:47.444623 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:40:47.444861 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:40:47.446127 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:40:47.446179 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:40:54.648066 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:40:54.648146 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:40:54.649909 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:40:54.650037 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:46:10.650891 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:46:10.651024 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:46:10.653628 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:46:10.653766 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
2021-03-17 14:46:12.441054 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:46:12.441081 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:46:12.442448 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
2021-03-17 14:46:12.442469 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed

juan131 commented 3 years ago

Hi @haroonb

Taking into account that you have an existing snapshot from you 1st installation (since it seems you enabled the "disaster-recovery feature"), you should be able to initialize the new etcd cluster recovering the snapshot. To do so, you'll have to use the parameters below:

Parameter	Description	Default
`startFromSnapshot.enabled`	Initialize new cluster recovering an existing snapshot	`false`
`startFromSnapshot.existingClaim`	PVC containing the existing snapshot	`nil`
`startFromSnapshot.snapshotFilename`	Snapshot filename	`nil`

Could you give that a try?

disfluxly commented 3 years ago

@juan131 - Sorry for the delay, tested out the changes this morning and the cluster came up smoothly!

juan131 commented 3 years ago

That's great @disfluxly ! Glad to hear that

haroonb commented 3 years ago

Edit: I'm pretty sure, that the cluster doesn't start from the snapshot. Because my db is a lot smaller, than the snapshot I try to recover from.

devops-m1:/data/archive/etcd-snaps$ ls -l
total 11220
-rw-r--r-- 1 1001 root 11485216 Mar 24 17:06 db

devops-m1:/data/archive/etcd-snaps$ kubectl exec -it etcd-0 -- etcdctl endpoint status --endpoints=http://etcd-2.etcd-headless.default.svc.cluster.local:2380,http://etcd-1.etcd-headless.default.svc.cluster.local:2380,http://etcd-0.etcd-headless.default.svc.cluster.local:2380 -w table
+------------------------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|                          ENDPOINT                          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://etcd-2.etcd-headless.default.svc.cluster.local:2380 | e3d4a2fb9a237360 |  3.4.15 |  111 kB |     false |      false |        28 |         49 |                 49 |        |
| http://etcd-1.etcd-headless.default.svc.cluster.local:2380 | 936ce633ac273d75 |  3.4.15 |  119 kB |     false |      false |        28 |         49 |                 49 |        |
| http://etcd-0.etcd-headless.default.svc.cluster.local:2380 | df3b2df95cd5fd29 |  3.4.15 |  119 kB |      true |      false |        28 |         49 |                 49 |        |
+------------------------------------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Hi @juan131 thanks for the suggestion I tried to upgrade like you suggested on a new cluster and it worked fine, but when I tried to fix the broken cluster that way it did not work. All of my test-cluster are basically the same only difference being the age and the amount of workers they have. And the upgrade process is also always the same ( executed with ansible ). I will attach the details of both cluster, maybe some can see my mistake or what is going on.

As far as I can see the broken cluster still has the issue, that the other member are not recognised. It also seems, that the cluster doesn't start from the snapshot.

                /snap/bin/helm install etcd bitnami/etcd
                --version 6.1.2
                -f /tmp/etcdvalues.yaml
                --set replicaCount=3
                --set image.tag=3.4.15-debian-10-r14
                --set image.debug=true
                --set metrics.enabled=true
                --set auth.rbac.enabled=false
                --set disasterRecovery.pvc.existingClaim=etcd-snapshot-volume
                --set persistence.enabled=true
                --set persistence.storageClass=rook-ceph-block
                --set persistence.size={{ etcd_pvsize }}Gi
                --set podAntiAffinity=soft
                --set startFromSnapshot.enabled=true
                --set startFromSnapshot.existingClaim=etcd-snapshot-volume
                --set startFromSnapshot.snapshotFilename=db
                --set initialClusterState=existing

Cluster 1 ( broken ):

perftest-m1:~$ kubectl get po | grep etcd
etcd-0                                    0/1     CrashLoopBackOff   6          10m
etcd-1                                    0/1     CrashLoopBackOff   6          10m
etcd-2                                    1/1     Running            0          10m

perftest-m1:~$ ls -l /data/archive/etcd-snaps/
total 120
-rw------- 1 1001 root 118816 Mar 17 13:30 db

@perftest-m1:~$ helm get values etcd
USER-SUPPLIED VALUES:
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/instance
            operator: In
            values:
            - etcd
        topologyKey: kubernetes.io/hostname
      weight: 100
args: []
auth:
  client:
    caFilename: ""
    certFilename: cert.pem
    certKeyFilename: key.pem
    enableAuthentication: false
    existingSecret: ""
    secureTransport: false
    useAutoTLS: false
  peer:
    caFilename: ""
    certFilename: cert.pem
    certKeyFilename: key.pem
    enableAuthentication: false
    existingSecret: ""
    secureTransport: false
    useAutoTLS: false
  rbac:
    allowNoneAuthentication: true
    enabled: false
    existingSecret: ""
    rootPassword: ""
clusterDomain: cluster.local
command: []
commonAnnotations: {}
commonLabels: {}
containerPorts:
  client: 2379
  peer: 2380
containerSecurityContext:
  enabled: true
  runAsNonRoot: true
  runAsUser: 1001
customLivenessProbe: {}
customReadinessProbe: {}
customStartupProbe: {}
disasterRecovery:
  cronjob:
    historyLimit: 1
    podAnnotations: {}
    resources:
      limits: {}
      requests: {}
    schedule: '*/30 * * * *'
    snapshotHistoryLimit: 1
  enabled: false
  pvc:
    existingClaim: etcd-snapshot-volume
    size: 2Gi
    storageClassName: nfs
extraDeploy: []
extraEnvVars: []
extraEnvVarsCM: ""
extraEnvVarsSecret: ""
extraVolumeMounts: []
extraVolumes: []
hostAliases: []
image:
  debug: true
  pullPolicy: IfNotPresent
  pullSecrets: []
  registry: docker.io
  repository: bitnami/etcd
  tag: 3.4.15-debian-10-r14
initContainers: {}
initialClusterState: existing
kubeVersion: ""
lifecycleHooks: {}
livenessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 60
  periodSeconds: 30
  successThreshold: 1
  timeoutSeconds: 5
metrics:
  enabled: true
  podAnnotations:
    prometheus.io/port: "2379"
    prometheus.io/scrape: "true"
  podMonitor:
    additionalLabels: {}
    enabled: false
    interval: 30s
    scheme: http
    tlsConfig: {}
nodeAffinityPreset:
  key: ""
  type: ""
  values: []
nodeSelector: {}
pdb:
  create: false
  minAvailable: 1
persistence:
  accessModes:
  - ReadWriteOnce
  annotations: {}
  enabled: true
  selector: {}
  size: 16Gi
  storageClass: rook-ceph-block
podAffinityPreset: ""
podAnnotations:
  backup.velero.io/backup-volumes: data
podAntiAffinity: soft
podAntiAffinityPreset: soft
podLabels: {}
podManagementPolicy: Parallel
podSecurityContext:
  enabled: true
  fsGroup: 1001
priorityClassName: ""
readinessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
removeMemberOnContainerTermination: true
replicaCount: 3
resources:
  limits: {}
  requests: {}
service:
  annotations: {}
  clientPortNameOverride: ""
  externalIPs: []
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  nodePorts:
    clientPort: ""
    peerPort: ""
  peerPort: 2380
  peerPortNameOverride: ""
  port: 2379
  type: ClusterIP
sidecars: {}
startFromSnapshot:
  enabled: true
  existingClaim: etcd-snapshot-volume
  snapshotFilename: db
startupProbe:
  enabled: false
  failureThreshold: 60
  initialDelaySeconds: 0
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
tolerations: []
updateStrategy:
  type: RollingUpdate
volumePermissions:
  enabled: false
  image:
    pullPolicy: Always
    pullSecrets: []
    registry: docker.io
    repository: bitnami/bitnami-shell
    tag: "10"
  resources:
    limits: {}
    requests: {}

etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.93 [0m
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.93 [0m[1mWelcome to the Bitnami etcd container[0m
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.93 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-etcd[0m
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.93 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-etcd/issues[0m
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.93 [0m
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.94 [0m[38;5;2mINFO [0m ==> ** Starting etcd setup **
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.95 [0m[38;5;2mINFO [0m ==> Validating settings in ETCD_* env vars..
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.95 [0m[38;5;3mWARN [0m ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.95 [0m[38;5;2mINFO [0m ==> Initializing etcd
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.96 [0m[38;5;2mINFO [0m ==> Detected data from previous deployments
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:00.96 [0m[38;5;5mDEBUG[0m ==> Setting data directory permissions to 700 in a recursive way (required in etcd >=3.4.10)
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:11.02 [0m[38;5;5mDEBUG[0m ==> There are no enough active endpoints!
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:11.03 [0m[38;5;3mWARN [0m ==> Cluster not responding!
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:11.03 [0m[38;5;3mWARN [0m ==> Disaster recovery is disabled, the cluster will try to recover on it's own
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:11.03 [0m[38;5;2mINFO [0m ==> ** etcd setup finished! **
etcd-1 etcd 
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:14.99 [0m
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:14.99 [0m[1mWelcome to the Bitnami etcd container[0m
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:14.99 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-etcd[0m
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:14.99 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-etcd/issues[0m
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:14.99 [0m
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:15.00 [0m[38;5;2mINFO [0m ==> ** Starting etcd setup **
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:15.00 [0m[38;5;2mINFO [0m ==> Validating settings in ETCD_* env vars..
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:15.01 [0m[38;5;3mWARN [0m ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:15.01 [0m[38;5;2mINFO [0m ==> Initializing etcd
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:15.02 [0m[38;5;2mINFO [0m ==> Detected data from previous deployments
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:15.02 [0m[38;5;5mDEBUG[0m ==> Setting data directory permissions to 700 in a recursive way (required in etcd >=3.4.10)
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.07 [0m
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.07 [0m[1mWelcome to the Bitnami etcd container[0m
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.07 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/bitnami-docker-etcd[0m
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.07 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/bitnami-docker-etcd/issues[0m
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.07 [0m
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.07 [0m[38;5;2mINFO [0m ==> ** Starting etcd setup **
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.09 [0m[38;5;2mINFO [0m ==> Validating settings in ETCD_* env vars..
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.09 [0m[38;5;3mWARN [0m ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.09 [0m[38;5;2mINFO [0m ==> Initializing etcd
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.10 [0m[38;5;2mINFO [0m ==> Detected data from previous deployments
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:15.10 [0m[38;5;5mDEBUG[0m ==> Setting data directory permissions to 700 in a recursive way (required in etcd >=3.4.10)
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:16.11 [0m[38;5;5mDEBUG[0m ==> Loading env vars of existing cluster
etcd-1 etcd [38;5;6metcd [38;5;5m16:25:16.11 [0m[38;5;2mINFO [0m ==> ** Starting etcd **
etcd-1 etcd 2021-03-24 16:25:16.125713 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd-1.etcd-headless.default.svc.cluster.local:2379
etcd-1 etcd 2021-03-24 16:25:16.125769 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false
etcd-1 etcd 2021-03-24 16:25:16.125777 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
etcd-1 etcd 2021-03-24 16:25:16.125807 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-1.etcd-headless.default.svc.cluster.local:2380
etcd-1 etcd 2021-03-24 16:25:16.125817 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380,etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380
etcd-1 etcd 2021-03-24 16:25:16.125821 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
etcd-1 etcd 2021-03-24 16:25:16.125828 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
etcd-1 etcd 2021-03-24 16:25:16.125837 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
etcd-1 etcd 2021-03-24 16:25:16.125849 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
etcd-1 etcd 2021-03-24 16:25:16.125859 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=debug
etcd-1 etcd 2021-03-24 16:25:16.125871 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-1
etcd-1 etcd 2021-03-24 16:25:16.125879 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false
etcd-1 etcd 2021-03-24 16:25:16.125912 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE=
etcd-1 etcd 2021-03-24 16:25:16.125925 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.98.230.131
etcd-1 etcd 2021-03-24 16:25:16.125928 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_ADDR=10.98.230.131
etcd-1 etcd 2021-03-24 16:25:16.125932 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.98.230.131:2379
etcd-1 etcd 2021-03-24 16:25:16.125947 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes
etcd-1 etcd 2021-03-24 16:25:16.125953 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PROTO=tcp
etcd-1 etcd 2021-03-24 16:25:16.125957 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots
etcd-1 etcd 2021-03-24 16:25:16.125962 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin
etcd-1 etcd 2021-03-24 16:25:16.125969 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd
etcd-1 etcd 2021-03-24 16:25:16.125974 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
etcd-1 etcd 2021-03-24 16:25:16.125980 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.98.230.131
etcd-1 etcd 2021-03-24 16:25:16.125988 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PORT=2380
etcd-1 etcd 2021-03-24 16:25:16.125996 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD=
etcd-1 etcd 2021-03-24 16:25:16.125998 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP=tcp://10.98.230.131:2380
etcd-1 etcd 2021-03-24 16:25:16.126002 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=etcd-headless.default.svc.cluster.local
etcd-1 etcd 2021-03-24 16:25:16.126008 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no
etcd-1 etcd 2021-03-24 16:25:16.126011 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE=
etcd-1 etcd 2021-03-24 16:25:16.126018 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd
etcd-1 etcd 2021-03-24 16:25:16.126022 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=yes
etcd-1 etcd 2021-03-24 16:25:16.126028 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_CLIENT=2379
etcd-1 etcd 2021-03-24 16:25:16.126038 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot
etcd-1 etcd 2021-03-24 16:25:16.126046 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_PEER=2380
etcd-1 etcd 2021-03-24 16:25:16.126051 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd
etcd-1 etcd 2021-03-24 16:25:16.126057 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
etcd-1 etcd 2021-03-24 16:25:16.126064 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE=
etcd-1 etcd 2021-03-24 16:25:16.126068 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.98.230.131:2379
etcd-1 etcd 2021-03-24 16:25:16.126071 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
etcd-1 etcd 2021-03-24 16:25:16.126073 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs
etcd-1 etcd 2021-03-24 16:25:16.126076 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd
etcd-1 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-1 etcd 2021-03-24 16:25:16.126111 I | etcdmain: etcd Version: 3.4.15
etcd-1 etcd 2021-03-24 16:25:16.126113 I | etcdmain: Git SHA: aa7126864
etcd-1 etcd 2021-03-24 16:25:16.126116 I | etcdmain: Go Version: go1.12.17
etcd-1 etcd 2021-03-24 16:25:16.126118 I | etcdmain: Go OS/Arch: linux/amd64
etcd-1 etcd 2021-03-24 16:25:16.126121 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
etcd-1 etcd 2021-03-24 16:25:16.126177 W | etcdmain: found invalid file/dir new_member_envs under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
etcd-1 etcd 2021-03-24 16:25:16.126186 N | etcdmain: the server is already initialized as member before, starting as etcd member...
etcd-1 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-1 etcd 2021-03-24 16:25:16.126321 I | embed: name = etcd-1
etcd-1 etcd 2021-03-24 16:25:16.126329 I | embed: data dir = /bitnami/etcd/data
etcd-1 etcd 2021-03-24 16:25:16.126332 I | embed: member dir = /bitnami/etcd/data/member
etcd-1 etcd 2021-03-24 16:25:16.126335 I | embed: heartbeat = 100ms
etcd-1 etcd 2021-03-24 16:25:16.126337 I | embed: election = 1000ms
etcd-1 etcd 2021-03-24 16:25:16.126340 I | embed: snapshot count = 100000
etcd-1 etcd 2021-03-24 16:25:16.126349 I | embed: advertise client URLs = http://etcd-1.etcd-headless.default.svc.cluster.local:2379
etcd-1 etcd 2021-03-24 16:25:16.126352 I | embed: initial advertise peer URLs = http://etcd-1.etcd-headless.default.svc.cluster.local:2380
etcd-1 etcd 2021-03-24 16:25:16.126360 I | embed: initial cluster = 
etcd-1 etcd 2021-03-24 16:25:16.133321 I | etcdserver: restarting member 936ce633ac273d75 in cluster 9e98e654be1e22d7 at commit index 3
etcd-1 etcd raft2021/03/24 16:25:16 INFO: 936ce633ac273d75 switched to configuration voters=()
etcd-1 etcd raft2021/03/24 16:25:16 INFO: 936ce633ac273d75 became follower at term 13
etcd-1 etcd raft2021/03/24 16:25:16 INFO: newRaft 936ce633ac273d75 [peers: [], term: 13, commit: 3, applied: 0, lastindex: 3, lastterm: 1]
etcd-1 etcd 2021-03-24 16:25:16.138686 W | auth: simple token is not cryptographically signed
etcd-1 etcd 2021-03-24 16:25:16.141655 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
etcd-1 etcd raft2021/03/24 16:25:16 INFO: 936ce633ac273d75 switched to configuration voters=(5017444064630024030)
etcd-1 etcd 2021-03-24 16:25:16.142961 I | etcdserver/membership: added member 45a18acb10aa275e [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-1 etcd 2021-03-24 16:25:16.142979 I | rafthttp: starting peer 45a18acb10aa275e...
etcd-1 etcd 2021-03-24 16:25:16.143003 I | rafthttp: started HTTP pipelining with peer 45a18acb10aa275e
etcd-1 etcd 2021-03-24 16:25:16.143841 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-1 etcd 2021-03-24 16:25:16.144250 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-1 etcd 2021-03-24 16:25:16.144795 I | rafthttp: started peer 45a18acb10aa275e
etcd-1 etcd 2021-03-24 16:25:16.144824 I | rafthttp: added peer 45a18acb10aa275e
etcd-1 etcd 2021-03-24 16:25:16.144919 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-1 etcd 2021-03-24 16:25:16.144936 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-1 etcd raft2021/03/24 16:25:16 INFO: 936ce633ac273d75 switched to configuration voters=(5017444064630024030 10623118730666130805)
etcd-1 etcd 2021-03-24 16:25:16.145089 I | etcdserver/membership: added member 936ce633ac273d75 [http://etcd-1.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-1 etcd raft2021/03/24 16:25:16 INFO: 936ce633ac273d75 switched to configuration voters=(5017444064630024030 10623118730666130805 16085501043111492905)
etcd-1 etcd 2021-03-24 16:25:16.145197 I | etcdserver/membership: added member df3b2df95cd5fd29 [http://etcd-0.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-1 etcd 2021-03-24 16:25:16.145233 I | rafthttp: starting peer df3b2df95cd5fd29...
etcd-1 etcd 2021-03-24 16:25:16.145262 I | rafthttp: started HTTP pipelining with peer df3b2df95cd5fd29
etcd-1 etcd 2021-03-24 16:25:16.145469 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
etcd-1 etcd 2021-03-24 16:25:16.145669 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
etcd-1 etcd 2021-03-24 16:25:16.147693 I | embed: listening for peers on [::]:2380
etcd-1 etcd 2021-03-24 16:25:16.148101 I | rafthttp: started peer df3b2df95cd5fd29
etcd-1 etcd 2021-03-24 16:25:16.148128 I | rafthttp: added peer df3b2df95cd5fd29
etcd-1 etcd 2021-03-24 16:25:16.148318 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream Message reader)
etcd-1 etcd 2021-03-24 16:25:16.148372 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-1 etcd raft2021/03/24 16:25:17 INFO: 936ce633ac273d75 is starting a new election at term 13
etcd-1 etcd raft2021/03/24 16:25:17 INFO: 936ce633ac273d75 became candidate at term 14
etcd-1 etcd raft2021/03/24 16:25:17 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 14
etcd-1 etcd raft2021/03/24 16:25:17 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 14
etcd-1 etcd raft2021/03/24 16:25:17 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 14
etcd-1 etcd raft2021/03/24 16:25:18 INFO: 936ce633ac273d75 is starting a new election at term 14
etcd-1 etcd raft2021/03/24 16:25:18 INFO: 936ce633ac273d75 became candidate at term 15
etcd-1 etcd raft2021/03/24 16:25:18 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 15
etcd-1 etcd raft2021/03/24 16:25:18 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 15
etcd-1 etcd raft2021/03/24 16:25:18 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 15
etcd-1 etcd raft2021/03/24 16:25:20 INFO: 936ce633ac273d75 is starting a new election at term 15
etcd-1 etcd raft2021/03/24 16:25:20 INFO: 936ce633ac273d75 became candidate at term 16
etcd-1 etcd raft2021/03/24 16:25:20 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 16
etcd-1 etcd raft2021/03/24 16:25:20 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 16
etcd-1 etcd raft2021/03/24 16:25:20 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 16
etcd-1 etcd 2021-03-24 16:25:21.145028 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:21.145290 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:21.148417 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:21.148444 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd raft2021/03/24 16:25:22 INFO: 936ce633ac273d75 is starting a new election at term 16
etcd-1 etcd raft2021/03/24 16:25:22 INFO: 936ce633ac273d75 became candidate at term 17
etcd-1 etcd raft2021/03/24 16:25:22 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 17
etcd-1 etcd raft2021/03/24 16:25:22 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 17
etcd-1 etcd raft2021/03/24 16:25:22 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 17
etcd-1 etcd 2021-03-24 16:25:23.142425 E | etcdserver: publish error: etcdserver: request timed out
etcd-1 etcd raft2021/03/24 16:25:23 INFO: 936ce633ac273d75 is starting a new election at term 17
etcd-1 etcd raft2021/03/24 16:25:23 INFO: 936ce633ac273d75 became candidate at term 18
etcd-1 etcd raft2021/03/24 16:25:23 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 18
etcd-1 etcd raft2021/03/24 16:25:23 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 18
etcd-1 etcd raft2021/03/24 16:25:23 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 18
etcd-1 etcd raft2021/03/24 16:25:25 INFO: 936ce633ac273d75 is starting a new election at term 18
etcd-1 etcd raft2021/03/24 16:25:25 INFO: 936ce633ac273d75 became candidate at term 19
etcd-1 etcd raft2021/03/24 16:25:25 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 19
etcd-1 etcd raft2021/03/24 16:25:25 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 19
etcd-1 etcd raft2021/03/24 16:25:25 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 19
etcd-1 etcd 2021-03-24 16:25:26.145200 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:26.145725 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:26.148537 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:26.148585 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd raft2021/03/24 16:25:27 INFO: 936ce633ac273d75 is starting a new election at term 19
etcd-1 etcd raft2021/03/24 16:25:27 INFO: 936ce633ac273d75 became candidate at term 20
etcd-1 etcd raft2021/03/24 16:25:27 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 20
etcd-1 etcd raft2021/03/24 16:25:27 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 20
etcd-1 etcd raft2021/03/24 16:25:27 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 20
etcd-1 etcd raft2021/03/24 16:25:29 INFO: 936ce633ac273d75 is starting a new election at term 20
etcd-1 etcd raft2021/03/24 16:25:29 INFO: 936ce633ac273d75 became candidate at term 21
etcd-1 etcd raft2021/03/24 16:25:29 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 21
etcd-1 etcd raft2021/03/24 16:25:29 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 21
etcd-1 etcd raft2021/03/24 16:25:29 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 21
etcd-1 etcd 2021-03-24 16:25:30.142595 E | etcdserver: publish error: etcdserver: request timed out
etcd-1 etcd raft2021/03/24 16:25:30 INFO: 936ce633ac273d75 is starting a new election at term 21
etcd-1 etcd raft2021/03/24 16:25:30 INFO: 936ce633ac273d75 became candidate at term 22
etcd-1 etcd raft2021/03/24 16:25:30 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 22
etcd-1 etcd raft2021/03/24 16:25:30 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 22
etcd-1 etcd raft2021/03/24 16:25:30 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 22
etcd-1 etcd 2021-03-24 16:25:31.145379 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:31.145950 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:31.148731 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:31.148761 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd raft2021/03/24 16:25:31 INFO: 936ce633ac273d75 is starting a new election at term 22
etcd-1 etcd raft2021/03/24 16:25:31 INFO: 936ce633ac273d75 became candidate at term 23
etcd-1 etcd raft2021/03/24 16:25:31 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 23
etcd-1 etcd raft2021/03/24 16:25:31 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 23
etcd-1 etcd raft2021/03/24 16:25:31 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 23
etcd-1 etcd raft2021/03/24 16:25:32 INFO: 936ce633ac273d75 is starting a new election at term 23
etcd-1 etcd raft2021/03/24 16:25:32 INFO: 936ce633ac273d75 became candidate at term 24
etcd-1 etcd raft2021/03/24 16:25:32 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 24
etcd-1 etcd raft2021/03/24 16:25:32 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 24
etcd-1 etcd raft2021/03/24 16:25:32 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 24
etcd-1 etcd raft2021/03/24 16:25:34 INFO: 936ce633ac273d75 is starting a new election at term 24
etcd-1 etcd raft2021/03/24 16:25:34 INFO: 936ce633ac273d75 became candidate at term 25
etcd-1 etcd raft2021/03/24 16:25:34 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 25
etcd-1 etcd raft2021/03/24 16:25:34 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 25
etcd-1 etcd raft2021/03/24 16:25:34 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 25
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:35.10 [0m[38;5;5mDEBUG[0m ==> There are no enough active endpoints!
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:35.10 [0m[38;5;3mWARN [0m ==> Cluster not responding!
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:35.10 [0m[38;5;3mWARN [0m ==> Disaster recovery is disabled, the cluster will try to recover on it's own
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:35.11 [0m[38;5;2mINFO [0m ==> ** etcd setup finished! **
etcd-0 etcd 
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:35.20 [0m[38;5;5mDEBUG[0m ==> There are no enough active endpoints!
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:35.20 [0m[38;5;3mWARN [0m ==> Cluster not responding!
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:35.20 [0m[38;5;3mWARN [0m ==> Disaster recovery is disabled, the cluster will try to recover on it's own
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:35.20 [0m[38;5;2mINFO [0m ==> ** etcd setup finished! **
etcd-2 etcd 
etcd-1 etcd raft2021/03/24 16:25:35 INFO: 936ce633ac273d75 is starting a new election at term 25
etcd-1 etcd raft2021/03/24 16:25:35 INFO: 936ce633ac273d75 became candidate at term 26
etcd-1 etcd raft2021/03/24 16:25:35 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 26
etcd-1 etcd raft2021/03/24 16:25:35 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 26
etcd-1 etcd raft2021/03/24 16:25:35 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 26
etcd-1 etcd 2021-03-24 16:25:36.145530 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:36.146070 W | rafthttp: health check for peer 45a18acb10aa275e could not connect: dial tcp 10.244.8.24:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:36.148809 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd 2021-03-24 16:25:36.148834 W | rafthttp: health check for peer df3b2df95cd5fd29 could not connect: dial tcp 10.244.4.50:2380: connect: connection refused
etcd-1 etcd raft2021/03/24 16:25:36 INFO: 936ce633ac273d75 is starting a new election at term 26
etcd-1 etcd raft2021/03/24 16:25:36 INFO: 936ce633ac273d75 became candidate at term 27
etcd-1 etcd raft2021/03/24 16:25:36 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 27
etcd-1 etcd raft2021/03/24 16:25:36 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 27
etcd-1 etcd raft2021/03/24 16:25:36 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 27
etcd-1 etcd 2021-03-24 16:25:37.142860 E | etcdserver: publish error: etcdserver: request timed out
etcd-1 etcd raft2021/03/24 16:25:37 INFO: 936ce633ac273d75 is starting a new election at term 27
etcd-1 etcd raft2021/03/24 16:25:37 INFO: 936ce633ac273d75 became candidate at term 28
etcd-1 etcd raft2021/03/24 16:25:37 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 28
etcd-1 etcd raft2021/03/24 16:25:37 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 28
etcd-1 etcd raft2021/03/24 16:25:37 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 28
etcd-1 etcd raft2021/03/24 16:25:39 INFO: 936ce633ac273d75 is starting a new election at term 28
etcd-1 etcd raft2021/03/24 16:25:39 INFO: 936ce633ac273d75 became candidate at term 29
etcd-1 etcd raft2021/03/24 16:25:39 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 29
etcd-1 etcd raft2021/03/24 16:25:39 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 29
etcd-1 etcd raft2021/03/24 16:25:39 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 29
etcd-0 etcd [38;5;6metcd [38;5;5m16:25:40.17 [0m[38;5;2mINFO [0m ==> ** Starting etcd **
etcd-0 etcd 2021-03-24 16:25:40.187716 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd-0.etcd-headless.default.svc.cluster.local:2379
etcd-0 etcd 2021-03-24 16:25:40.187763 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false
etcd-0 etcd 2021-03-24 16:25:40.187770 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
etcd-0 etcd 2021-03-24 16:25:40.187791 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-0.etcd-headless.default.svc.cluster.local:2380
etcd-0 etcd 2021-03-24 16:25:40.187801 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
etcd-0 etcd 2021-03-24 16:25:40.187806 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
etcd-0 etcd 2021-03-24 16:25:40.187811 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
etcd-0 etcd 2021-03-24 16:25:40.187820 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
etcd-0 etcd 2021-03-24 16:25:40.187828 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
etcd-0 etcd 2021-03-24 16:25:40.187835 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=debug
etcd-0 etcd 2021-03-24 16:25:40.187843 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-0
etcd-0 etcd 2021-03-24 16:25:40.187847 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false
etcd-0 etcd 2021-03-24 16:25:40.187878 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE=
etcd-0 etcd 2021-03-24 16:25:40.187888 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.98.230.131
etcd-0 etcd 2021-03-24 16:25:40.187891 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_ADDR=10.98.230.131
etcd-0 etcd 2021-03-24 16:25:40.187898 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.98.230.131:2379
etcd-0 etcd 2021-03-24 16:25:40.187915 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes
etcd-0 etcd 2021-03-24 16:25:40.187920 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PROTO=tcp
etcd-0 etcd 2021-03-24 16:25:40.187924 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots
etcd-0 etcd 2021-03-24 16:25:40.187929 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin
etcd-0 etcd 2021-03-24 16:25:40.187933 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd
etcd-0 etcd 2021-03-24 16:25:40.187940 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
etcd-0 etcd 2021-03-24 16:25:40.187943 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.98.230.131
etcd-0 etcd 2021-03-24 16:25:40.187951 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PORT=2380
etcd-0 etcd 2021-03-24 16:25:40.187959 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD=
etcd-0 etcd 2021-03-24 16:25:40.187961 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP=tcp://10.98.230.131:2380
etcd-0 etcd 2021-03-24 16:25:40.187965 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=etcd-headless.default.svc.cluster.local
etcd-0 etcd 2021-03-24 16:25:40.187970 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no
etcd-0 etcd 2021-03-24 16:25:40.187973 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE=
etcd-0 etcd 2021-03-24 16:25:40.187979 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd
etcd-0 etcd 2021-03-24 16:25:40.187982 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=yes
etcd-0 etcd 2021-03-24 16:25:40.187989 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_CLIENT=2379
etcd-0 etcd 2021-03-24 16:25:40.187993 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot
etcd-0 etcd 2021-03-24 16:25:40.188002 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_PEER=2380
etcd-0 etcd 2021-03-24 16:25:40.188005 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd
etcd-0 etcd 2021-03-24 16:25:40.188010 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
etcd-0 etcd 2021-03-24 16:25:40.188014 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE=
etcd-0 etcd 2021-03-24 16:25:40.188017 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.98.230.131:2379
etcd-0 etcd 2021-03-24 16:25:40.188019 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
etcd-0 etcd 2021-03-24 16:25:40.188022 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs
etcd-0 etcd 2021-03-24 16:25:40.188025 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd
etcd-0 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-0 etcd 2021-03-24 16:25:40.188056 I | etcdmain: etcd Version: 3.4.15
etcd-0 etcd 2021-03-24 16:25:40.188059 I | etcdmain: Git SHA: aa7126864
etcd-0 etcd 2021-03-24 16:25:40.188061 I | etcdmain: Go Version: go1.12.17
etcd-0 etcd 2021-03-24 16:25:40.188063 I | etcdmain: Go OS/Arch: linux/amd64
etcd-0 etcd 2021-03-24 16:25:40.188066 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
etcd-0 etcd 2021-03-24 16:25:40.188796 W | etcdmain: found invalid file/dir member_id under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
etcd-0 etcd 2021-03-24 16:25:40.188802 N | etcdmain: the server is already initialized as member before, starting as etcd member...
etcd-0 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-0 etcd 2021-03-24 16:25:40.189029 I | embed: name = etcd-0
etcd-0 etcd 2021-03-24 16:25:40.189037 I | embed: data dir = /bitnami/etcd/data
etcd-0 etcd 2021-03-24 16:25:40.189040 I | embed: member dir = /bitnami/etcd/data/member
etcd-0 etcd 2021-03-24 16:25:40.189042 I | embed: heartbeat = 100ms
etcd-0 etcd 2021-03-24 16:25:40.189044 I | embed: election = 1000ms
etcd-0 etcd 2021-03-24 16:25:40.189046 I | embed: snapshot count = 100000
etcd-0 etcd 2021-03-24 16:25:40.189052 I | embed: advertise client URLs = http://etcd-0.etcd-headless.default.svc.cluster.local:2379
etcd-0 etcd 2021-03-24 16:25:40.189055 I | embed: initial advertise peer URLs = http://etcd-0.etcd-headless.default.svc.cluster.local:2380
etcd-0 etcd 2021-03-24 16:25:40.189058 I | embed: initial cluster = 
etcd-0 etcd 2021-03-24 16:25:40.196114 I | etcdserver: restarting member df3b2df95cd5fd29 in cluster 9e98e654be1e22d7 at commit index 33
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 switched to configuration voters=()
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 became follower at term 13
etcd-0 etcd raft2021/03/24 16:25:40 INFO: newRaft df3b2df95cd5fd29 [peers: [], term: 13, commit: 33, applied: 0, lastindex: 34, lastterm: 13]
etcd-0 etcd 2021-03-24 16:25:40.197995 W | auth: simple token is not cryptographically signed
etcd-0 etcd 2021-03-24 16:25:40.203744 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030)
etcd-0 etcd 2021-03-24 16:25:40.204267 I | etcdserver/membership: added member 45a18acb10aa275e [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:25:40.204287 I | rafthttp: starting peer 45a18acb10aa275e...
etcd-0 etcd 2021-03-24 16:25:40.204308 I | rafthttp: started HTTP pipelining with peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:25:40.204814 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:25:40.205088 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:25:40.206003 I | rafthttp: started peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:25:40.206030 I | rafthttp: added peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:25:40.206075 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-0 etcd 2021-03-24 16:25:40.206110 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805)
etcd-0 etcd 2021-03-24 16:25:40.206851 I | etcdserver/membership: added member 936ce633ac273d75 [http://etcd-1.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:25:40.206864 I | rafthttp: starting peer 936ce633ac273d75...
etcd-0 etcd 2021-03-24 16:25:40.206922 I | rafthttp: started HTTP pipelining with peer 936ce633ac273d75
etcd-0 etcd 2021-03-24 16:25:40.206946 I | embed: listening for peers on [::]:2380
etcd-0 etcd 2021-03-24 16:25:40.208030 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-0 etcd 2021-03-24 16:25:40.208727 I | rafthttp: started peer 936ce633ac273d75
etcd-0 etcd 2021-03-24 16:25:40.208818 I | rafthttp: added peer 936ce633ac273d75
etcd-0 etcd 2021-03-24 16:25:40.208886 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805 16085501043111492905)
etcd-0 etcd 2021-03-24 16:25:40.208992 I | etcdserver/membership: added member df3b2df95cd5fd29 [http://etcd-0.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:25:40.209031 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-0 etcd 2021-03-24 16:25:40.209216 N | etcdserver/membership: set the initial cluster version to 3.0
etcd-0 etcd 2021-03-24 16:25:40.209298 I | etcdserver/api: enabled capabilities for version 3.0
etcd-0 etcd 2021-03-24 16:25:40.209350 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream Message reader)
etcd-0 etcd 2021-03-24 16:25:40.209387 N | etcdserver/membership: updated the cluster version from 3.0 to 3.4
etcd-0 etcd 2021-03-24 16:25:40.209504 I | etcdserver/api: enabled capabilities for version 3.4
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805)
etcd-0 etcd 2021-03-24 16:25:40.210157 I | etcdserver/membership: removed member df3b2df95cd5fd29 from cluster 9e98e654be1e22d7
etcd-1 etcd 2021-03-24 16:25:40.213674 I | rafthttp: peer df3b2df95cd5fd29 became active
etcd-1 etcd 2021-03-24 16:25:40.213707 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream MsgApp v2 writer)
etcd-1 etcd 2021-03-24 16:25:40.213873 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream Message writer)
etcd-0 etcd 2021-03-24 16:25:40.212900 I | rafthttp: peer 936ce633ac273d75 became active
etcd-0 etcd 2021-03-24 16:25:40.212922 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-0 etcd 2021-03-24 16:25:40.213266 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream Message reader)
etcd-0 etcd 2021-03-24 16:25:40.250077 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream Message writer)
etcd-0 etcd 2021-03-24 16:25:40.250171 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 writer)
etcd-1 etcd 2021-03-24 16:25:40.250771 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream Message reader)
etcd-1 etcd 2021-03-24 16:25:40.250838 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-2 etcd [38;5;6metcd [38;5;5m16:25:40.28 [0m[38;5;2mINFO [0m ==> ** Starting etcd **
etcd-2 etcd 2021-03-24 16:25:40.299206 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd-2.etcd-headless.default.svc.cluster.local:2379
etcd-2 etcd 2021-03-24 16:25:40.299281 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false
etcd-2 etcd 2021-03-24 16:25:40.299289 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
etcd-2 etcd 2021-03-24 16:25:40.299319 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
etcd-2 etcd 2021-03-24 16:25:40.299332 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380
etcd-2 etcd 2021-03-24 16:25:40.299339 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
etcd-2 etcd 2021-03-24 16:25:40.299342 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
etcd-2 etcd 2021-03-24 16:25:40.299349 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
etcd-2 etcd 2021-03-24 16:25:40.299360 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
etcd-2 etcd 2021-03-24 16:25:40.299366 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=debug
etcd-2 etcd 2021-03-24 16:25:40.299375 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-2
etcd-2 etcd 2021-03-24 16:25:40.299382 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false
etcd-2 etcd 2021-03-24 16:25:40.299425 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE=
etcd-2 etcd 2021-03-24 16:25:40.299436 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.98.230.131
etcd-2 etcd 2021-03-24 16:25:40.299438 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_ADDR=10.98.230.131
etcd-2 etcd 2021-03-24 16:25:40.299441 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.98.230.131:2379
etcd-2 etcd 2021-03-24 16:25:40.299455 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes
etcd-2 etcd 2021-03-24 16:25:40.299457 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PROTO=tcp
etcd-2 etcd 2021-03-24 16:25:40.299461 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots
etcd-2 etcd 2021-03-24 16:25:40.299465 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin
etcd-2 etcd 2021-03-24 16:25:40.299469 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd
etcd-2 etcd 2021-03-24 16:25:40.299474 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
etcd-2 etcd 2021-03-24 16:25:40.299476 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.98.230.131
etcd-2 etcd 2021-03-24 16:25:40.299492 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PORT=2380
etcd-2 etcd 2021-03-24 16:25:40.299498 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD=
etcd-2 etcd 2021-03-24 16:25:40.299500 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP=tcp://10.98.230.131:2380
etcd-2 etcd 2021-03-24 16:25:40.299503 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=etcd-headless.default.svc.cluster.local
etcd-2 etcd 2021-03-24 16:25:40.299506 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no
etcd-2 etcd 2021-03-24 16:25:40.299510 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE=
etcd-2 etcd 2021-03-24 16:25:40.299516 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd
etcd-2 etcd 2021-03-24 16:25:40.299518 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=yes
etcd-2 etcd 2021-03-24 16:25:40.299524 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_CLIENT=2379
etcd-2 etcd 2021-03-24 16:25:40.299528 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot
etcd-2 etcd 2021-03-24 16:25:40.299537 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_PEER=2380
etcd-2 etcd 2021-03-24 16:25:40.299540 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd
etcd-2 etcd 2021-03-24 16:25:40.299548 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
etcd-2 etcd 2021-03-24 16:25:40.299552 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE=
etcd-2 etcd 2021-03-24 16:25:40.299554 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.98.230.131:2379
etcd-2 etcd 2021-03-24 16:25:40.299564 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
etcd-2 etcd 2021-03-24 16:25:40.299567 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs
etcd-2 etcd 2021-03-24 16:25:40.299569 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd
etcd-2 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-2 etcd 2021-03-24 16:25:40.299604 I | etcdmain: etcd Version: 3.4.15
etcd-2 etcd 2021-03-24 16:25:40.299609 I | etcdmain: Git SHA: aa7126864
etcd-2 etcd 2021-03-24 16:25:40.299611 I | etcdmain: Go Version: go1.12.17
etcd-2 etcd 2021-03-24 16:25:40.299613 I | etcdmain: Go OS/Arch: linux/amd64
etcd-2 etcd 2021-03-24 16:25:40.299616 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
etcd-2 etcd 2021-03-24 16:25:40.299673 W | etcdmain: found invalid file/dir member_id under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
etcd-2 etcd 2021-03-24 16:25:40.299678 N | etcdmain: the server is already initialized as member before, starting as etcd member...
etcd-2 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-2 etcd 2021-03-24 16:25:40.299825 I | embed: name = etcd-2
etcd-2 etcd 2021-03-24 16:25:40.299829 I | embed: data dir = /bitnami/etcd/data
etcd-2 etcd 2021-03-24 16:25:40.299831 I | embed: member dir = /bitnami/etcd/data/member
etcd-2 etcd 2021-03-24 16:25:40.299833 I | embed: heartbeat = 100ms
etcd-2 etcd 2021-03-24 16:25:40.299835 I | embed: election = 1000ms
etcd-2 etcd 2021-03-24 16:25:40.299837 I | embed: snapshot count = 100000
etcd-2 etcd 2021-03-24 16:25:40.299844 I | embed: advertise client URLs = http://etcd-2.etcd-headless.default.svc.cluster.local:2379
etcd-2 etcd 2021-03-24 16:25:40.299847 I | embed: initial advertise peer URLs = http://etcd-2.etcd-headless.default.svc.cluster.local:2380
etcd-2 etcd 2021-03-24 16:25:40.299850 I | embed: initial cluster = 
etcd-2 etcd 2021-03-24 16:25:40.309557 I | etcdserver: restarting member 45a18acb10aa275e in cluster 9e98e654be1e22d7 at commit index 40
etcd-2 etcd raft2021/03/24 16:25:40 INFO: 45a18acb10aa275e switched to configuration voters=()
etcd-2 etcd raft2021/03/24 16:25:40 INFO: 45a18acb10aa275e became follower at term 16
etcd-2 etcd raft2021/03/24 16:25:40 INFO: newRaft 45a18acb10aa275e [peers: [], term: 16, commit: 40, applied: 0, lastindex: 40, lastterm: 16]
etcd-2 etcd 2021-03-24 16:25:40.311938 W | auth: simple token is not cryptographically signed
etcd-2 etcd 2021-03-24 16:25:40.316893 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
etcd-2 etcd raft2021/03/24 16:25:40 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030)
etcd-2 etcd 2021-03-24 16:25:40.317746 I | etcdserver/membership: added member 45a18acb10aa275e [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-2 etcd raft2021/03/24 16:25:40 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030 10623118730666130805)
etcd-2 etcd 2021-03-24 16:25:40.317827 I | etcdserver/membership: added member 936ce633ac273d75 [http://etcd-1.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:25:40.317847 I | rafthttp: starting peer 936ce633ac273d75...
etcd-2 etcd 2021-03-24 16:25:40.317872 I | rafthttp: started HTTP pipelining with peer 936ce633ac273d75
etcd-2 etcd 2021-03-24 16:25:40.318591 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-2 etcd 2021-03-24 16:25:40.319132 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-2 etcd 2021-03-24 16:25:40.319254 I | embed: listening for peers on [::]:2380
etcd-2 etcd 2021-03-24 16:25:40.320836 I | rafthttp: started peer 936ce633ac273d75
etcd-2 etcd 2021-03-24 16:25:40.320856 I | rafthttp: added peer 936ce633ac273d75
etcd-2 etcd raft2021/03/24 16:25:40 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030 10623118730666130805 16085501043111492905)
etcd-2 etcd 2021-03-24 16:25:40.320938 I | etcdserver/membership: added member df3b2df95cd5fd29 [http://etcd-0.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:25:40.320967 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:25:40.321324 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream Message reader)
etcd-2 etcd 2021-03-24 16:25:40.321404 I | rafthttp: starting peer df3b2df95cd5fd29...
etcd-2 etcd 2021-03-24 16:25:40.321417 I | rafthttp: started HTTP pipelining with peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:25:40.327129 I | rafthttp: started peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:25:40.327168 I | rafthttp: added peer df3b2df95cd5fd29
etcd-1 etcd 2021-03-24 16:25:40.329779 I | rafthttp: peer 45a18acb10aa275e became active
etcd-1 etcd 2021-03-24 16:25:40.329813 I | rafthttp: established a TCP streaming connection with peer 45a18acb10aa275e (stream Message writer)
etcd-1 etcd 2021-03-24 16:25:40.329880 I | rafthttp: established a TCP streaming connection with peer 45a18acb10aa275e (stream MsgApp v2 writer)
etcd-2 etcd 2021-03-24 16:25:40.327380 N | etcdserver/membership: set the initial cluster version to 3.0
etcd-2 etcd 2021-03-24 16:25:40.327477 I | etcdserver/api: enabled capabilities for version 3.0
etcd-2 etcd 2021-03-24 16:25:40.327851 N | etcdserver/membership: updated the cluster version from 3.0 to 3.4
etcd-2 etcd 2021-03-24 16:25:40.327886 I | etcdserver/api: enabled capabilities for version 3.4
etcd-2 etcd 2021-03-24 16:25:40.327934 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
etcd-2 etcd 2021-03-24 16:25:40.327942 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
etcd-2 etcd 2021-03-24 16:25:40.327945 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:25:40.328085 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream Message reader)
etcd-2 etcd 2021-03-24 16:25:40.328187 E | rafthttp: failed to find member df3b2df95cd5fd29 in cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:25:40.328248 E | rafthttp: failed to find member df3b2df95cd5fd29 in cluster 9e98e654be1e22d7
etcd-2 etcd raft2021/03/24 16:25:40 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030 10623118730666130805)
etcd-2 etcd 2021-03-24 16:25:40.328580 I | etcdserver/membership: removed member df3b2df95cd5fd29 from cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:25:40.328587 I | rafthttp: stopping peer df3b2df95cd5fd29...
etcd-2 etcd 2021-03-24 16:25:40.328641 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (writer)
etcd-2 etcd 2021-03-24 16:25:40.328646 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (writer)
etcd-2 etcd 2021-03-24 16:25:40.328668 I | rafthttp: stopped HTTP pipelining with peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:25:40.328683 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:25:40.328747 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (stream Message reader)
etcd-2 etcd 2021-03-24 16:25:40.328756 I | rafthttp: stopped peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:25:40.328761 I | rafthttp: removed peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:25:40.328872 I | rafthttp: peer 936ce633ac273d75 became active
etcd-2 etcd 2021-03-24 16:25:40.328891 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:25:40.328970 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream Message reader)
etcd-2 etcd raft2021/03/24 16:25:40 INFO: 45a18acb10aa275e switched to configuration voters=(5017444064630024030)
etcd-2 etcd 2021-03-24 16:25:40.329116 I | etcdserver/membership: removed member 936ce633ac273d75 from cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:25:40.329121 I | rafthttp: stopping peer 936ce633ac273d75...
etcd-2 etcd 2021-03-24 16:25:40.329127 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
etcd-2 etcd 2021-03-24 16:25:40.329132 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
etcd-2 etcd 2021-03-24 16:25:40.329154 I | rafthttp: stopped HTTP pipelining with peer 936ce633ac273d75
etcd-2 etcd 2021-03-24 16:25:40.329260 W | rafthttp: lost the TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:25:40.329273 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:25:40.329297 W | rafthttp: lost the TCP streaming connection with peer 936ce633ac273d75 (stream Message reader)
etcd-2 etcd 2021-03-24 16:25:40.329340 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream Message reader)
etcd-2 etcd 2021-03-24 16:25:40.329350 I | rafthttp: stopped peer 936ce633ac273d75
etcd-2 etcd 2021-03-24 16:25:40.329355 I | rafthttp: removed peer 936ce633ac273d75
etcd-1 etcd raft2021/03/24 16:25:40 INFO: 936ce633ac273d75 is starting a new election at term 29
etcd-1 etcd raft2021/03/24 16:25:40 INFO: 936ce633ac273d75 became candidate at term 30
etcd-1 etcd raft2021/03/24 16:25:40 INFO: 936ce633ac273d75 received MsgVoteResp from 936ce633ac273d75 at term 30
etcd-1 etcd raft2021/03/24 16:25:40 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to 45a18acb10aa275e at term 30
etcd-1 etcd raft2021/03/24 16:25:40 INFO: 936ce633ac273d75 [logterm: 1, index: 3] sent MsgVote request to df3b2df95cd5fd29 at term 30
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 [term: 13] received a MsgVote message with higher term from 936ce633ac273d75 [term: 30]
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 became follower at term 30
etcd-0 etcd raft2021/03/24 16:25:40 INFO: df3b2df95cd5fd29 [logterm: 13, index: 34, vote: 0] rejected MsgVote from 936ce633ac273d75 [logterm: 1, index: 3] at term 30
etcd-1 etcd raft2021/03/24 16:25:40 INFO: 936ce633ac273d75 received MsgVoteResp rejection from df3b2df95cd5fd29 at term 30
etcd-1 etcd raft2021/03/24 16:25:40 INFO: 936ce633ac273d75 has received 1 MsgVoteResp votes and 1 vote rejections
etcd-2 etcd 2021-03-24 16:25:40.346807 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-1 etcd 2021-03-24 16:25:40.347999 E | rafthttp: failed to dial 45a18acb10aa275e on stream Message (the member has been permanently removed from the cluster)
etcd-1 etcd 2021-03-24 16:25:40.348019 I | rafthttp: peer 45a18acb10aa275e became inactive (message send to peer failed)
etcd-1 etcd 2021-03-24 16:25:40.348023 E | etcdserver: the member has been permanently removed from the cluster
etcd-1 etcd 2021-03-24 16:25:40.348044 I | etcdserver: the data-dir used by this member must be removed.
etcd-1 etcd 2021-03-24 16:25:40.348085 E | etcdserver: publish error: etcdserver: request cancelled
etcd-1 etcd 2021-03-24 16:25:40.348101 E | etcdserver: publish error: etcdserver: request cancelled
etcd-1 etcd 2021-03-24 16:25:40.348109 E | etcdserver: publish error: etcdserver: request cancelled
etcd-1 etcd 2021-03-24 16:25:40.348114 I | etcdserver: aborting publish because server is stopped
etcd-1 etcd 2021-03-24 16:25:40.348134 I | rafthttp: stopping peer 45a18acb10aa275e...
etcd-1 etcd 2021-03-24 16:25:40.348728 I | rafthttp: closed the TCP streaming connection with peer 45a18acb10aa275e (stream MsgApp v2 writer)
etcd-1 etcd 2021-03-24 16:25:40.348738 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-2 etcd 2021-03-24 16:25:40.347020 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-0 etcd 2021-03-24 16:25:40.350260 W | rafthttp: lost the TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-1 etcd 2021-03-24 16:25:40.350020 I | rafthttp: closed the TCP streaming connection with peer 45a18acb10aa275e (stream Message writer)
etcd-1 etcd 2021-03-24 16:25:40.350040 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-1 etcd 2021-03-24 16:25:40.350303 I | rafthttp: stopped HTTP pipelining with peer 45a18acb10aa275e
etcd-1 etcd 2021-03-24 16:25:40.350334 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-1 etcd 2021-03-24 16:25:40.350345 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-1 etcd 2021-03-24 16:25:40.350356 I | rafthttp: stopped peer 45a18acb10aa275e
etcd-1 etcd 2021-03-24 16:25:40.350369 I | rafthttp: stopping peer df3b2df95cd5fd29...
etcd-1 etcd 2021-03-24 16:25:40.350868 I | rafthttp: closed the TCP streaming connection with peer df3b2df95cd5fd29 (stream MsgApp v2 writer)
etcd-1 etcd 2021-03-24 16:25:40.350877 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (writer)
etcd-0 etcd 2021-03-24 16:25:40.350566 W | rafthttp: lost the TCP streaming connection with peer 936ce633ac273d75 (stream Message reader)
etcd-0 etcd 2021-03-24 16:25:40.351844 E | rafthttp: failed to dial 936ce633ac273d75 on stream MsgApp v2 (peer 936ce633ac273d75 failed to find local node df3b2df95cd5fd29)
etcd-0 etcd 2021-03-24 16:25:40.351862 I | rafthttp: peer 936ce633ac273d75 became inactive (message send to peer failed)
etcd-1 etcd 2021-03-24 16:25:40.351575 I | rafthttp: closed the TCP streaming connection with peer df3b2df95cd5fd29 (stream Message writer)
etcd-1 etcd 2021-03-24 16:25:40.351955 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (writer)
etcd-1 etcd 2021-03-24 16:25:40.351977 I | rafthttp: stopped HTTP pipelining with peer df3b2df95cd5fd29
etcd-1 etcd 2021-03-24 16:25:40.352069 W | rafthttp: lost the TCP streaming connection with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-1 etcd 2021-03-24 16:25:40.352079 E | rafthttp: failed to read df3b2df95cd5fd29 on stream MsgApp v2 (context canceled)
etcd-1 etcd 2021-03-24 16:25:40.352083 I | rafthttp: peer df3b2df95cd5fd29 became inactive (message send to peer failed)
etcd-1 etcd 2021-03-24 16:25:40.352098 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-1 etcd 2021-03-24 16:25:40.352143 W | rafthttp: lost the TCP streaming connection with peer df3b2df95cd5fd29 (stream Message reader)
etcd-1 etcd 2021-03-24 16:25:40.352154 I | rafthttp: stopped streaming with peer df3b2df95cd5fd29 (stream Message reader)
etcd-1 etcd 2021-03-24 16:25:40.352159 I | rafthttp: stopped peer df3b2df95cd5fd29
etcd-1 etcd 2021-03-24 16:25:40.352247 E | rafthttp: failed to find member df3b2df95cd5fd29 in cluster 9e98e654be1e22d7
etcd-1 etcd 2021-03-24 16:25:40.352279 E | rafthttp: failed to find member df3b2df95cd5fd29 in cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:25:40.409199 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-0 etcd 2021-03-24 16:25:40.409757 E | etcdserver: the member has been permanently removed from the cluster
etcd-0 etcd 2021-03-24 16:25:40.409778 I | etcdserver: the data-dir used by this member must be removed.
etcd-0 etcd 2021-03-24 16:25:40.409815 I | etcdserver: aborting publish because server is stopped
etcd-0 etcd 2021-03-24 16:25:40.409841 I | rafthttp: stopping peer 45a18acb10aa275e...
etcd-0 etcd 2021-03-24 16:25:40.409854 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:25:40.409865 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:25:40.409895 I | rafthttp: stopped HTTP pipelining with peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:25:40.409918 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-0 etcd 2021-03-24 16:25:40.409928 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-0 etcd 2021-03-24 16:25:40.409933 I | rafthttp: stopped peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:25:40.409940 I | rafthttp: stopping peer 936ce633ac273d75...
etcd-0 etcd 2021-03-24 16:25:40.410146 I | rafthttp: closed the TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 writer)
etcd-0 etcd 2021-03-24 16:25:40.410156 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
etcd-0 etcd 2021-03-24 16:25:40.410582 I | rafthttp: closed the TCP streaming connection with peer 936ce633ac273d75 (stream Message writer)
etcd-0 etcd 2021-03-24 16:25:40.410590 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (writer)
etcd-0 etcd 2021-03-24 16:25:40.410751 I | rafthttp: stopped HTTP pipelining with peer 936ce633ac273d75
etcd-0 etcd 2021-03-24 16:25:40.410764 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-0 etcd 2021-03-24 16:25:40.410778 I | rafthttp: stopped streaming with peer 936ce633ac273d75 (stream Message reader)
etcd-0 etcd 2021-03-24 16:25:40.410782 I | rafthttp: stopped peer 936ce633ac273d75
etcd-2 etcd raft2021/03/24 16:25:41 INFO: 45a18acb10aa275e is starting a new election at term 16
etcd-2 etcd raft2021/03/24 16:25:41 INFO: 45a18acb10aa275e became candidate at term 17
etcd-2 etcd raft2021/03/24 16:25:41 INFO: 45a18acb10aa275e received MsgVoteResp from 45a18acb10aa275e at term 17
etcd-2 etcd raft2021/03/24 16:25:41 INFO: 45a18acb10aa275e became leader at term 17
etcd-2 etcd raft2021/03/24 16:25:41 INFO: raft.node: 45a18acb10aa275e elected leader 45a18acb10aa275e at term 17
etcd-2 etcd 2021-03-24 16:25:41.910461 I | etcdserver: published {Name:etcd-2 ClientURLs:[http://etcd-2.etcd-headless.default.svc.cluster.local:2379]} to cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:25:41.910548 I | embed: ready to serve client requests
etcd-2 etcd 2021-03-24 16:25:41.911245 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
etcd-2 etcd 2021-03-24 16:25:50.976183 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:25:50.975872955 +0000 UTC m=+10.684161520, time spent = 241.723Âµs, remote = 10.244.4.50:53346, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:25:56.661785 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:25:56.661617273 +0000 UTC m=+16.369905838, time spent = 137.868Âµs, remote = 10.244.6.27:37016, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:26:01.101678 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:01.101900 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:01.103371 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:01.103487 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:01.775054 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:26:01.775256 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:26:01.776438 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:26:22.410726 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:26:22.410588245 +0000 UTC m=+42.118876810, time spent = 107.03Âµs, remote = 10.244.8.24:38196, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:26:26.857786 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:26:26.857634572 +0000 UTC m=+46.565923137, time spent = 118.723Âµs, remote = 10.244.6.27:37100, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:26:31.409319 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:26:31.409083912 +0000 UTC m=+51.117372477, time spent = 197.25Âµs, remote = 10.244.4.50:53476, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:26:32.395157 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:26:32.395020913 +0000 UTC m=+52.103309478, time spent = 103.824Âµs, remote = 10.244.8.24:38222, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:26:32.842102 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:26:32.84197434 +0000 UTC m=+52.550262905, time spent = 100.618Âµs, remote = 10.244.8.24:38224, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:26:36.515944 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:36.516037 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:36.517884 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:36.518090 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:26:36.985340 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:26:36.985608 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:26:36.987723 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:26:36.988040 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:26:42.399315 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:26:42.39913182 +0000 UTC m=+62.107420405, time spent = 144.611Âµs, remote = 10.244.8.24:38252, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:26:52.399151 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:26:52.398984636 +0000 UTC m=+72.107273201, time spent = 131.356Âµs, remote = 10.244.8.24:38278, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:02.388981 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:02.388815382 +0000 UTC m=+82.097103947, time spent = 132.338Âµs, remote = 10.244.8.24:38306, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:02.842801 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:02.842619653 +0000 UTC m=+82.550908208, time spent = 147.046Âµs, remote = 10.244.8.24:38308, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:12.403179 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:12.403026835 +0000 UTC m=+92.111315390, time spent = 117.47Âµs, remote = 10.244.8.24:38334, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:14.665200 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:14.664969517 +0000 UTC m=+94.373258082, time spent = 199.083Âµs, remote = 10.244.6.27:37288, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:17.385846 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:17.385697227 +0000 UTC m=+97.093985792, time spent = 117.5Âµs, remote = 10.244.4.50:53612, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:19.778428 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:27:19.778532 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:27:22.394994 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:22.394781959 +0000 UTC m=+102.103070524, time spent = 176.892Âµs, remote = 10.244.8.24:38362, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:22.496639 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:27:22.496973 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:27:32.389936 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:32.389792921 +0000 UTC m=+112.098081476, time spent = 112.721Âµs, remote = 10.244.8.24:38388, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:32.840225 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:32.840070139 +0000 UTC m=+112.548358704, time spent = 126.227Âµs, remote = 10.244.8.24:38390, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:42.396599 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:42.396468793 +0000 UTC m=+122.104757358, time spent = 100.718Âµs, remote = 10.244.8.24:38418, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:27:52.405657 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:27:52.405520082 +0000 UTC m=+132.113808637, time spent = 107.291Âµs, remote = 10.244.8.24:38444, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:02.393450 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:02.393310603 +0000 UTC m=+142.101599168, time spent = 110.257Âµs, remote = 10.244.8.24:38470, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:02.857897 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:02.857767802 +0000 UTC m=+142.566056357, time spent = 102.282Âµs, remote = 10.244.8.24:38472, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:12.406569 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:12.406406661 +0000 UTC m=+152.114695226, time spent = 118.962Âµs, remote = 10.244.8.24:38500, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:22.389239 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:22.389103875 +0000 UTC m=+162.097392430, time spent = 104.996Âµs, remote = 10.244.8.24:38528, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:23.625119 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:23.624980603 +0000 UTC m=+163.333269168, time spent = 110.597Âµs, remote = 10.244.6.27:37476, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:24.398479 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:24.398307308 +0000 UTC m=+164.106595863, time spent = 130.294Âµs, remote = 10.244.4.50:53778, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:29.524464 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:28:29.524559 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:28:29.526118 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:28:29.526148 W | rafthttp: rejected the stream from peer df3b2df95cd5fd29 since it was removed
etcd-2 etcd 2021-03-24 16:28:32.394206 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:32.394054552 +0000 UTC m=+172.102343137, time spent = 115.947Âµs, remote = 10.244.8.24:38554, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:32.834829 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:32.834690017 +0000 UTC m=+172.542978592, time spent = 112.281Âµs, remote = 10.244.8.24:38556, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:33.751046 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:28:33.751085 W | rafthttp: rejected the stream from peer 936ce633ac273d75 since it was removed
etcd-2 etcd 2021-03-24 16:28:42.395684 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:42.395542992 +0000 UTC m=+182.103831557, time spent = 111.378Âµs, remote = 10.244.8.24:38584, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:28:52.394384 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:52.394243416 +0000 UTC m=+192.102531981, time spent = 110.367Âµs, remote = 10.244.8.24:38610, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:02.400373 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:02.400188184 +0000 UTC m=+202.108476749, time spent = 127.248Âµs, remote = 10.244.8.24:38636, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:02.903963 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:02.903835084 +0000 UTC m=+202.612123639, time spent = 99.466Âµs, remote = 10.244.8.24:38638, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:12.407796 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:12.407602174 +0000 UTC m=+212.115890729, time spent = 160.32Âµs, remote = 10.244.8.24:38664, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:22.392678 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:22.39252995 +0000 UTC m=+222.100818505, time spent = 115.235Âµs, remote = 10.244.8.24:38690, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:32.390638 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:32.390489207 +0000 UTC m=+232.098777772, time spent = 114.063Âµs, remote = 10.244.8.24:38716, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:32.842345 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:32.842148999 +0000 UTC m=+232.550437564, time spent = 160.32Âµs, remote = 10.244.8.24:38718, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:42.397744 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:42.397564025 +0000 UTC m=+242.105852590, time spent = 145.953Âµs, remote = 10.244.8.24:38746, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health" 
etcd-2 etcd 2021-03-24 16:29:52.403450 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:52.403316026 +0000 UTC m=+252.111604591, time spent = 104.646Âµs, remote = 10.244.8.24:38772, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 27, request content = key:"health"

Cluster 2 ( working ):

devops-m1:~$ !ls
ls -l /data/archive/etcd-snaps/
total 120
-rw------- 1 1001 root 118816 Mar 24 15:30 db

devops-m1:~$ kubectl get po | grep etcd
etcd-0                                         1/1     Running   0          10m
etcd-1                                         1/1     Running   1          10m
etcd-2                                         1/1     Running   0          10m

devops-m1:~$ helm get values etcd
USER-SUPPLIED VALUES:
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/instance
            operator: In
            values:
            - etcd
        topologyKey: kubernetes.io/hostname
      weight: 100
args: []
auth:
  client:
    caFilename: ""
    certFilename: cert.pem
    certKeyFilename: key.pem
    enableAuthentication: false
    existingSecret: ""
    secureTransport: false
    useAutoTLS: false
  peer:
    caFilename: ""
    certFilename: cert.pem
    certKeyFilename: key.pem
    enableAuthentication: false
    existingSecret: ""
    secureTransport: false
    useAutoTLS: false
  rbac:
    allowNoneAuthentication: true
    enabled: false
    existingSecret: ""
    rootPassword: ""
clusterDomain: cluster.local
command: []
commonAnnotations: {}
commonLabels: {}
containerPorts:
  client: 2379
  peer: 2380
containerSecurityContext:
  enabled: true
  runAsNonRoot: true
  runAsUser: 1001
customLivenessProbe: {}
customReadinessProbe: {}
customStartupProbe: {}
disasterRecovery:
  cronjob:
    historyLimit: 1
    podAnnotations: {}
    resources:
      limits: {}
      requests: {}
    schedule: '*/30 * * * *'
    snapshotHistoryLimit: 1
  enabled: false
  pvc:
    existingClaim: etcd-snapshot-volume
    size: 2Gi
    storageClassName: nfs
extraDeploy: []
extraEnvVars: []
extraEnvVarsCM: ""
extraEnvVarsSecret: ""
extraVolumeMounts: []
extraVolumes: []
hostAliases: []
image:
  debug: true
  pullPolicy: IfNotPresent
  pullSecrets: []
  registry: docker.io
  repository: bitnami/etcd
  tag: 3.4.15-debian-10-r14
initContainers: {}
initialClusterState: existing
kubeVersion: ""
lifecycleHooks: {}
livenessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 60
  periodSeconds: 30
  successThreshold: 1
  timeoutSeconds: 5
metrics:
  enabled: true
  podAnnotations:
    prometheus.io/port: "2379"
    prometheus.io/scrape: "true"
  podMonitor:
    additionalLabels: {}
    enabled: false
    interval: 30s
    scheme: http
    tlsConfig: {}
nodeAffinityPreset:
  key: ""
  type: ""
  values: []
nodeSelector: {}
pdb:
  create: false
  minAvailable: 1
persistence:
  accessModes:
  - ReadWriteOnce
  annotations: {}
  enabled: true
  selector: {}
  size: 16Gi
  storageClass: rook-ceph-block
podAffinityPreset: ""
podAnnotations:
  backup.velero.io/backup-volumes: data
podAntiAffinity: soft
podAntiAffinityPreset: soft
podLabels: {}
podManagementPolicy: Parallel
podSecurityContext:
  enabled: true
  fsGroup: 1001
priorityClassName: ""
readinessProbe:
  enabled: true
  failureThreshold: 5
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
removeMemberOnContainerTermination: true
replicaCount: 3
resources:
  limits: {}
  requests: {}
service:
  annotations: {}
  clientPortNameOverride: ""
  externalIPs: []
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  nodePorts:
    clientPort: ""
    peerPort: ""
  peerPort: 2380
  peerPortNameOverride: ""
  port: 2379
  type: ClusterIP
sidecars: {}
startFromSnapshot:
  enabled: true
  existingClaim: etcd-snapshot-volume
  snapshotFilename: db
startupProbe:
  enabled: false
  failureThreshold: 60
  initialDelaySeconds: 0
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 5
tolerations: []
updateStrategy:
  type: RollingUpdate
volumePermissions:
  enabled: false
  image:
    pullPolicy: Always
    pullSecrets: []
    registry: docker.io
    repository: bitnami/bitnami-shell
    tag: "10"
  resources:
    limits: {}
    requests: {}

devops-m1:~$ cat etcd.log
etcd-1 etcd etcd 16:27:55.96
etcd-1 etcd etcd 16:27:55.96 Welcome to the Bitnami etcd container
etcd-1 etcd etcd 16:27:55.96 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd-1 etcd etcd 16:27:55.96 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd-1 etcd etcd 16:27:55.97
etcd-1 etcd etcd 16:27:55.97 INFO  ==> ** Starting etcd setup **
etcd-1 etcd etcd 16:27:55.99 INFO  ==> Validating settings in ETCD_* env vars..
etcd-1 etcd etcd 16:27:55.99 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd-1 etcd etcd 16:27:56.00 INFO  ==> Initializing etcd
etcd-1 etcd etcd 16:27:56.01 INFO  ==> Detected data from previous deployments
etcd-1 etcd etcd 16:27:56.02 DEBUG ==> Setting data directory permissions to 700 in a recursive way (required in etcd >=3.4.10)
etcd-1 etcd etcd 16:28:01.12 INFO  ==> Updating member in existing cluster
etcd-0 etcd etcd 16:28:03.04
etcd-0 etcd etcd 16:28:03.04 Welcome to the Bitnami etcd container
etcd-0 etcd etcd 16:28:03.04 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd-0 etcd etcd 16:28:03.05 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd-0 etcd etcd 16:28:03.05
etcd-0 etcd etcd 16:28:03.05 INFO  ==> ** Starting etcd setup **
etcd-0 etcd etcd 16:28:03.08 INFO  ==> Validating settings in ETCD_* env vars..
etcd-0 etcd etcd 16:28:03.08 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd-0 etcd etcd 16:28:03.09 INFO  ==> Initializing etcd
etcd-0 etcd etcd 16:28:03.10 INFO  ==> Detected data from previous deployments
etcd-0 etcd etcd 16:28:03.11 DEBUG ==> Setting data directory permissions to 700 in a recursive way (required in etcd >=3.4.10)
etcd-2 etcd etcd 16:28:03.14
etcd-2 etcd etcd 16:28:03.14 Welcome to the Bitnami etcd container
etcd-2 etcd etcd 16:28:03.15 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd
etcd-2 etcd etcd 16:28:03.15 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues
etcd-2 etcd etcd 16:28:03.15
etcd-2 etcd etcd 16:28:03.16 INFO  ==> ** Starting etcd setup **
etcd-2 etcd etcd 16:28:03.18 INFO  ==> Validating settings in ETCD_* env vars..
etcd-2 etcd etcd 16:28:03.19 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd-2 etcd etcd 16:28:03.19 INFO  ==> Initializing etcd
etcd-2 etcd etcd 16:28:03.21 INFO  ==> Detected data from previous deployments
etcd-2 etcd etcd 16:28:03.22 DEBUG ==> Setting data directory permissions to 700 in a recursive way (required in etcd >=3.4.10)
etcd-1 etcd {"level":"warn","ts":"2021-03-24T16:28:11.248Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-6d184985-b14b-4cff-81c7-e965616ed694/etcd-0.etcd-headless.default.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp: lookup etcd-2.etcd-headless.default.svc.cluster.local on 10.96.0.10:53: no such host\""}
etcd-1 etcd Error: context deadline exceeded
etcd-0 etcd etcd 16:28:23.29 DEBUG ==> There are no enough active endpoints!
etcd-0 etcd etcd 16:28:23.29 WARN  ==> Cluster not responding!
etcd-0 etcd etcd 16:28:23.30 WARN  ==> Disaster recovery is disabled, the cluster will try to recover on it's own
etcd-0 etcd etcd 16:28:23.31 INFO  ==> ** etcd setup finished! **
etcd-0 etcd
etcd-2 etcd etcd 16:28:23.39 DEBUG ==> There are no enough active endpoints!
etcd-2 etcd etcd 16:28:23.40 WARN  ==> Cluster not responding!
etcd-2 etcd etcd 16:28:23.40 WARN  ==> Disaster recovery is disabled, the cluster will try to recover on it's own
etcd-2 etcd
etcd-2 etcd etcd 16:28:23.41 INFO  ==> ** etcd setup finished! **
etcd-0 etcd etcd 16:28:28.46 INFO  ==> ** Starting etcd **
etcd-0 etcd 2021-03-24 16:28:28.486630 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd-0.etcd-headless.default.svc.cluster.local:2379
etcd-0 etcd 2021-03-24 16:28:28.486707 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false
etcd-0 etcd 2021-03-24 16:28:28.486722 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
etcd-0 etcd 2021-03-24 16:28:28.486764 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-0.etcd-headless.default.svc.cluster.local:2380
etcd-0 etcd 2021-03-24 16:28:28.486804 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
etcd-0 etcd 2021-03-24 16:28:28.486811 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
etcd-0 etcd 2021-03-24 16:28:28.486817 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
etcd-0 etcd 2021-03-24 16:28:28.486833 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
etcd-0 etcd 2021-03-24 16:28:28.486861 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
etcd-0 etcd 2021-03-24 16:28:28.486869 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=debug
etcd-0 etcd 2021-03-24 16:28:28.486884 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-0
etcd-0 etcd 2021-03-24 16:28:28.486891 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false
etcd-0 etcd 2021-03-24 16:28:28.486944 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE=
etcd-0 etcd 2021-03-24 16:28:28.486972 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.99.65.159
etcd-0 etcd 2021-03-24 16:28:28.486977 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_ADDR=10.99.65.159
etcd-0 etcd 2021-03-24 16:28:28.486982 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.99.65.159:2379
etcd-0 etcd 2021-03-24 16:28:28.487004 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes
etcd-0 etcd 2021-03-24 16:28:28.487008 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PROTO=tcp
etcd-0 etcd 2021-03-24 16:28:28.487015 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots
etcd-0 etcd 2021-03-24 16:28:28.487023 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin
etcd-0 etcd 2021-03-24 16:28:28.487032 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd
etcd-0 etcd 2021-03-24 16:28:28.487043 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
etcd-0 etcd 2021-03-24 16:28:28.487048 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.99.65.159
etcd-0 etcd 2021-03-24 16:28:28.487064 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PORT=2380
etcd-0 etcd 2021-03-24 16:28:28.487073 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD=
etcd-0 etcd 2021-03-24 16:28:28.487083 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP=tcp://10.99.65.159:2380
etcd-0 etcd 2021-03-24 16:28:28.487089 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=etcd-headless.default.svc.cluster.local
etcd-0 etcd 2021-03-24 16:28:28.487095 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no
etcd-0 etcd 2021-03-24 16:28:28.487101 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE=
etcd-0 etcd 2021-03-24 16:28:28.487112 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd
etcd-0 etcd 2021-03-24 16:28:28.487116 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=yes
etcd-0 etcd 2021-03-24 16:28:28.487130 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_CLIENT=2379
etcd-0 etcd 2021-03-24 16:28:28.487138 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot
etcd-0 etcd 2021-03-24 16:28:28.487152 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_PEER=2380
etcd-0 etcd 2021-03-24 16:28:28.487162 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd
etcd-0 etcd 2021-03-24 16:28:28.487176 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
etcd-0 etcd 2021-03-24 16:28:28.487183 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE=
etcd-0 etcd 2021-03-24 16:28:28.487189 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.99.65.159:2379
etcd-0 etcd 2021-03-24 16:28:28.487193 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
etcd-0 etcd 2021-03-24 16:28:28.487197 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs
etcd-0 etcd 2021-03-24 16:28:28.487201 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd
etcd-0 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-0 etcd 2021-03-24 16:28:28.487254 I | etcdmain: etcd Version: 3.4.15
etcd-0 etcd 2021-03-24 16:28:28.487258 I | etcdmain: Git SHA: aa7126864
etcd-0 etcd 2021-03-24 16:28:28.487262 I | etcdmain: Go Version: go1.12.17
etcd-0 etcd 2021-03-24 16:28:28.487266 I | etcdmain: Go OS/Arch: linux/amd64
etcd-0 etcd 2021-03-24 16:28:28.487271 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
etcd-0 etcd 2021-03-24 16:28:28.487342 W | etcdmain: found invalid file/dir member_id under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
etcd-0 etcd 2021-03-24 16:28:28.487355 N | etcdmain: the server is already initialized as member before, starting as etcd member...
etcd-0 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-0 etcd 2021-03-24 16:28:28.487740 I | embed: name = etcd-0
etcd-0 etcd 2021-03-24 16:28:28.487758 I | embed: data dir = /bitnami/etcd/data
etcd-0 etcd 2021-03-24 16:28:28.487763 I | embed: member dir = /bitnami/etcd/data/member
etcd-0 etcd 2021-03-24 16:28:28.487768 I | embed: heartbeat = 100ms
etcd-0 etcd 2021-03-24 16:28:28.487772 I | embed: election = 1000ms
etcd-0 etcd 2021-03-24 16:28:28.487776 I | embed: snapshot count = 100000
etcd-0 etcd 2021-03-24 16:28:28.487787 I | embed: advertise client URLs = http://etcd-0.etcd-headless.default.svc.cluster.local:2379
etcd-0 etcd 2021-03-24 16:28:28.487796 I | embed: initial advertise peer URLs = http://etcd-0.etcd-headless.default.svc.cluster.local:2380
etcd-0 etcd 2021-03-24 16:28:28.487804 I | embed: initial cluster =
etcd-0 etcd 2021-03-24 16:28:28.521285 I | etcdserver: restarting member df3b2df95cd5fd29 in cluster 9e98e654be1e22d7 at commit index 40
etcd-0 etcd raft2021/03/24 16:28:28 INFO: df3b2df95cd5fd29 switched to configuration voters=()
etcd-0 etcd raft2021/03/24 16:28:28 INFO: df3b2df95cd5fd29 became follower at term 21
etcd-0 etcd raft2021/03/24 16:28:28 INFO: newRaft df3b2df95cd5fd29 [peers: [], term: 21, commit: 40, applied: 0, lastindex: 40, lastterm: 18]
etcd-0 etcd 2021-03-24 16:28:28.528237 W | auth: simple token is not cryptographically signed
etcd-0 etcd 2021-03-24 16:28:28.540077 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
etcd-0 etcd raft2021/03/24 16:28:28 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030)
etcd-0 etcd 2021-03-24 16:28:28.540793 I | etcdserver/membership: added member 45a18acb10aa275e [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:28:28.540812 I | rafthttp: starting peer 45a18acb10aa275e...
etcd-0 etcd 2021-03-24 16:28:28.540918 I | rafthttp: started HTTP pipelining with peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:28:28.541644 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:28:28.543233 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:28:28.545774 I | embed: listening for peers on [::]:2380
etcd-0 etcd 2021-03-24 16:28:28.547923 I | rafthttp: started peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:28:28.547967 I | rafthttp: added peer 45a18acb10aa275e
etcd-0 etcd raft2021/03/24 16:28:28 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805)
etcd-0 etcd 2021-03-24 16:28:28.548313 I | etcdserver/membership: added member 936ce633ac273d75 [http://etcd-1.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:28:28.548346 I | rafthttp: starting peer 936ce633ac273d75...
etcd-0 etcd 2021-03-24 16:28:28.548451 I | rafthttp: started HTTP pipelining with peer 936ce633ac273d75
etcd-0 etcd 2021-03-24 16:28:28.548712 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-0 etcd 2021-03-24 16:28:28.548749 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-0 etcd 2021-03-24 16:28:28.551245 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-0 etcd 2021-03-24 16:28:28.551347 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-0 etcd 2021-03-24 16:28:28.551976 I | rafthttp: started peer 936ce633ac273d75
etcd-0 etcd 2021-03-24 16:28:28.552009 I | rafthttp: added peer 936ce633ac273d75
etcd-0 etcd 2021-03-24 16:28:28.552059 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream Message reader)
etcd-0 etcd raft2021/03/24 16:28:28 INFO: df3b2df95cd5fd29 switched to configuration voters=(5017444064630024030 10623118730666130805 16085501043111492905)
etcd-0 etcd 2021-03-24 16:28:28.552281 I | etcdserver/membership: added member df3b2df95cd5fd29 [http://etcd-0.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:28:28.552537 N | etcdserver/membership: set the initial cluster version to 3.0
etcd-0 etcd 2021-03-24 16:28:28.552696 I | etcdserver/api: enabled capabilities for version 3.0
etcd-0 etcd 2021-03-24 16:28:28.552761 N | etcdserver/membership: updated the cluster version from 3.0 to 3.4
etcd-0 etcd 2021-03-24 16:28:28.552905 I | etcdserver/api: enabled capabilities for version 3.4
etcd-0 etcd 2021-03-24 16:28:28.554165 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-0 etcd raft2021/03/24 16:28:28 INFO: df3b2df95cd5fd29 switched to configuration voters=(10623118730666130805 16085501043111492905)
etcd-0 etcd 2021-03-24 16:28:28.554438 I | etcdserver/membership: removed member 45a18acb10aa275e from cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:28:28.554454 I | rafthttp: stopping peer 45a18acb10aa275e...
etcd-0 etcd 2021-03-24 16:28:28.554508 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:28:28.554522 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-0 etcd 2021-03-24 16:28:28.554553 I | rafthttp: stopped HTTP pipelining with peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:28:28.554576 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-0 etcd 2021-03-24 16:28:28.554591 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-0 etcd 2021-03-24 16:28:28.554597 I | rafthttp: stopped peer 45a18acb10aa275e
etcd-0 etcd 2021-03-24 16:28:28.554608 I | rafthttp: removed peer 45a18acb10aa275e
etcd-0 etcd raft2021/03/24 16:28:28 INFO: df3b2df95cd5fd29 switched to configuration voters=(10623118730666130805 16085501043111492905 16416925743178806112)
etcd-0 etcd 2021-03-24 16:28:28.554802 I | etcdserver/membership: added member e3d4a2fb9a237360 [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:28:28.554814 I | rafthttp: starting peer e3d4a2fb9a237360...
etcd-0 etcd 2021-03-24 16:28:28.554838 I | rafthttp: started HTTP pipelining with peer e3d4a2fb9a237360
etcd-0 etcd 2021-03-24 16:28:28.555138 I | rafthttp: started streaming with peer e3d4a2fb9a237360 (writer)
etcd-0 etcd 2021-03-24 16:28:28.555462 I | rafthttp: started peer e3d4a2fb9a237360
etcd-0 etcd 2021-03-24 16:28:28.555495 I | rafthttp: added peer e3d4a2fb9a237360
etcd-0 etcd 2021-03-24 16:28:28.555585 I | rafthttp: started streaming with peer e3d4a2fb9a237360 (writer)
etcd-0 etcd 2021-03-24 16:28:28.555613 I | rafthttp: started streaming with peer e3d4a2fb9a237360 (stream MsgApp v2 reader)
etcd-0 etcd 2021-03-24 16:28:28.555734 I | rafthttp: started streaming with peer e3d4a2fb9a237360 (stream Message reader)
etcd-2 etcd etcd 16:28:28.55 DEBUG ==> Loading env vars of existing cluster
etcd-2 etcd etcd 16:28:28.56 INFO  ==> ** Starting etcd **
etcd-2 etcd 2021-03-24 16:28:28.578887 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://etcd-2.etcd-headless.default.svc.cluster.local:2379
etcd-2 etcd 2021-03-24 16:28:28.578979 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false
etcd-2 etcd 2021-03-24 16:28:28.579134 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data
etcd-2 etcd 2021-03-24 16:28:28.579198 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
etcd-2 etcd 2021-03-24 16:28:28.579254 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=etcd-1=http://etcd-1.etcd-headless.default.svc.cluster.local:2380,etcd-0=http://etcd-0.etcd-headless.default.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.default.svc.cluster.local:2380
etcd-2 etcd 2021-03-24 16:28:28.579272 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
etcd-2 etcd 2021-03-24 16:28:28.579279 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s
etcd-2 etcd 2021-03-24 16:28:28.579294 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
etcd-2 etcd 2021-03-24 16:28:28.579376 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
etcd-2 etcd 2021-03-24 16:28:28.579391 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=debug
etcd-2 etcd 2021-03-24 16:28:28.579414 I | pkg/flags: recognized and used environment variable ETCD_NAME=etcd-2
etcd-2 etcd 2021-03-24 16:28:28.579424 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false
etcd-2 etcd 2021-03-24 16:28:28.579497 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE=
etcd-2 etcd 2021-03-24 16:28:28.579522 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=10.99.65.159
etcd-2 etcd 2021-03-24 16:28:28.579527 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_ADDR=10.99.65.159
etcd-2 etcd 2021-03-24 16:28:28.579585 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://10.99.65.159:2379
etcd-2 etcd 2021-03-24 16:28:28.579612 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes
etcd-2 etcd 2021-03-24 16:28:28.579624 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PROTO=tcp
etcd-2 etcd 2021-03-24 16:28:28.579677 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots
etcd-2 etcd 2021-03-24 16:28:28.579687 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin
etcd-2 etcd 2021-03-24 16:28:28.579755 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd
etcd-2 etcd 2021-03-24 16:28:28.579766 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
etcd-2 etcd 2021-03-24 16:28:28.579772 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=10.99.65.159
etcd-2 etcd 2021-03-24 16:28:28.579789 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP_PORT=2380
etcd-2 etcd 2021-03-24 16:28:28.579801 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD=
etcd-2 etcd 2021-03-24 16:28:28.579837 W | pkg/flags: unrecognized environment variable ETCD_PORT_2380_TCP=tcp://10.99.65.159:2380
etcd-2 etcd 2021-03-24 16:28:28.579845 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=etcd-headless.default.svc.cluster.local
etcd-2 etcd 2021-03-24 16:28:28.579850 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no
etcd-2 etcd 2021-03-24 16:28:28.579857 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE=
etcd-2 etcd 2021-03-24 16:28:28.579869 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd
etcd-2 etcd 2021-03-24 16:28:28.579873 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=yes
etcd-2 etcd 2021-03-24 16:28:28.579886 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_CLIENT=2379
etcd-2 etcd 2021-03-24 16:28:28.579893 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot
etcd-2 etcd 2021-03-24 16:28:28.579911 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_PEER=2380
etcd-2 etcd 2021-03-24 16:28:28.579917 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd
etcd-2 etcd 2021-03-24 16:28:28.579928 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
etcd-2 etcd 2021-03-24 16:28:28.579936 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE=
etcd-2 etcd 2021-03-24 16:28:28.579947 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
etcd-2 etcd 2021-03-24 16:28:28.579952 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://10.99.65.159:2379
etcd-2 etcd 2021-03-24 16:28:28.579957 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs
etcd-2 etcd 2021-03-24 16:28:28.579961 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd
etcd-2 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-2 etcd 2021-03-24 16:28:28.580012 I | etcdmain: etcd Version: 3.4.15
etcd-2 etcd 2021-03-24 16:28:28.580017 I | etcdmain: Git SHA: aa7126864
etcd-2 etcd 2021-03-24 16:28:28.580021 I | etcdmain: Go Version: go1.12.17
etcd-2 etcd 2021-03-24 16:28:28.580025 I | etcdmain: Go OS/Arch: linux/amd64
etcd-2 etcd 2021-03-24 16:28:28.580030 I | etcdmain: setting maximum number of CPUs to 6, total number of available CPUs is 6
etcd-2 etcd 2021-03-24 16:28:28.580109 W | etcdmain: found invalid file/dir member_id under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
etcd-2 etcd 2021-03-24 16:28:28.580122 W | etcdmain: found invalid file/dir new_member_envs under data dir /bitnami/etcd/data (Ignore this if you are upgrading etcd)
etcd-2 etcd 2021-03-24 16:28:28.580131 N | etcdmain: the server is already initialized as member before, starting as etcd member...
etcd-2 etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
etcd-2 etcd 2021-03-24 16:28:28.580344 I | embed: name = etcd-2
etcd-2 etcd 2021-03-24 16:28:28.580355 I | embed: data dir = /bitnami/etcd/data
etcd-2 etcd 2021-03-24 16:28:28.580360 I | embed: member dir = /bitnami/etcd/data/member
etcd-2 etcd 2021-03-24 16:28:28.580365 I | embed: heartbeat = 100ms
etcd-2 etcd 2021-03-24 16:28:28.580369 I | embed: election = 1000ms
etcd-2 etcd 2021-03-24 16:28:28.580374 I | embed: snapshot count = 100000
etcd-2 etcd 2021-03-24 16:28:28.580384 I | embed: advertise client URLs = http://etcd-2.etcd-headless.default.svc.cluster.local:2379
etcd-2 etcd 2021-03-24 16:28:28.580395 I | embed: initial advertise peer URLs = http://etcd-2.etcd-headless.default.svc.cluster.local:2380
etcd-2 etcd 2021-03-24 16:28:28.580401 I | embed: initial cluster =
etcd-2 etcd 2021-03-24 16:28:28.591424 I | etcdserver: restarting member e3d4a2fb9a237360 in cluster 9e98e654be1e22d7 at commit index 39
etcd-2 etcd raft2021/03/24 16:28:28 INFO: e3d4a2fb9a237360 switched to configuration voters=()
etcd-2 etcd raft2021/03/24 16:28:28 INFO: e3d4a2fb9a237360 became follower at term 17
etcd-2 etcd raft2021/03/24 16:28:28 INFO: newRaft e3d4a2fb9a237360 [peers: [], term: 17, commit: 39, applied: 0, lastindex: 39, lastterm: 17]
etcd-2 etcd 2021-03-24 16:28:28.593713 W | auth: simple token is not cryptographically signed
etcd-2 etcd 2021-03-24 16:28:28.601255 I | etcdserver: starting server... [version: 3.4.15, cluster version: to_be_decided]
etcd-2 etcd raft2021/03/24 16:28:28 INFO: e3d4a2fb9a237360 switched to configuration voters=(5017444064630024030)
etcd-2 etcd 2021-03-24 16:28:28.602326 I | etcdserver/membership: added member 45a18acb10aa275e [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:28.602363 I | rafthttp: starting peer 45a18acb10aa275e...
etcd-2 etcd 2021-03-24 16:28:28.602456 I | rafthttp: started HTTP pipelining with peer 45a18acb10aa275e
etcd-2 etcd 2021-03-24 16:28:28.604433 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-2 etcd 2021-03-24 16:28:28.604545 I | rafthttp: started streaming with peer 45a18acb10aa275e (writer)
etcd-2 etcd 2021-03-24 16:28:28.605498 I | rafthttp: started peer 45a18acb10aa275e
etcd-2 etcd 2021-03-24 16:28:28.605549 I | rafthttp: added peer 45a18acb10aa275e
etcd-2 etcd 2021-03-24 16:28:28.606337 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-2 etcd raft2021/03/24 16:28:28 INFO: e3d4a2fb9a237360 switched to configuration voters=(5017444064630024030 10623118730666130805)
etcd-2 etcd 2021-03-24 16:28:28.606435 I | embed: listening for peers on [::]:2380
etcd-2 etcd 2021-03-24 16:28:28.606606 I | etcdserver/membership: added member 936ce633ac273d75 [http://etcd-1.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:28.606689 I | rafthttp: starting peer 936ce633ac273d75...
etcd-2 etcd 2021-03-24 16:28:28.606753 I | rafthttp: started streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-2 etcd 2021-03-24 16:28:28.607273 I | rafthttp: started HTTP pipelining with peer 936ce633ac273d75
etcd-2 etcd 2021-03-24 16:28:28.609808 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-2 etcd 2021-03-24 16:28:28.612765 I | rafthttp: started streaming with peer 936ce633ac273d75 (writer)
etcd-2 etcd 2021-03-24 16:28:28.613028 I | rafthttp: started peer 936ce633ac273d75
etcd-2 etcd 2021-03-24 16:28:28.613061 I | rafthttp: added peer 936ce633ac273d75
etcd-2 etcd 2021-03-24 16:28:28.613068 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-2 etcd raft2021/03/24 16:28:28 INFO: e3d4a2fb9a237360 switched to configuration voters=(5017444064630024030 10623118730666130805 16085501043111492905)
etcd-2 etcd 2021-03-24 16:28:28.613230 I | etcdserver/membership: added member df3b2df95cd5fd29 [http://etcd-0.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:28.613284 I | rafthttp: started streaming with peer 936ce633ac273d75 (stream Message reader)
etcd-2 etcd 2021-03-24 16:28:28.613340 I | rafthttp: starting peer df3b2df95cd5fd29...
etcd-2 etcd 2021-03-24 16:28:28.613700 I | rafthttp: started HTTP pipelining with peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:28:28.615561 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
etcd-2 etcd 2021-03-24 16:28:28.615647 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (writer)
etcd-0 etcd 2021-03-24 16:28:28.622459 I | rafthttp: peer e3d4a2fb9a237360 became active
etcd-0 etcd 2021-03-24 16:28:28.622636 I | rafthttp: established a TCP streaming connection with peer e3d4a2fb9a237360 (stream Message writer)
etcd-2 etcd 2021-03-24 16:28:28.616393 I | rafthttp: started peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:28:28.616416 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream Message reader)
etcd-2 etcd 2021-03-24 16:28:28.616425 I | rafthttp: added peer df3b2df95cd5fd29
etcd-2 etcd 2021-03-24 16:28:28.616486 I | rafthttp: started streaming with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:28:28.616527 I | rafthttp: started HTTP pipelining with peer e3d4a2fb9a237360
etcd-2 etcd 2021-03-24 16:28:28.616543 E | rafthttp: failed to find member e3d4a2fb9a237360 in cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:28.616618 E | rafthttp: failed to find member e3d4a2fb9a237360 in cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:28.616752 N | etcdserver/membership: set the initial cluster version to 3.0
etcd-2 etcd 2021-03-24 16:28:28.616847 I | etcdserver/api: enabled capabilities for version 3.0
etcd-2 etcd 2021-03-24 16:28:28.617046 N | etcdserver/membership: updated the cluster version from 3.0 to 3.4
etcd-2 etcd 2021-03-24 16:28:28.617134 I | etcdserver/api: enabled capabilities for version 3.4
etcd-2 etcd 2021-03-24 16:28:28.617389 E | rafthttp: failed to find member e3d4a2fb9a237360 in cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:28.617427 E | rafthttp: failed to find member e3d4a2fb9a237360 in cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:28.618296 I | rafthttp: peer df3b2df95cd5fd29 became active
etcd-2 etcd 2021-03-24 16:28:28.618317 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream MsgApp v2 reader)
etcd-0etcd-2 etcd 2021-03-24 16:28:28.619162 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream Message reader)
etcd-2 etcd raft2021/03/24 16:28:28 INFO: e3d4a2fb9a237360 switched to configuration voters=(10623118730666130805 16085501043111492905)
etcd-2  etcd 2021-03-24 16:28:28.619283 I | etcdserver/membership: removed member 45a18acb10aa275e from cluster 9e98e654be1e22d7
etcdetcd-2 etcd  2021-03-24 16:28:28.619302 I | rafthttp: stopping peer 45a18acb10aa275e...
etcd-22021-03-24 16:28:28.625509 I | rafthttp: established a TCP streaming connection with peer e3d4a2fb9a237360 (stream MsgApp v2 writer)
 etcd 2021-03-24 16:28:28.619329 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-2 etcd 2021-03-24 16:28:28.619345 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (writer)
etcd-2 etcd 2021-03-24 16:28:28.619372 I | rafthttp: stopped HTTP pipelining with peer 45a18acb10aa275e
etcd-2 etcd 2021-03-24 16:28:28.619384 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:28:28.619447 I | rafthttp: stopped streaming with peer 45a18acb10aa275e (stream Message reader)
etcd-2 etcd 2021-03-24 16:28:28.619463 I | rafthttp: stopped peer 45a18acb10aa275e
etcd-2 etcd 2021-03-24 16:28:28.619480 I | rafthttp: removed peer 45a18acb10aa275e
etcd-2 etcd raft2021/03/24 16:28:28 INFO: e3d4a2fb9a237360 switched to configuration voters=(10623118730666130805 16085501043111492905 16416925743178806112)
etcd-2 etcd 2021-03-24 16:28:28.619893 I | etcdserver/membership: added member e3d4a2fb9a237360 [http://etcd-2.etcd-headless.default.svc.cluster.local:2380] to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:28:28.659087 I | rafthttp: established a TCP streaming connection with peer e3d4a2fb9a237360 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:28:28.656010 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream MsgApp v2 writer)
etcd-2 etcd 2021-03-24 16:28:28.656176 I | rafthttp: established a TCP streaming connection with peer df3b2df95cd5fd29 (stream Message writer)
etcd-0 etcd 2021-03-24 16:28:28.660584 I | rafthttp: established a TCP streaming connection with peer e3d4a2fb9a237360 (stream Message reader)
etcd-2 etcd raft2021/03/24 16:28:29 INFO: e3d4a2fb9a237360 is starting a new election at term 17
etcd-2 etcd raft2021/03/24 16:28:29 INFO: e3d4a2fb9a237360 became candidate at term 18
etcd-2 etcd raft2021/03/24 16:28:29 INFO: e3d4a2fb9a237360 received MsgVoteResp from e3d4a2fb9a237360 at term 18
etcd-2 etcd raft2021/03/24 16:28:29 INFO: e3d4a2fb9a237360 [logterm: 17, index: 39] sent MsgVote request to 936ce633ac273d75 at term 18
etcd-2 etcd raft2021/03/24 16:28:29 INFO: e3d4a2fb9a237360 [logterm: 17, index: 39] sent MsgVote request to df3b2df95cd5fd29 at term 18
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 [term: 21] ignored a MsgVote message with lower term from e3d4a2fb9a237360 [term: 18]
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 is starting a new election at term 21
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 became candidate at term 22
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 received MsgVoteResp from df3b2df95cd5fd29 at term 22
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 [logterm: 18, index: 40] sent MsgVote request to 936ce633ac273d75 at term 22
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 [logterm: 18, index: 40] sent MsgVote request to e3d4a2fb9a237360 at term 22
etcd-2 etcd raft2021/03/24 16:28:30 INFO: e3d4a2fb9a237360 [term: 18] received a MsgVote message with higher term from df3b2df95cd5fd29 [term: 22]
etcd-2 etcd raft2021/03/24 16:28:30 INFO: e3d4a2fb9a237360 became follower at term 22
etcd-2 etcd raft2021/03/24 16:28:30 INFO: e3d4a2fb9a237360 [logterm: 17, index: 39, vote: 0] cast MsgVote for df3b2df95cd5fd29 [logterm: 18, index: 40] at term 22
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 received MsgVoteResp from e3d4a2fb9a237360 at term 22
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 has received 2 MsgVoteResp votes and 0 vote rejections
etcd-0 etcd raft2021/03/24 16:28:30 INFO: df3b2df95cd5fd29 became leader at term 22
etcd-0 etcd raft2021/03/24 16:28:30 INFO: raft.node: df3b2df95cd5fd29 elected leader df3b2df95cd5fd29 at term 22
etcd-2 etcd raft2021/03/24 16:28:30 INFO: raft.node: e3d4a2fb9a237360 elected leader df3b2df95cd5fd29 at term 22
etcd-2 etcd 2021-03-24 16:28:30.455994 I | embed: ready to serve client requests
etcd-0 etcd 2021-03-24 16:28:30.458830 I | etcdserver: published {Name:etcd-0 ClientURLs:[http://etcd-0.etcd-headless.default.svc.cluster.local:2379]} to cluster 9e98e654be1e22d7
etcd-0 etcd 2021-03-24 16:28:30.458990 I | embed: ready to serve client requests
etcd-0 etcd 2021-03-24 16:28:30.460488 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
etcd-0 etcd 2021-03-24 16:28:30.460914 W | etcdserver: failed to reach the peerURL(http://etcd-1.etcd-headless.default.svc.cluster.local:2380) of member 936ce633ac273d75 (Get http://etcd-1.etcd-headless.default.svc.cluster.local:2380/version: dial tcp 10.244.1.55:2380: connect: connection refused)
etcd-2 etcd 2021-03-24 16:28:30.456426 I | etcdserver: published {Name:etcd-2 ClientURLs:[http://etcd-2.etcd-headless.default.svc.cluster.local:2379]} to cluster 9e98e654be1e22d7
etcd-2 etcd 2021-03-24 16:28:30.457188 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
etcd-2 etcd 2021-03-24 16:28:30.460513 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:28:30.458933802 +0000 UTC m=+1.890722228, time spent = 1.481951ms, remote = 10.244.1.55:47112, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-0 etcd 2021-03-24 16:28:30.460940 W | etcdserver: cannot get the version of member 936ce633ac273d75 (Get http://etcd-1.etcd-headless.default.svc.cluster.local:2380/version: dial tcp 10.244.1.55:2380: connect: connection refused)
etcd-0 etcd 2021-03-24 16:28:33.553990 W | rafthttp: health check for peer 936ce633ac273d75 could not connect: dial tcp 10.244.1.55:2380: connect: connection refused
etcd-0 etcd 2021-03-24 16:28:33.554864 W | rafthttp: health check for peer 936ce633ac273d75 could not connect: dial tcp 10.244.1.55:2380: connect: connection refused
etcd-2 etcd 2021-03-24 16:28:33.613529 W | rafthttp: health check for peer 936ce633ac273d75 could not connect: dial tcp 10.244.1.55:2380: connect: connection refused
etcd-2 etcd 2021-03-24 16:28:33.613647 W | rafthttp: health check for peer 936ce633ac273d75 could not connect: dial tcp 10.244.1.55:2380: connect: connection refused
etcd-0 etcd 2021-03-24 16:28:34.465170 W | etcdserver: failed to reach the peerURL(http://etcd-1.etcd-headless.default.svc.cluster.local:2380) of member 936ce633ac273d75 (Get http://etcd-1.etcd-headless.default.svc.cluster.local:2380/version: dial tcp 10.244.1.55:2380: connect: connection refused)
etcd-0 etcd 2021-03-24 16:28:34.465429 W | etcdserver: cannot get the version of member 936ce633ac273d75 (Get http://etcd-1.etcd-headless.default.svc.cluster.local:2380/version: dial tcp 10.244.1.55:2380: connect: connection refused)
etcd-2 etcd 2021-03-24 16:28:35.680542 E | rafthttp: streaming request ignored (ID mismatch got 45a18acb10aa275e want e3d4a2fb9a237360)
etcd-2 etcd 2021-03-24 16:28:35.681987 E | rafthttp: streaming request ignored (ID mismatch got 45a18acb10aa275e want e3d4a2fb9a237360)
etcd-0 etcd 2021-03-24 16:28:35.688651 I | rafthttp: peer 936ce633ac273d75 became active
etcd-0 etcd 2021-03-24 16:28:35.688710 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream Message reader)
etcd-0 etcd 2021-03-24 16:28:35.688924 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:28:35.688785 E | rafthttp: streaming request ignored (ID mismatch got 45a18acb10aa275e want e3d4a2fb9a237360)
etcd-2 etcd 2021-03-24 16:28:35.689559 E | rafthttp: streaming request ignored (ID mismatch got 45a18acb10aa275e want e3d4a2fb9a237360)
etcd-0 etcd 2021-03-24 16:28:35.694403 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream Message writer)
etcd-0 etcd 2021-03-24 16:28:35.698523 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 writer)
etcd-2 etcd 2021-03-24 16:28:35.696858 I | rafthttp: peer 936ce633ac273d75 became active
etcd-2 etcd 2021-03-24 16:28:35.697153 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream Message writer)
etcd-2 etcd 2021-03-24 16:28:35.697943 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 writer)
etcd-2 etcd 2021-03-24 16:28:35.717624 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream MsgApp v2 reader)
etcd-2 etcd 2021-03-24 16:28:35.719001 I | rafthttp: established a TCP streaming connection with peer 936ce633ac273d75 (stream Message reader)
etcd-2 etcd 2021-03-24 16:29:05.515600 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:05.513803287 +0000 UTC m=+36.945591671, time spent = 1.723346ms, remote = 10.244.3.57:38848, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-0 etcd 2021-03-24 16:29:06.713013 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:06.711829598 +0000 UTC m=+38.235874683, time spent = 1.113123ms, remote = 10.244.2.56:36784, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-0 etcd 2021-03-24 16:29:12.272006 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:12.269905337 +0000 UTC m=+43.793950408, time spent = 2.035727ms, remote = 10.244.2.56:36818, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-2 etcd 2021-03-24 16:29:15.518793 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:15.516946744 +0000 UTC m=+46.948735193, time spent = 1.80952ms, remote = 10.244.3.57:38990, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-0 etcd 2021-03-24 16:29:22.260282 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:22.259228087 +0000 UTC m=+53.783273213, time spent = 1.000592ms, remote = 10.244.2.56:36868, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-2 etcd 2021-03-24 16:29:25.528702 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:25.527600749 +0000 UTC m=+56.959389098, time spent = 1.069915ms, remote = 10.244.3.57:39070, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-2 etcd 2021-03-24 16:29:32.091200 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:32.090081984 +0000 UTC m=+63.521870419, time spent = 1.081758ms, remote = 10.244.3.57:39180, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-0 etcd 2021-03-24 16:29:32.278530 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:32.277642202 +0000 UTC m=+63.801687231, time spent = 845.103µs, remote = 10.244.2.56:36974, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-2 etcd 2021-03-24 16:29:35.496913 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:35.495687773 +0000 UTC m=+66.927476178, time spent = 1.195862ms, remote = 10.244.3.57:39228, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-0 etcd 2021-03-24 16:29:36.709830 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:36.708764686 +0000 UTC m=+68.232809714, time spent = 1.035126ms, remote = 10.244.2.56:37004, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-0 etcd 2021-03-24 16:29:42.285811 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:42.284335915 +0000 UTC m=+73.808381039, time spent = 1.419721ms, remote = 10.244.2.56:37058, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"
etcd-2 etcd 2021-03-24 16:29:45.514149 D | etcdserver/api/v3rpc: start time = 2021-03-24 16:29:45.512724953 +0000 UTC m=+76.944513353, time spent = 1.38792ms, remote = 10.244.3.57:39404, response type = /etcdserverpb.KV/Range, request count = 0, request size = 8, response count = 0, response size = 28, request content = key:"health"

stale[bot] commented 3 years ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 3 years ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

danfromtitan commented 3 years ago

Still experiencing request cluster ID mismatch with chart ver 6.8.0 after disaster recovery.

Steps to reproduce:

fixed the snapshot cronjob in #7498 to actually write a snapshot file
helm uninstalled the chart (3 PVC's from cluster nodes left behind with data)
left the PVC's untouched, made no changes whatsoever to chart values
installed helm chart with

  --set etcd.auth.rbac.rootPassword=$ETCD_ROOT_PASSWORD \
  --set etcd.initialClusterState=existing

Outcome:

one member node comes back healthy
the other members keep erroring with request cluster ID mismatch

IMO this is caused by run.sh starting the cluster members without following etcd.io's disaster recovery instructions

juan131 commented 3 years ago

Hi @danfromtitan

Are you setting the startFromSnapshot.* parameters? See:

https://github.com/bitnami/charts/tree/master/bitnami/etcd#snapshotting-parameters

danfromtitan commented 3 years ago

Yes, neither disaster recovery or start from snapshot scenarios bring the cluster up healthy, same request cluster ID mismatch outcome in both cases.

AliM113 commented 3 years ago

Hello @juan131 , i am seeing the same behaviour when using the start from snapshot options. One pod is up and ready the 2 others has the cluster ID mismatch error.

juan131 commented 3 years ago

Hi @danfromtitan @AliM113

Did you remove the existing PVCs before installing a new release when using the "start from snapshot" option??

AliM113 commented 3 years ago

Hello @juan131, i started with new PVCs. I will retest and paste the whole stepup and logs. In my scenario i am trying to migrate a cluster installed on 3 VMs to a 3 pods cluster using the restore from snapshot feature.

AliM113 commented 3 years ago

Scenario: ETCD cluter migration from VMs to K8S

steps:

create a snapshot of the existing cluster using etcdctl
Copy the Snapshot to a PVC in RWX access mode (using NFS)
recreate a 3 pods cluster using the bitnami/etcd chart with the deploy from snapshot option

RWX PVC & no old ETCD PVC:

k -n get pvc | grep etcd
etcd-snapshot   Bound    pvc-02fe344e-ea98-4fd6-a767-688b2561b708   5Gi        RWX            nfs-sc         19h

deploy from snapshot setup (values.yaml file):
   startFromSnapshot:
      enabled: true
      existingClaim: "etcd-snapshot"
      snapshotFilename: "etcd-snapshot-20210916_1200.db"

AliM113 commented 3 years ago

ETCD deployment:

k get pvc | grep etcd-ama6     
data-etcd-ama6-0  Bound    pvc-111a564c-9f62-4898-9e0e-be6a5e7991bf   5Gi        RWO            gp2            3m44s
data-etcd-ama6-1  Bound    pvc-dad3d2e8-d31b-4a89-a440-2741c02b67e9   5Gi        RWO            gp2            3m43s
data-etcd-ama6-2  Bound    pvc-92777af0-7d9d-490f-9d63-ef71379f5542   5Gi        RWO            gp2            3m43s

k get po | grep etcd-ama6
etcd-ama6-0                                                       1/2     Running     2          3m27s
etcd-ama6-1                                                       1/2     Running     3          3m27s
etcd-ama6-2                                                       2/2     Running     0          3m27s

One pod is up and ready and the 2 other pods are up but not ready.

AliM113 commented 3 years ago

etcd-ama6-2 pod logs:

{"level":"warn","ts":"2021-09-17T09:03:19.415Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.488Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.488Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.588Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.588Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.687Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.687Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.788Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}
{"level":"warn","ts":"2021-09-17T09:03:19.788Z","caller":"rafthttp/http.go:500","msg":"request cluster ID mismatch","local-member-id":"a448b6ab63edce5","local-member-cluster-id":"630775b8d08ced7b","local-member-server-version":"3.5.0","local-member-server-minimum-cluster-version":"3.0.0","remote-peer-server-name":"3becdd6f96c6fbec","remote-peer-server-version":"3.5.0","remote-peer-server-minimum-cluster-version":"3.0.0","remote-peer-cluster-id":"d05260972b56818b"}

AliM113 commented 3 years ago

etcd-ama6-0 && etcd-ama6-1 pod logs:

{"level":"debug","ts":"2021-09-17T09:05:19.789Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"a448b6ab63edce5","error":"failed to dial a448b6ab63edce5 on stream MsgApp v2 (cluster ID mismatch)"}
{"level":"debug","ts":"2021-09-17T09:05:19.789Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"a448b6ab63edce5","error":"failed to dial a448b6ab63edce5 on stream Message (cluster ID mismatch)"}
{"level":"debug","ts":"2021-09-17T09:05:19.885Z","caller":"rafthttp/stream.go:570","msg":"dial stream reader","from":"3becdd6f96c6fbec","to":"a448b6ab63edce5","address":"http://etcd-ama6-2.etcd-ama6-headless.admin.svc.cluster.local:2380/raft/stream/msgapp/3becdd6f96c6fbec"}
{"level":"debug","ts":"2021-09-17T09:05:19.885Z","caller":"rafthttp/stream.go:570","msg":"dial stream reader","from":"3becdd6f96c6fbec","to":"a448b6ab63edce5","address":"http://etcd-ama6-2.etcd-ama6-headless.admin.svc.cluster.local:2380/raft/stream/message/3becdd6f96c6fbec"}
{"level":"warn","ts":"2021-09-17T09:05:19.888Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"a448b6ab63edce5","remote-peer-cluster-id":"630775b8d08ced7b","local-member-id":"3becdd6f96c6fbec","local-member-cluster-id":"d05260972b56818b","error":"cluster ID mismatch"}
{"level":"debug","ts":"2021-09-17T09:05:19.888Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"a448b6ab63edce5","error":"failed to dial a448b6ab63edce5 on stream Message (cluster ID mismatch)"}
{"level":"warn","ts":"2021-09-17T09:05:19.888Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"a448b6ab63edce5","remote-peer-cluster-id":"630775b8d08ced7b","local-member-id":"3becdd6f96c6fbec","local-member-cluster-id":"d05260972b56818b","error":"cluster ID mismatch"}
{"level":"debug","ts":"2021-09-17T09:05:19.888Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"a448b6ab63edce5","error":"failed to dial a448b6ab63edce5 on stream MsgApp v2 (cluster ID mismatch)"}
{"level":"debug","ts":"2021-09-17T09:05:19.987Z","caller":"rafthttp/stream.go:570","msg":"dial stream reader","from":"3becdd6f96c6fbec","to":"a448b6ab63edce5","address":"http://etcd-ama6-2.etcd-ama6-headless.admin.svc.cluster.local:2380/raft/stream/msgapp/3becdd6f96c6fbec"}
{"level":"debug","ts":"2021-09-17T09:05:19.987Z","caller":"rafthttp/stream.go:570","msg":"dial stream reader","from":"3becdd6f96c6fbec","to":"a448b6ab63edce5","address":"http://etcd-ama6-2.etcd-ama6-headless.admin.svc.cluster.local:2380/raft/stream/message/3becdd6f96c6fbec"}
{"level":"warn","ts":"2021-09-17T09:05:19.990Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"a448b6ab63edce5","remote-peer-cluster-id":"630775b8d08ced7b","local-member-id":"3becdd6f96c6fbec","local-member-cluster-id":"d05260972b56818b","error":"cluster ID mismatch"}
{"level":"debug","ts":"2021-09-17T09:05:19.990Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"a448b6ab63edce5","error":"failed to dial a448b6ab63edce5 on stream Message (cluster ID mismatch)"}
{"level":"warn","ts":"2021-09-17T09:05:19.990Z","caller":"rafthttp/stream.go:653","msg":"request sent was ignored by remote peer due to cluster ID mismatch","remote-peer-id":"a448b6ab63edce5","remote-peer-cluster-id":"630775b8d08ced7b","local-member-id":"3becdd6f96c6fbec","local-member-cluster-id":"d05260972b56818b","error":"cluster ID mismatch"}
{"level":"debug","ts":"2021-09-17T09:05:19.990Z","caller":"rafthttp/peer_status.go:76","msg":"peer deactivated again","peer-id":"a448b6ab63edce5","error":"failed to dial a448b6ab63edce5 on stream MsgApp v2 (cluster ID mismatch)"}

haroonb commented 3 years ago

@AliM113 I struggled with the latest chart for a while and decided, to just use the old one, because I couldn't resolve the issue you are having. See my post above.

I use the following flags for the old char. If you're willing to use the old one you will get it to work with the following commands. substitute your specifics.

To restore the cluster use the following commands.

/snap/bin/helm upgrade --install etcd bitnami/etcd 
--version 5.6.2
--namespace etcd
--set statefulset.replicaCount=3 
--set image.tag=3.4.14-debian-10-r69
--set image.debug=true 
--set metrics.enabled=true 
--set auth.rbac.enabled=false 
--set disasterRecovery.enabled=true 
--set disasterRecovery.cronjob.schedule='*/10 * * * *' 
--set disasterRecovery.pvc.existingClaim=etcd-snapshot-volume 
--set persistence.enabled=true 
--set persistence.storageClass=rook-ceph-block 
--set persistence.size=1Gi 
--set podAntiAffinity=soft 
--set startFromSnapshot.enabled=true 
--set startFromSnapshot.existingClaim=etcd-restore 
--set startFromSnapshot.snapshotFilename={{ etcd_backup_name }}
--set initialClusterState=new

After the cluster is running you have to upgrade the deployment else the pods will always start from the backup on crash/scale/etc

/snap/bin/helm upgrade --install etcd bitnami/etcd 
--version 5.6.2
--namespace etcd
--reuse-values
--set image.tag=3.4.14-debian-10-r69
--set startFromSnapshot.enabled=false 
--set initialClusterState=existing

EDIT: The Issues persist with all the charts/images later 6.0.0, after the config map was integrated into the image itself.

AliM113 commented 3 years ago

@haroonb thank you for your feedback

juan131 commented 3 years ago

Thanks for the detailed explanation @AliM113 !! And thanks so much for sharing this workaround @haroonb !!

One question: are you unable to use the "start from snapshot" feature using 6.x.x chart regardless the snapshot size? Or is this a specific issue that only happens when restoring a big database?

haroonb commented 3 years ago

Hello @juan131, starting from chart 6.x.x the restore of the cluster from snapshot is not possible anymore for me. Like @AliM113 I can restore the cluster from a snapshot, but after that the cluster has only one working db with the other unable to join the cluster, because of the id mismatch. My snapshot is about 20 MB. I tried it with smaller snapshots about 2MB, but not larger than 20 MB.

AliM113 commented 3 years ago

Hello @juan131, the restore is not working whatever is the snapshot size. If you restore a snapshot to a single node cluster it will work, but multi node cluster is failing. For now i am restoring the snapshot to a single node cluster then i am using the Make mirror command to copy data to a multi node cluster.

AliM113 commented 3 years ago

I tried also to restore to a single node cluster then to scale the cluster to 3 nodes (in the below test i scaled to 2 pods to confirm the behaviour is not related to 3 nodes cluster), but this is failing. The bootstrap on the new nodes is trying to reach the first pod on localhost.

here's the logs from the second pod:

{"level":"info","ts":"2021-09-15T09:54:32.917Z","caller":"embed/etcd.go:307","msg":"starting an etcd server","etcd-version":"3.5.0","git-sha":"946a5a6f2","go-version":"go1.16.3","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":false,"name":"etcd-ama6-1","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"etcd-ama6-0=http://localhost:2380,etcd-ama6-1=http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2380","initial-cluster-state":"existing","initial-cluster-token":"etcd-cluster-k8s","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2021-09-15T09:54:32.918Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/bitnami/etcd/data/member/snap/db","took":"151.111µs"}
{"level":"warn","ts":"2021-09-15T09:54:42.919Z","caller":"etcdserver/cluster_util.go:79","msg":"failed to get cluster response","address":"http://localhost:2380/members","error":"Get \"http://localhost:2380/members\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
{"level":"info","ts":"2021-09-15T09:54:42.924Z","caller":"embed/etcd.go:367","msg":"closing etcd server","name":"etcd-ama6-1","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2380"],"advertise-client-urls":["http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2379"]}
{"level":"info","ts":"2021-09-15T09:54:42.924Z","caller":"embed/etcd.go:369","msg":"closed etcd server","name":"etcd-ama6-1","data-dir":"/bitnami/etcd/data","advertise-peer-urls":["http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2380"],"advertise-client-urls":["http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2379"]}
{"level":"fatal","ts":"2021-09-15T09:54:42.924Z","caller":"etcdmain/etcd.go:203","msg":"discovery failed","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/etcd.go:203\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/etcdmain/main.go:40\nmain.main\n\t/tmp/etcd-release-3.5.0/etcd/release/etcd/server/main.go:32\nruntime.main\n\t/home/remote/sbatsche/.gvm/gos/go1.16.3/src/runtime/proc.go:225"}

Please note the initial cluster value configured on pod-1 (trying to reach pod 0 on localhost instead of the headless service)

"initial-cluster":"etcd-ama6-0=http://localhost:2380,etcd-ama6-1=http://etcd-ama6-1.etcd-ama6-headless.admin.svc.cluster.local:2380","initial-cluster-state":"existing"

danfromtitan commented 3 years ago

To restore the cluster use the following commands.

/snap/bin/helm upgrade --install etcd bitnami/etcd 
--version 5.6.2
--namespace etcd
--set statefulset.replicaCount=3 
--set image.tag=3.4.14-debian-10-r69
--set image.debug=true 
--set metrics.enabled=true 
--set auth.rbac.enabled=false 
--set disasterRecovery.enabled=true 
--set disasterRecovery.cronjob.schedule='*/10 * * * *' 
--set disasterRecovery.pvc.existingClaim=etcd-snapshot-volume 
--set persistence.enabled=true 
--set persistence.storageClass=rook-ceph-block 
--set persistence.size=1Gi 
--set podAntiAffinity=soft 
--set startFromSnapshot.enabled=true 
--set startFromSnapshot.existingClaim=etcd-restore 
--set startFromSnapshot.snapshotFilename={{ etcd_backup_name }}
--set initialClusterState=new

After the cluster is running you have to upgrade the deployment else the pods will always start from the backup on crash/scale/etc

/snap/bin/helm upgrade --install etcd bitnami/etcd 
--version 5.6.2
--namespace etcd
--reuse-values
--set image.tag=3.4.14-debian-10-r69
--set startFromSnapshot.enabled=false 
--set initialClusterState=existing

EDIT: The Issues persist with all the charts/images later 6.0.0, after the config map was integrated into the image itself.

The latest chart from 5.x.x works indeed to restore from snapshot, but when I upgrade (to disable startFromSnapshot) the first cluster member that was re-created keeps looping with "rpc error: code = Unauthenticated desc = etcdserver: invalid auth token"

In my case I have RBAC and TLS enabled. There were some invalid auth token issue fixed in older etcd-io versions, so I tried building etcd-io ver 3.5.0-3 on top of 3.4.14-debian-10-r69 (i.e. using same image scripts as the old 3.4.14-debian-10-r69, only getting a newer etcd tarball deployed with the container image), it still loops with the same error.

They say simple auth token is for testing only and it's statefull, which explains why the re-created member complains about invalid token. Instead JWT tokens should be used in the chart, that's something @juan131 should consider moving towards in the newer 6.x.x.

danfromtitan commented 3 years ago

Instead JWT tokens should be used in the chart

Based on these findings I went on to add ETCD_AUTH_TOKEN to chart version 5.6.2 using an envVarsConfigMap since there is no other place to add extra env vars in this older chart.

Practically I re-used the TLS certificate private key for the JWT token and created the public portion of the same key next to it. It could be done better, but the old chart doesn't have a placeholder for extra volume mounts, so I re-used etcd-client-certs by adding the public key in the same secret. The auth-token in the logs looks like:

2021-09-21 05:20:03.965246 I | pkg/flags: recognized and used environment variable ETCD_AUTH_TOKEN=jwt,pub-key=/opt/bitnami/etcd/certs/client/key.pub,priv-key=/opt/bitnami/etcd/certs/client/key.pem,sign-method=RS256,ttl=10m

The outcome is now I can restore from snapshot, restore for disaster recovery and scale the cluster up & down without any of the instability issues I've been chasing in the last couple of weeks.

Sometimes, when I scale down, the member removed is not cleaned-up from the cluster quorum and the restarting node would fail to add itself to the cluster:

etcd-0                                1/1     Running       0          15m
etcd-1                                1/1     Running       0          16m
etcd-2                                0/1     Init:0/1      0          7s        # this node is roll restarted because env vars changes
etcd-3                                0/1     Terminating   1          19m       # this node gets removed at scale down

etcd-0                                1/1     Running            0          18m
etcd-1                                1/1     Running            0          19m
etcd-2                                0/1     CrashLoopBackOff   4          3m10s    

# etcd-2 keeps crashing because etcd-3 hasn't been properly removed and cluster is unhealthy
# etcd-3 should not be in the member list below 

211818ac116133c9, started, etcd-0, https://etcd-0.etcd-headless.kv.svc.cluster.local:2380, https://etcd-0.etcd-headless.kv.svc.cluster.local:2379, false
7aa790871456d576, started, etcd-3, https://etcd-3.etcd-headless.kv.svc.cluster.local:2380, https://etcd-3.etcd-headless.kv.svc.cluster.local:2379, false
fceb89c95ed3d045, started, etcd-1, https://etcd-1.etcd-headless.kv.svc.cluster.local:2380, https://etcd-1.etcd-headless.kv.svc.cluster.local:2379, false

etcdctl member remove from one of the healthy members would help resolve the issue and allow the restarting member to reach a READY state.

etcd-0                                1/1     Running     0          7m32s
etcd-1                                1/1     Running     0          8m50s
etcd-2                                1/1     Running     5          13m    # notice the 5 times restart while I was looking into removing the stall member

juan131 commented 3 years ago

Thanks for the analysis @danfromtitan !

We should definitely use JWT tokens, and the key is here:

JWT is stateless (...) include metadata including username and revision, so servers don’t need to remember correspondence between tokens and the metadata.

Otherwise, when enabling TLS, the members won't be able to rejoin the cluster since they'll try to reuse the wrong token:

rpc error: code = Unauthenticated desc = etcdserver: invalid auth token

Practically I re-used the TLS certificate private key for the JWT token and created the public portion of the same key next to it. It could be done better, but the old chart doesn't have a placeholder for extra volume mounts (...)

Would you like to submit a PR adding support to mount a custom cert file and key for the JWT token?

By the way, I have on question @danfromtitan : are you using auth.peer.useAutoTLS=true (which basically set ETCD_PEER_AUTO_TLS=true? I'd like to understand if the issue also appears in this case.

the restore is not working whatever is the snapshot size. If you restore a snapshot to a single node cluster it will work, but multi node cluster is failing.

Thanks @AliM113 !! I'll work on figuring out what's going on with 6.x.x and make this feature available again.

JFYI, I'm currently reviewing a contribution to reduce the chances for a race condition to appear when scaling up/down the solution. After this is solved, I'll start looking into this precise issue.

danfromtitan commented 3 years ago

are you using auth.peer.useAutoTLS=true

No, that's because client TLS auth is not possible with self-signed certs. I tried it nevertheless and I can't get anywhere close to where I was before, nodes won't come ready because the health probe fails with a cert error. That's with chart ver 5.6.2:

==> [DEBUG] Probe command: "etcdctl --user root:*****--cert /bitnami/etcd/data/fixtures/client/cert.pem --key /bitnami/etcd/data/fixtures/client/key.pem endpoint health"
{"level":"warn","ts":"2021-09-21T15:09:46.147Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-07ea42a1-95ab-4c68-a0d8-4855416567ac/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate is valid for 0.0.0.0, not 127.0.0.1\""}
127.0.0.1:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster

Regardless of that, IMO the JWT auth token is a client session token and it is not related to client or peer TLS auth functionality.

Would you like to submit a PR adding support to mount a custom cert file and key for the JWT token?

I could do that, yes, providing instability issues are sorted out with chart ver 6.x.x. There are so may issues I encountered there, I can't possibly test the JWT token works properly if I make this change with what's there now.

Speaking of the changes in 6.x.x, I find it odd that I didn’t encounter any of those issues in 5.6.2. I see the scripts in the older chart use the same idea of a member_id, so I'm not sure how it got in such a bad shape. One thing is for sure, you'd need to add a testing integration framework around the charts to cover the functionality supported.

juan131 commented 3 years ago

IMO the JWT auth token is a client session token and it is not related to client or peer TLS auth functionality.

You're 100% right

I could do that, yes, providing instability issues are sorted out with chart ver 6.x.x. There are so many issues I encountered there, I can't possibly test the JWT token works properly if I make this change with what's there now.

I can try to help there.. Let's follow the work started at https://github.com/bitnami/bitnami-docker-etcd/pull/21 and make the current version compatible with this feature again.

Speaking of the changes in 6.x.x, I find it odd that I didn’t encounter any of those issues in 5.6.2. I see the scripts in the older chart use the same idea of a member_id, so I'm not sure how it got in such a bad shape

Correct! When we moved the logic to the image itself, we didn't change this approach. We simply removed technical debt, but it's obvious that we broke things during that process.

One thing is for sure, you'd need to add a testing integration framework around the charts to cover the functionality supported.

:+1:

We have internal tests that install the chart on different K8s platforms (e.g. GKE, AKS, TKG, etc.) using different scenarios where we use different values combinations, run upgrade tests, etc. However, there's a lot of room for improvement in our current scenarios because there are many combis that escape our tests.

juv commented 3 years ago

I'm wondering if there is a chart version that should be used instead of the 6.x.x versions until this issue is resolved? Can someone suggest a stable version?

danfromtitan commented 2 years ago

I'm wondering if there is a chart version that should be used instead of the 6.x.x versions until this issue is resolved? Can someone suggest a stable version?

Please see https://github.com/bitnami/charts/issues/5771#issuecomment-921635557 and follow-up findings in https://github.com/bitnami/charts/issues/5771#issuecomment-923286910

juv commented 2 years ago

I'm wondering if there is a chart version that should be used instead of the 6.x.x versions until this issue is resolved? Can someone suggest a stable version?

Please see #5771 (comment) and follow-up findings in #5771 (comment)

Thanks, looks like this version is working better for me as well!

abdennour commented 2 years ago

still issue persist with chart etcd v6.10.2. I am not able to scale the cluster out from 1 to 3 I am getting:

"msg":"discovery failed","error":"cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs",

juan131 commented 2 years ago

@abdennour please provide the following information:

Did you enable persistence in the solution?
Did you scale the solution using helm upgrade ... --set replicaCount=3 or kubectl scale sts ... --replicas=3?

abdennour commented 2 years ago

Yes @juan131 ! I did it's helm upgrade. I am aware of what you want to reach. You want to say that scaling out is not only increasing the number of replicas. Thus, conducting it with helm is recommended.

juan131 commented 2 years ago

Hi @abdennour

Both ways of scaling the chart are supported. However, it's important for us to reproduce the issue to know the exact steps you're following. That's why I requested the information about the persistence and the upgrade method you used.

abdennour commented 2 years ago

I see. I will check it

bitnami / charts

[bitnami/etcd] - New Deployments Not Clustering in v6.0.0 #5771