search

apache / apisix-helm-chart

Apache APISIX Helm Chart
https://apisix.apache.org/
Apache License 2.0
231 stars 209 forks source link

request help: etcd-0 CrashLoopBackOff #414

Open SeibertronSS opened 1 year ago

SeibertronSS commented 1 year ago

Issue description

I installed apisix-ingress using helm, but the status of pod etcd-0 is always Completed or CrashLoopBackOff. At present, the functions of apisix can be used normally. What caused the error to occur ?

 NAME↑                                           PF     READY          RESTARTS STATUS                      CPU      MEM      %CPU/R      %CPU/L      %MEM/R      %MEM/L IP                NODE              AGE        │
│ apisix-dbd9d7c57-659f6                          ●      1/1                   0 Running                       0        0         n/a         n/a         n/a         n/a xx.xx.x.xxx       xx.xx.x.xxx       13d        │
│ apisix-etcd-0                                   ●      0/1                   0 CrashLoopBackOff              0        0         n/a         n/a         n/a         n/a xx.xx.xx.xx       xx.xx.x.xxx       11m        │
│ apisix-etcd-1                                   ●      1/1                   0 Running                       0        0         n/a         n/a         n/a         n/a xx.xx.xx.xx       xx.xx.x.xxx       12m        │
│ apisix-etcd-2                                   ●      1/1                   0 Running                       0        0         n/a         n/a         n/a         n/a xx.xx.xx.xx       xx.xx.x.xxx       14m        │
│ apisix-ingress-controller-6c694bcdc5-f97fz      ●      1/1                   0 Running                       0        0         n/a         n/a         n/a         n/a xx.xx.x.xxx       xx.xx.x.xxx       13d

The log of etcd-0 is

etcd 07:57:55.03
etcd 07:57:55.03 Welcome to the Bitnami etcd container
etcd 07:57:55.03 Subscribe to project updates by watching https://github.com/bitnami/containers
etcd 07:57:55.03 Submit issues and feature requests at https://github.com/bitnami/containers/issues
etcd 07:57:55.03
etcd 07:57:55.04 INFO  ==> ** Starting etcd setup **
etcd 07:57:55.05 INFO  ==> Validating settings in ETCD_* env vars..
etcd 07:57:55.05 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
etcd 07:57:55.05 INFO  ==> Initializing etcd
etcd 07:57:55.05 INFO  ==> Generating etcd config file using env variables
etcd 07:57:55.07 INFO  ==> Detected data from previous deployments
etcd 07:58:00.73 INFO  ==> Member ID wasn't properly stored, the member will try to join the cluster by it's own
etcd 07:58:00.73 INFO  ==> ** etcd setup finished! **

etcd 07:58:00.75 INFO  ==> ** Starting etcd **
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTH_TOKEN","variable-value":"jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_TLS","variable-value":"false"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"false"}
{"level":"info","ts":"2022-12-12T07:58:00.774Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/bitnami/etcd/data"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"existing"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd-cluster-k8s"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"http://0.0.0.0:2379"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"http://0.0.0.0:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LOG_LEVEL","variable-value":"info"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"apisix-etcd-0"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"false"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TRUSTED_CA_FILE="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_STORE_MEMBER_ID=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_FILE=/opt/bitnami/etcd/conf/etcd.yaml"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ON_K8S=yes"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_SNAPSHOTS_DIR=/snapshots"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BIN_DIR=/opt/bitnami/etcd/bin"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_VOLUME_DIR=/bitnami/etcd"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_ROOT_PASSWORD="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CLUSTER_DOMAIN=apisix-etcd-headless.ingress-apisix.svc.cluster"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISASTER_RECOVERY=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_KEY_FILE="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CONF_DIR=/opt/bitnami/etcd/conf"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_GROUP=etcd"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_START_FROM_SNAPSHOT=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOT_FILENAME="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DISABLE_PRESTOP=no"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_TMP_DIR=/opt/bitnami/etcd/tmp"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_BASE_DIR=/opt/bitnami/etcd"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_CERT_FILE="}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs"}
{"level":"warn","ts":"2022-12-12T07:58:00.775Z","caller":"flags/flag.go:93","msg":"unrecognized environment variable","environment-variable":"ETCD_DAEMON_USER=etcd"}
{"level":"info","ts":"2022-12-12T07:58:00.775Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd"]}
{"level":"warn","ts":"2022-12-12T07:58:00.776Z","caller":"etcdmain/etcd.go:446","msg":"found invalid file under data directory","filename":"new_member_envs","data-dir":"/bitnami/etcd/data"}
{"level":"info","ts":"2022-12-12T07:58:00.776Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/bitnami/etcd/data","dir-type":"member"}
{"level":"info","ts":"2022-12-12T07:58:00.776Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["http://0.0.0.0:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.776Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["http://0.0.0.0:2379"]}
{"level":"info","ts":"2022-12-12T07:58:00.784Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"08407ff76","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":32,"max-cpu-available":32,"member-initialized":true,"name":"apisix-etcd-0","data-dir":"/bitnami/etcd/data","wal-dir":"","wal-dir-dedicated":"","member-dir":"/bitnami/etcd/data/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"existing","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2022-12-12T07:58:00.811Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/bitnami/etcd/data/member/snap/db","took":"5.213367ms"}
{"level":"info","ts":"2022-12-12T07:58:00.833Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","commit-index":3}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=()"}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f became follower at term 67"}
{"level":"info","ts":"2022-12-12T07:58:00.842Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft e1719b29f8ef586f [peers: [], term: 67, commit: 3, applied: 0, lastindex: 3, lastterm: 1]"}
{"level":"info","ts":"2022-12-12T07:58:00.858Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2022-12-12T07:58:00.862Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2022-12-12T07:58:00.867Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"e1719b29f8ef586f","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2022-12-12T07:58:00.867Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2022-12-12T07:58:00.868Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=(11966697873334980856)"}
{"level":"info","ts":"2022-12-12T07:58:00.869Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","added-peer-id":"a6124050428dd0f8","added-peer-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.869Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.869Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"e1719b29f8ef586f","initial-advertise-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"],"listen-peer-urls":["http://0.0.0.0:2380"],"advertise-client-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379","http://apisix-etcd.ingress-apisix.svc.cluster.local:2379"],"listen-client-urls":["http://0.0.0.0:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"[::]:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"[::]:2380"}
{"level":"info","ts":"2022-12-12T07:58:00.870Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.871Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8","remote-peer-urls":["http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:426","msg":"ignored streaming request; ID mismatch","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-header":"e07f9c220823b232","remote-peer-id-from":"a6124050428dd0f8","cluster-id":"7018e2742d24ba53"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84","remote-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-from":"de632cc7e8a00e84","cluster-id":"7018e2742d24ba53"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:426","msg":"ignored streaming request; ID mismatch","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-header":"e07f9c220823b232","remote-peer-id-from":"a6124050428dd0f8","cluster-id":"7018e2742d24ba53"}
{"level":"warn","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"e1719b29f8ef586f","remote-peer-id-stream-handler":"e1719b29f8ef586f","remote-peer-id-from":"de632cc7e8a00e84","cluster-id":"7018e2742d24ba53"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=(11966697873334980856 16024701136227274372)"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: e1719b29f8ef586f elected leader a6124050428dd0f8 at term 67"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","added-peer-id":"de632cc7e8a00e84","added-peer-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.872Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84","remote-peer-urls":["http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"e1719b29f8ef586f switched to configuration voters=(11966697873334980856 16024701136227274372 16244935935474489455)"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.873Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"7018e2742d24ba53","local-member-id":"e1719b29f8ef586f","added-peer-id":"e1719b29f8ef586f","added-peer-peer-urls":["http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380"]}
{"level":"warn","ts":"2022-12-12T07:58:00.875Z","caller":"etcdserver/server.go:1150","msg":"server error","error":"the member has been permanently removed from the cluster"}
{"level":"warn","ts":"2022-12-12T07:58:00.875Z","caller":"etcdserver/server.go:1151","msg":"data-dir used by this member must be removed"}
{"level":"warn","ts":"2022-12-12T07:58:00.875Z","caller":"etcdserver/server.go:2053","msg":"stopped publish because server is stopped","local-member-id":"e1719b29f8ef586f","local-member-attributes":"{Name:apisix-etcd-0 ClientURLs:[http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379 http://apisix-etcd.ingress-apisix.svc.cluster.local:2379]}","publish-timeout":"7s","error":"etcdserver: server stopped"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"a6124050428dd0f8"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"e1719b29f8ef586f","remote-peer-id":"de632cc7e8a00e84"}
{"level":"info","ts":"2022-12-12T07:58:00.875Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"de632cc7e8a00e84"}
{"level":"warn","ts":"2022-12-12T07:58:00.884Z","caller":"rafthttp/http.go:145","msg":"failed to process Raft message","local-member-id":"e1719b29f8ef586f","error":"raft: stopped"}
{"level":"info","ts":"2022-12-12T07:58:00.886Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2022-12-12T07:58:00.886Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}

Environment

niuguy commented 1 year ago

I got a similar issue, etcd keeps crashing, this is my error log 

2022-12-14 05:04:04 | etcd 05:04:04.00
-- | --
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.00 Welcome to the Bitnami etcd container
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.00 Subscribe to project updates by watching https://github.com/bitnami/containers
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.00 Submit issues and feature requests at https://github.com/bitnami/containers/issues
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.00
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.01 INFO  ==> ** Starting etcd setup **
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.02 INFO  ==> Validating settings in ETCD_* env vars..
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.02 WARN  ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment.
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.03 INFO  ==> Initializing etcd
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.03 INFO  ==> Generating etcd config file using env variables
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.04 INFO  ==> Detected data from previous deployments
  |   | 2022-12-14 05:04:04 | etcd 05:04:04.16 INFO  ==> Updating member in existing cluster
  |   | 2022-12-14 05:04:04 | {"level":"warn","ts":"2022-12-14T05:04:04.211Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00031ca80/apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
  |   | 2022-12-14 05:04:04 | Error: etcdserver: member not found

I was able to make it work by first deleting the PersistenceVolumeClaim of the failed etcd pod, then restarting the pod. But I don't know why it worked and failed previously. If someone can help explain 🙏

tao12345666333 commented 1 year ago

you could search in issue first https://github.com/apache/apisix-helm-chart/issues?q=is%3Aissue+is%3Aopen+CrashLoopBackOff