search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.85k stars 9.77k forks source link

Unable to start ETCD and failing with error 'tls: failed to find any PEM data in key input' #16021

Closed IamSatyaonline closed 1 year ago

IamSatyaonline commented 1 year ago

Bug report criteria

What happened?

We have 3 members cluster of ETCD service. ETCD service is not running and throwing the error 'tls: failed to find any PEM data in key input'. As per our prilimarliy investigation , Key files which is being used for peer communication which might be corrupted. But we are not sure about the root cause. We are using the ETCD-3.5.7 and self sign certificates are being used for peer communication. Could you please help us to know the exact root cause of the issue. Sharing below the log snippet with the error coming in the logs.

2023-06-02T14:58:00.721849815Z {"caller":"embed/etcd.go:484","cipher-suites":[],"message":"starting with peer TLS","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:58:00.721+00:00","tls-info":"cert = /data/fixtures/peer/cert.pem, key = /data/fixtures/peer/key.pem, client-cert=/data/fixtures/peer/cert.pem, client-key=/data/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = ","version":"1.2.0"} 2023-06-02T14:58:00.721912667Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:373","data-dir":"/data","message":"closing etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:58:00.721+00:00","version":"1.2.0"} 2023-06-02T14:58:00.721960692Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:375","data-dir":"/data","message":"closed etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:58:00.721+00:00","version":"1.2.0"}

2023-06-02T14:58:00.722031174Z {"caller":"etcdmain/etcd.go:204","error":"tls: failed to find any PEM data in key input","message":"discovery failed","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"critical","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:32\nruntime.main\n\truntime/proc.go:255","timestamp":"2023-06-02T14:58:00.721+00:00","version":"1.2.0"}

What did you expect to happen?

We want ETCD up and running without any error.

How can we reproduce it (as minimally and precisely as possible)?

It's kind of intermittent and not reproducible. always

Anything else we need to know?

No response

Etcd version (please run commands below)

```console $ etcd --version bash-4.4$ etcd --version etcd Version: 3.5.7 Git SHA: 215b53cf3 Go Version: go1.17.13 Go OS/Arch: linux/amd64 $ etcdctl version bash-4.4$ etcd --version etcd Version: 3.5.7 Git SHA: 215b53cf3 Go Version: go1.17.13 Go OS/Arch: linux/amd64 bash-4.4$ etcdctl version etcdctl version: 3.5.7 API version: 3.5 ```

Etcd configuration (command line flags or environment variables)

# paste your configuration here bash-4.4$ env ETCD_INITIAL_CLUSTER_TOKEN=etcd TLS_ENABLED=true ETCD_MAX_SNAPSHOTS=3 CLIENT_PORTS=2379 TZ=UTC HOSTNAME=etcd-0 COMPONENT_VERSION=v3.5.7 HTTP_PROBE_CMD_DIR=/usr/local/bin/health HTTP_PROBE_READINESS_CMD_TIMEOUT_SEC=15 ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379 ETCD_HEARTBEAT_INTERVAL=100 ETCD_AUTO_COMPACTION_RETENTION=100 DISARM_ALARM_PEER_INTERVAL=6 ETCD_TRUSTED_CA_FILE=/data/combinedca/cacertbundle.pem DB_THRESHOLD_PERCENTAGE=70 MONITOR_ALARM_INTERVAL=5 PEER_CERT_AUTH_ENABLED=true TRUSTED_CA=/data/combinedca/cacertbundle.pem PEER_CLIENTS_CERTS=/run/sec/certs/peer/srvcert.pem FIFO_DIR=/fifo KUBERNETES_PORT_443_TCP_PROTO=tcp ENTRYPOINT_RESTART_ETCD=true HTTP_PROBE_NAMESPACE=ztissan KUBERNETES_PORT_443_TCP_ADDR= ETCDCTL_CERT=/run/sec/certs/client/clicert.pem ENTRYPOINT_DCED_PROCESS_INTERVAL=5 ETCD_LOG_LEVEL=debug ENTRYPOINT_CHECKSNUMBER=60 8889 KUBERNETES_PORT=tcp://:443 POD_NAME=etcd-0 ERIC_PM_SERVER_PORT_9089_TCP_PORT=9089 ERIC_ETCD_SERVICE_PORT=2379 PWD=/ ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380 HOME=/home/dced ERIC_ETCD_SERVICE_PORT_CLIENT_PORT_TLS=2379 ETCD_AUTO_COMPACTION_MODE=revision KUBERNETES_SERVICE_PORT_HTTPS=443 ERIC_ETCD_PORT_2379_TCP_ADDR= KUBERNETES_PORT_443_TCP_PORT=443 ETCD_LOGGER=zap PEER_AUTO_TLS_ENABLED=true ETCD_CERT_FILE=/run/sec/certs/server/srvcert.pem ETCD_PEER_AUTO_TLS=true ERIC_ETCD_PORT_2379_TCP_PORT=2379 KUBERNETES_PORT_443_TCP=tcp://:443 ERIC_ETCD_PORT_2379_TCP=tcp://:2379 LISTEN_PEER_URLS=https://0.0.0.0:2380 DEFRAGMENT_PERIODIC_INTERVAL=60 CONTAINER_NAME=etcd COMPONENT=etcd ETCD_DATA_DIR=/data ETCD_CLIENT_CERT_AUTH=true TERM=xterm ETCDCTL_ENDPOINTS=etcd.ztissan:2379 HTTP_PROBE_LIVENESS_CMD_TIMEOUT_SEC=15 ETCD_METRICS=basic PEER_CLIENT_KEY_FILE=/run/sec/certs/peer/srvprivkey.pem HTTP_PROBE_CONTAINER_NAME=etcd GODEBUG=tls13=1 ETCDCTL_API=3 ERIC_ETCD_PORT=tcp://:2379 ETCD_SNAPSHOT_COUNT=5000 ETCD_MAX_WALS=3 SHLVL=1 HTTP_PROBE_POD_NAME=etcd-0 KUBERNETES_SERVICE_PORT=443 ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-0.etcd-peer.ztissan.svc.cluster.local:2380 HTTP_PROBE_STARTUP_CMD_TIMEOUT_SEC=15 ETCD_KEY_FILE=/run/sec/certs/server/srvprivkey.pem ETCD_ELECTION_TIMEOUT=1000 HTTP_PROBE_SERVICE_NAME=etcd ETCDCTL_CACERT=/data/combinedca/cacertbundle.pem ETCD_NAME=etcd-0 ETCD_QUOTA_BACKEND_BYTES=268435456 ENTRYPOINT_PIPE_TIMEOUT=5 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ETCD_ADVERTISE_CLIENT_URLS=https://etcd-0.etcd.ztissan:2379 DCED_PORT=2379 KUBERNETES_SERVICE_HOST= FLAVOUR=etcd-v3.5.7-linux-amd64 ETCDCTL_KEY=/run/sec/certs/client/cliprivkey.pem _=/usr/bin/env

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

```console $ etcdctl member list -w table # paste output here $ etcdctl --endpoints= endpoint status -w table # paste output here ```

Relevant log output

2023-06-02T14:57:40.311883089Z {"message":"The current timezone is UTC","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.311+00:00","version":"1.2.0"}
2023-06-02T14:57:40.319907939Z {"message":"Startup command: /usr/local/bin/health/StartupProbe.sh","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.319+00:00","version":"1.2.0"}
2023-06-02T14:57:40.319935154Z {"message":"Startup commmand timeout value 15 seconds","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.319+00:00","version":"1.2.0"}
2023-06-02T14:57:40.320022235Z {"message":"Readiness command: /usr/local/bin/health/ReadinessProbe.sh","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.319+00:00","version":"1.2.0"}
2023-06-02T14:57:40.320095449Z {"message":"Readiness commmand timeout value 15 seconds","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"}
2023-06-02T14:57:40.320247443Z {"message":"Liveness command: /usr/local/bin/health/LivenessProbe.sh","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"}
2023-06-02T14:57:40.320326277Z {"message":"Liveness commmand timeout value 15 seconds","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"}
2023-06-02T14:57:40.320415316Z {"message":"The http probe service 1.5.0 is ready to listen and serve on :9000","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346234288Z {"message":"+ getopts cs option","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346314039Z {"message":"+ case \"${option}\" in","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346522082Z {"message":"+ CHECK=true","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346568194Z {"message":"+ getopts cs option","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346611005Z {"message":"+ case \"${option}\" in","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346647224Z {"message":"+ SUICIDE=true","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346688012Z {"message":"+ getopts cs option","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.346723100Z {"message":"+ mkdir -p /data/combinedca/","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"}
2023-06-02T14:57:40.425138435Z {"message":"+ [[ -v SUICIDE ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.424+00:00","version":"1.2.0"}
2023-06-02T14:57:40.425604259Z {"message":"++ ps -ef","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.425+00:00","version":"1.2.0"}
2023-06-02T14:57:40.425653943Z {"message":"++ grep watch_cert","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.425+00:00","version":"1.2.0"}
2023-06-02T14:57:40.425805950Z {"message":"++ grep -v grep","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.425+00:00","version":"1.2.0"}
2023-06-02T14:57:40.429301486Z {"message":"+ [[ -z '' ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.429+00:00","version":"1.2.0"}
2023-06-02T14:57:40.429517679Z {"message":"+ rm -f /data/combinedca/cacertbundle.pem","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.429+00:00","version":"1.2.0"}
2023-06-02T14:57:40.429618524Z {"message":"+ nohup /usr/local/bin/scripts/watch_cert.sh /run/sec/cas/clientca /run/sec/cas/pmca /run/sec/cas/siptlsca","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.429+00:00","version":"1.2.0"}
2023-06-02T14:57:40.430713026Z {"message":"+ [[ ! -s /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.430+00:00","version":"1.2.0"}
2023-06-02T14:57:40.430801734Z {"message":"+ awk 1 /run/sec/cas/clientca/client-cacertbundle.pem /run/sec/cas/siptlsca/cacertbundle.pem","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.430+00:00","version":"1.2.0"}
2023-06-02T14:57:40.432971688Z {"message":"+ [[ ! -s /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.432+00:00","version":"1.2.0"}
2023-06-02T14:57:40.433048788Z {"message":"+ [[ ! -s /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.432+00:00","version":"1.2.0"}
2023-06-02T14:57:40.433118466Z {"message":"+ [[ -v CHECK ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"}
2023-06-02T14:57:40.433222071Z {"message":"+ [[ ! -f /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"}
2023-06-02T14:57:40.433295675Z {"message":"Setup peer certs for etcd-0 ","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"}
2023-06-02T14:57:40.433391709Z {"message":"Auto TLS enable , using ETCD auto generated certs for peer communication","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"}
2023-06-02T14:57:40.532124693Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.531+00:00","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"https://etcd-0.etcd.spider3:2379","version":"1.2.0"}
2023-06-02T14:57:40.532173064Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision","version":"1.2.0"}
2023-06-02T14:57:40.532230541Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"100","version":"1.2.0"}
2023-06-02T14:57:40.532290393Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_CERT_FILE","variable-value":"/run/sec/certs/server/srvcert.pem","version":"1.2.0"}
2023-06-02T14:57:40.532348883Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"true","version":"1.2.0"}
2023-06-02T14:57:40.532423774Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_DATA_DIR","variable-value":"/data","version":"1.2.0"}
2023-06-02T14:57:40.532461823Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"1000","version":"1.2.0"}
2023-06-02T14:57:40.532524552Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"100","version":"1.2.0"}
2023-06-02T14:57:40.532579293Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380","version":"1.2.0"}
2023-06-02T14:57:40.532623744Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"etcd-0=https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380","version":"1.2.0"}
2023-06-02T14:57:40.532681218Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new","version":"1.2.0"}
2023-06-02T14:57:40.532731491Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd","version":"1.2.0"}
2023-06-02T14:57:40.532780027Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_KEY_FILE","variable-value":"/run/sec/certs/server/srvprivkey.pem","version":"1.2.0"}
2023-06-02T14:57:40.532836995Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"https://0.0.0.0:2379","version":"1.2.0"}
2023-06-02T14:57:40.532902896Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"https://0.0.0.0:2380","version":"1.2.0"}
2023-06-02T14:57:40.532956146Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LOG_LEVEL","variable-value":"info","version":"1.2.0"}
2023-06-02T14:57:40.533002550Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LOGGER","variable-value":"zap","version":"1.2.0"}
2023-06-02T14:57:40.533065561Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_MAX_SNAPSHOTS","variable-value":"3","version":"1.2.0"}
2023-06-02T14:57:40.533109094Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_MAX_WALS","variable-value":"3","version":"1.2.0"}
2023-06-02T14:57:40.533156961Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_METRICS","variable-value":"basic","version":"1.2.0"}
2023-06-02T14:57:40.533206629Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_NAME","variable-value":"etcd-0","version":"1.2.0"}
2023-06-02T14:57:40.533252146Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"true","version":"1.2.0"}
2023-06-02T14:57:40.533316465Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"268435456","version":"1.2.0"}
2023-06-02T14:57:40.533377769Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_SNAPSHOT_COUNT","variable-value":"5000","version":"1.2.0"}
2023-06-02T14:57:40.533414350Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_TRUSTED_CA_FILE","variable-value":"/data/combinedca/cacertbundle.pem","version":"1.2.0"}
2023-06-02T14:57:40.533524832Z {"args":["/usr/local/bin/etcd"],"caller":"etcdmain/etcd.go:73","message":"Running: ","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"}
2023-06-02T14:57:40.533607997Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"auth_successful","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"}
2023-06-02T14:57:40.533677042Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"cert_watcher.txt","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"}
2023-06-02T14:57:40.533825720Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"combinedca","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"}
2023-06-02T14:57:40.533895896Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"etcd.liveness","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"}
2023-06-02T14:57:40.534042861Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"fixtures","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"}
2023-06-02T14:57:40.534098248Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"lost+found","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"}
2023-06-02T14:57:40.534163014Z {"caller":"etcdmain/etcd.go:116","data-dir":"/data","dir-type":"member","message":"server has been already initialized","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"}
2023-06-02T14:57:40.534209953Z {"caller":"embed/etcd.go:124","listen-peer-urls":["https://0.0.0.0:2380"],"message":"configuring peer listeners","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"}
2023-06-02T14:57:40.534339542Z {"caller":"embed/etcd.go:484","cipher-suites":[],"message":"starting with peer TLS","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","tls-info":"cert = /data/fixtures/peer/cert.pem, key = /data/fixtures/peer/key.pem, client-cert=/data/fixtures/peer/cert.pem, client-key=/data/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = ","version":"1.2.0"}
2023-06-02T14:57:40.534429623Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:373","data-dir":"/data","message":"closing etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"}
2023-06-02T14:57:40.534508748Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:375","data-dir":"/data","message":"closed etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"}
2023-06-02T14:57:40.534567414Z {"caller":"etcdmain/etcd.go:204","error":"tls: failed to find any PEM data in key input","message":"discovery failed","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"critical","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:32\nruntime.main\n\truntime/proc.go:255","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"}
IamSatyaonline commented 1 year ago

Hi @ahrtr ETCD service is not getting up and failing with error 'tls: failed to find any PEM data in key input'. Could you please help us here to resolve this issue. It's intermittent issue.

Thanks and Regards, Satya

tjungblu commented 1 year ago

failed to find any PEM data in key input

I'd recommend that you check that your PEM files all can be properly loaded. You can also use openssl for that, it should give similar error messages.

jmhbnz commented 1 year ago

Hi @IamSatyaonline - Thank you for your question, this support issue will be moved to our Discussion Forums.

We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases.

We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on GitHub is making it difficult for us to use issues to identify real bugs.

Members of the etcd community use Discussion Forums to field support requests. Before posting a new question, please search these for answers to similar questions, and also familiarize yourself with:

  1. user documentation
  2. frequently asked questions

Again, thanks for using etcd and raising this question.

The etcd team