search

rancher / rke

Rancher Kubernetes Engine (RKE), an extremely simple, lightning fast Kubernetes distribution that runs entirely within containers.
Apache License 2.0
3.22k stars 582 forks source link

rke up - runs into untrusted CA issues with etcd healthcheck #3022

Closed sidvijay1989 closed 2 years ago

sidvijay1989 commented 2 years ago

RKE version:currently on 1.2.2, attempting to move 1.3.12

Docker version: (docker version,docker info preferred) Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Docker Buildx (Docker Inc., v0.8.2-docker) scan: Docker Scan (Docker Inc., v0.17.0)

Server: Containers: 76 Running: 42 Paused: 0 Stopped: 34 Images: 445 Server Version: 20.10.17 Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb runc version: v1.1.3-0-g6724737 init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 3.10.0-1160.71.1.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 11.58GiB Name: xxx.yyy.com ID: ACZY:UFXW:3EMM:Y47L:L376:XJYN:3XTD:QXM7:6L42:O3XL:EUJU:IIIP Docker Root Dir: /data/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled WARNING: bridge-nf-call-ip6tables is disabled

Operating system and kernel: (cat /etc/os-release, uname -r preferred) NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) Baremetal

cluster.yml file:

Steps to Reproduce:

we upgraded the rancher from 2.5.3 to 2.6.6 and currently updating kubernetes.

  1. copied the new version of rke 1.3.12
  2. ran the command -> rke up --config rancher-cluster.yaml

Results:

ran into the issue -

INFO[0038] [etcd] Successfully started etcd plane.. Checking etcd cluster health WARN[0134] [etcd] host [161.211.a.b] failed to check etcd health: failed to get /health for host [161.211.a.b]: Get "https://161.211.a.b:2379/health": remote error: tls: bad certificate WARN[0230] [etcd] host [161.211.c.d] failed to check etcd health: failed to get /health for host [161.211.c.d]: Get "https://161.211.c.d:2379/health": remote error: tls: bad certificate FATA[0230] [etcd] Failed to bring up Etcd Plane: etcd cluster is unhealthy: hosts [161.211.a.b,161.211.c.d] failed to report healthy. Check etcd container logs on each host for more information

on checking the etcd logs, I see the following -

":1} {"level":"warn","ts":"2022-08-23T04:11:53.409Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56526","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:11:59.057Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56556","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:04.473Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56604","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:09.864Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56688","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:15.222Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56754","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:20.748Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56784","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:26.119Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56822","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:31.473Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56860","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:36.977Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56890","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:42.393Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56926","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:47.750Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:56970","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:53.115Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:57006","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:12:58.490Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:57042","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:13:03.887Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:57074","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:13:09.276Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:57100","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:13:14.666Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:57144","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:13:20.068Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:57170","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:13:25.428Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"161.211.a.b:57208","server-name":"","error":"tls: failed to verify client certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")"} {"level":"warn","ts":"2022-08-23T04:14:08.275Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54120","server-name":"","error":"remote error: tls: unknown certificate"} {"level":"warn","ts":"2022-08-23T04:14:08.276Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54119","server-name":"","error":"remote error: tls: unknown certificate"} {"level":"warn","ts":"2022-08-23T04:14:08.500Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54121","server-name":"","error":"remote error: tls: unknown certificate"} {"level":"warn","ts":"2022-08-23T04:14:12.462Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54124","server-name":"","error":"remote error: tls: unknown certificate"} {"level":"warn","ts":"2022-08-23T04:14:12.465Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54125","server-name":"","error":"remote error: tls: unknown certificate"} {"level":"warn","ts":"2022-08-23T04:14:12.713Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54126","server-name":"","error":"remote error: tls: unknown certificate"} {"level":"warn","ts":"2022-08-23T04:14:13.001Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54127","server-name":"","error":"EOF"} {"level":"warn","ts":"2022-08-23T04:14:13.061Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54128","server-name":"","error":"EOF"} {"level":"warn","ts":"2022-08-23T04:14:13.600Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54129","server-name":"","error":"tls: client didn't provide a certificate"} {"level":"warn","ts":"2022-08-23T04:14:14.009Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"x.y.z.z1:54130","server-name":"","error":"tls: client didn't provide a certificate"} {"level":"info","ts":"2022-08-23T04:15:38.371Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":905721000} {"level":"info","ts":"2022-08-23T04:15:38.804Z","caller":"mvcc/kvstore_compaction.go:57","msg":"finished scheduled compaction","compact-revision":905721000,"took":"417.824049ms"}

sidvijay1989 commented 2 years ago

rkestate file was missed in the local path.

nanited commented 6 months ago

I am also getting same issue while upgrading rancher local cluster by using rke command by providing cluster.yml by changing images in it to latest version.