Cert rotate should not proceed if kube-ca has expired

sowmyav27 commented 4 years ago

RKE version: v1.1.0-rc16

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) AWS

cluster.yml file:

nodes:
  - address: <ip-1>
    internal_address: <p-ip-1>
    user: ubuntu
    role: [controlplane,worker,etcd]
    ssh_key_path: <>
  - address: <ip-2>
    internal_address: <p-ip-2>
    user: ubuntu
    role: [controlplane,worker,etcd]
    ssh_key_path: <>
  - address: <ip-3>
    internal_address: <p-ip-3>
    user: ubuntu
    role: [controlplane,worker,etcd]
    ssh_key_path: <>

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

Steps to Reproduce:

Deploy a cluster expired with certs (using rke cli binary with certs expiry in 15 minutes)
Wait for the cluster certs to expire
Do a cert rotate
Rotate certs - ./rke cert rotate --config rancher-cluster.yml (rotates all certs except kube-ca)

Results:

certs get rotate but kube-apiserver on the nodes in restarting state and the cluster is not accessible via kubectl.
There is panic in the logs of the kube-apiserver
'[' kube-apiserver = kubelet ']'
exec kube-apiserver --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-servers=https://172.31.0.167:2379,https://172.31.6.140:2379,https://172.31.4.104:2379 --service-account-lookup=true --runtime-config=authorization.k8s.io/v1beta1=true --authorization-mode=Node,RBAC --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --insecure-port=0 --storage-backend=etcd3 --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --requestheader-username-headers=X-Remote-User --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --requestheader-extra-headers-prefix=X-Remote-Extra- --secure-port=6443 --advertise-address=172.31.0.167 --service-cluster-ip-range=10.43.0.0/16 --cloud-provider= --profiling=false --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem Flag --insecure-port has been deprecated, This flag will be removed in a future version. I0317 20:07:37.475409 1 server.go:596] external host was not specified, using 172.31.0.167 I0317 20:07:37.475706 1 server.go:150] Version: v1.17.4 I0317 20:07:38.280504 1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass. I0317 20:07:38.280528 1 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota. I0317 20:07:38.281351 1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass. I0317 20:07:38.281371 1 plugins.go:161] Loaded 7 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota. I0317 20:07:38.282332 1 client.go:361] parsed scheme: "endpoint" I0317 20:07:38.282375 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://172.31.0.167:2379 0 } {https://172.31.6.140:2379 0 } {https://172.31.4.104:2379 0 }] I0317 20:07:39.279664 1 client.go:361] parsed scheme: "endpoint" I0317 20:07:39.279706 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://172.31.0.167:2379 0 } {https://172.31.6.140:2379 0 } {https://172.31.4.104:2379 0 }] W0317 20:07:58.284076 1 clientconn.go:1120] grpc: addrConn.createTransport failed to connect to {https://172.31.6.140:2379 0 }. Err :connection error: desc = "transport: authentication handshake failed: context deadline exceeded". Reconnecting... panic: context deadline exceeded

goroutine 1 [running]: k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition.NewREST(0xc0007f0af0, 0x4f6dd40, 0xc000156a20, 0xc000156c48) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/registry/customresourcedefinition/etcd.go:56 +0x3c1 k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver.completedConfig.New(0xc0008511e0, 0xc000220988, 0x502e420, 0x75682f0, 0x10, 0x0, 0x0) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go:147 +0x152b k8s.io/kubernetes/cmd/kube-apiserver/app.createAPIExtensionsServer(0xc000220980, 0x502e420, 0x75682f0, 0x0, 0x4f6d980, 0xc0005e4f90) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/apiextensions.go:99 +0x59 k8s.io/kubernetes/cmd/kube-apiserver/app.CreateServerChain(0xc00032bb80, 0xc0002a6ea0, 0x449aeca, 0xc, 0xc0006dbc48) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:183 +0x292 k8s.io/kubernetes/cmd/kube-apiserver/app.Run(0xc00032bb80, 0xc0002a6ea0, 0x0, 0x0) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:152 +0x101 k8s.io/kubernetes/cmd/kube-apiserver/app.NewAPIServerCommand.func1(0xc0002c8500, 0xc00051c480, 0x0, 0x23, 0x0, 0x0) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:119 +0x104 k8s.io/kubernetes/vendor/github.com/spf13/cobra.(Command).execute(0xc0002c8500, 0xc0000be010, 0x23, 0x23, 0xc0002c8500, 0xc0000be010) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:826 +0x460 k8s.io/kubernetes/vendor/github.com/spf13/cobra.(Command).ExecuteC(0xc0002c8500, 0x15fd30372f0644d7, 0x754a340, 0xc00006a750) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 +0x2fb k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...) /workspace/anago-v1.17.4-beta.0.54+12bf0cb73007af/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864 main.main() _output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:43 +0xcd

echo kube-apiserver --anonymous-auth=false --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-group-headers=X-Remote-Group --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-servers=https://172.31.0.167:2379,https://172.31.6.140:2379,https://172.31.4.104:2379 --service-account-lookup=true --runtime-config=authorization.k8s.io/v1beta1=true --authorization-mode=Node,RBAC --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --insecure-port=0 --storage-backend=etcd3 --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --etcd-prefix=/registry --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --allow-privileged=true --requestheader-username-headers=X-Remote-User --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --requestheader-extra-headers-prefix=X-Remote-Extra- --secure-port=6443 --advertise-address=172.31.0.167 --service-cluster-ip-range=10.43.0.0/16 --cloud-provider= --profiling=false --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem
grep -q cloud-provider=azure
'[' kube-apiserver = kubelet ']'

mrajashree commented 4 years ago

@sowmyav27 Can you share the RKE logs?

mrajashree commented 4 years ago

Based on offline discussion with @superseb this happens because we rotate certs even if kube-ca has expired, and that should be avoided

mrajashree commented 4 years ago

Fix available with v1.1.0-rc17

sowmyav27 commented 4 years ago

Verified with rke: v1.1.0-rc17

Deploy a cluster, with certs expiring in 10 minutes.
Wait for the certs to expire
Do a cert rotate on the cluster

./rke cert rotate --config rancher-cluster.yml
WARN[0000] This is not an officially supported version (v1.1.0-rc17) of RKE. Please download the latest official release at https://github.com/rancher/rke/releases/latest 
INFO[0000] Running RKE version: v1.1.0-rc17             
INFO[0000] Initiating Kubernetes cluster                
INFO[0000] Rotating Kubernetes cluster certificates     
FATA[0000] Failed to rotate certificates: CA certificate is invalid, please use the --rotate-ca flag to rotate CA certificate, error: x509: certificate has expired or is not yet valid

do a cert rotate using ./rke cert rotate --rotate-ca --config rancher-cluster.yml
certs are rotated.
Cluster is accessible using kubectl - kubectl get nodes --kubeconfig kube_config_rancher-cluster.yml
Now on the same cluster - cert rotate works - ./rke cert rotate --config rancher-cluster.yml
certs are rotated.

rancher / rke

Cert rotate should not proceed if kube-ca has expired #1974