Closed sagb closed 1 month ago
Thanks for opening your first issue here! Be sure to follow the issue template!
@sagb I don't think current code has the auto-renew functionality. https://github.com/apache/cloudstack/blob/main/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterStartWorker.java#L147-L149
Also, I can see certificates from ACS are issued for 10 years so it is interesting why it expired in your case.
The logs you shared are WARN level. They are logged initially when ACS MS tries to connect to the control node. The operation was tried repeatedly several times. It may be possible that the management server was able to connect to the control node later
The certificates on k8s control nodes are really expired:
control-node-1:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 18, 2024 08:23 UTC <invalid> ca no
apiserver Jul 18, 2024 08:22 UTC <invalid> ca no
apiserver-etcd-client Jul 18, 2024 08:22 UTC <invalid> etcd-ca no
apiserver-kubelet-client Jul 18, 2024 08:22 UTC <invalid> ca no
controller-manager.conf Jul 18, 2024 08:22 UTC <invalid> ca no
etcd-healthcheck-client May 26, 2024 08:19 UTC <invalid> etcd-ca no
etcd-peer May 26, 2024 08:19 UTC <invalid> etcd-ca no
etcd-server May 26, 2024 08:19 UTC <invalid> etcd-ca no
front-proxy-client Jul 18, 2024 08:22 UTC <invalid> front-proxy-ca no
scheduler.conf Jul 18, 2024 08:23 UTC <invalid> ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca May 24, 2033 08:19 UTC 8y no
etcd-ca May 24, 2033 08:19 UTC 8y no
front-proxy-ca May 24, 2033 08:19 UTC 8y no
Since Cloudstack shows the expired certificate in "kubernetes control" web ui page, it has some control over them. How can I trigger the renewal?
It doesn't seem that Cloudstack will be able to connect to the control nodes later, as it has already been trying for several hours.
@sagb
Cloudstack doesn't provide a way to automatic renewal k8s component certificates.
Your request can be an improvement request.
When you launch a cks cluster , internally CKS uses kubeadm to setup the kubernetes cluster
Client certificates generated by kubeadm expire after 1 year.
root@test-control-190e8277e14:~# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 25, 2025 04:32 UTC 364d ca no
apiserver Jul 25, 2025 04:32 UTC 364d ca no
apiserver-etcd-client Jul 25, 2025 04:32 UTC 364d etcd-ca no
apiserver-kubelet-client Jul 25, 2025 04:32 UTC 364d ca no
controller-manager.conf Jul 25, 2025 04:32 UTC 364d ca no
etcd-healthcheck-client Jul 25, 2025 04:32 UTC 364d etcd-ca no
etcd-peer Jul 25, 2025 04:32 UTC 364d etcd-ca no
etcd-server Jul 25, 2025 04:32 UTC 364d etcd-ca no
front-proxy-client Jul 25, 2025 04:32 UTC 364d front-proxy-ca no
scheduler.conf Jul 25, 2025 04:32 UTC 364d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 23, 2034 04:32 UTC 9y no
etcd-ca Jul 23, 2034 04:32 UTC 9y no
front-proxy-ca Jul 23, 2034 04:32 UTC 9y no
Its up the admin user to login to control node and renew the client certificates
As a workaround
Login to the control node and try to delete the following pods after executing "kubeadm certs renew all".
root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-apiserver
root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-scheduler
root@primary1-node:~# kubectl delete pod -n kube-system -l component=kube-controller-manager
root@primary1-node:~# kubectl delete pod -n kube-system -l component=etcd
Another workaround is to upgrade the kubernetes version
kubeadm renews all the certificates during control plane upgrade.
Register 1.28.4 cks iso and upgrade the cks cluster which should renew the certificates
https://download.cloudstack.org/cks/
ref:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
Thank you for the detailed reply.
My issue was complicated by a split-brain situation with etcd on the control nodes, likely caused by restoring control nodes from snapshots. We manually upgraded the certificates, then removed one of the nodes from the cluster and bootstrapped it again. This seems to have fixed everything.
The new kubectl appeared in the CloudStack GUI soon after this, though I still don't understand where it comes from.
To summarize your reply and the third-party documentation, the options for upgrading Kubernetes certificates in CloudStack are clear: either upgrade the certificates manually or upgrade the Kubernetes version.
I think the issue can be closed.
ISSUE TYPE
Bug Report
COMPONENT NAME
Kubernetes
CLOUDSTACK VERSION
4.19.0.2
CONFIGURATION
Kubernetes 1.27.3, two control nodes.
SUMMARY
Our K8s certificates expired. Seems like Cloudstack didn't automatically renew them.
I tried to renew them manually on both control nodes using:
This updated the certificates separately for each control node, and both are recognized by Kubernetes when using /etc/kubernetes/admin.conf as kubeconfig. However, CloudStack's "Kubernetes access" page only showed the old, expired certificate.
In attempt to trigger an automatic renewal, I've restored nodes from snapshot, stopped the k8s cluster from Cloudstack's web UI, and started it. It doesn't start, and there is an exception in /var/log/cloudstack/management/management-server.log:
The abscence of automatic renewal is probably a bug.
Also, I would be grateful for a hint how to recover from the current situation.