After updating the certificates, all of my deployments were lost:

victory4software commented 3 months ago

          After updating the certificates, all of my deployments were lost:

root@minikube:/home/docker# PATH="/var/lib/minikube/binaries/v1.26.3:$PATH"

root@minikube:/home/docker# kubeadm certs renew all --cert-dir /var/lib/minikube/certs MISSING! certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed MISSING! certificate embedded in the kubeconfig file for the controller manager to use certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed MISSING! certificate embedded in the kubeconfig file for the scheduler manager to use Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Is it possible to recover my deployments somehow?

Originally posted by @valsv in https://github.com/kubernetes/minikube/issues/10122#issuecomment-2078653733

kundan2707 commented 3 months ago

/kind support

medyagh commented 2 weeks ago

hi @victory4software I am curious why you had to renew your certs manually ? minikube since 1.24 version has been auto-renewing the certs for you, what version were you on https://github.com/kubernetes/minikube/pull/12534

medyagh commented 2 weeks ago

hope that PR was hellpful to your question feel free to reopen this issue

medyagh commented 2 weeks ago

https://github.com/kubernetes/minikube/pull/16249

medyagh commented 2 weeks ago

it would be cool to Experiment to see if deployments will be Gone in newest minikube version if we change the time inside minikube to 2 years from now, and restart minikube...to see if this a issue

xcarolan commented 2 weeks ago

/assign

xcarolan commented 1 week ago

@medyagh , I've tried this and and the API server doesn't start.

Here's my terminal audit..

ubuntu@artful-teal:~$ sudo timedatectl set-ntp false
ubuntu@artful-teal:~$ sudo date
Tue Sep 24 21:42:51 UTC 2024
ubuntu@artful-teal:~$ sudo date --set "24 Sep 2027"
Fri Sep 24 00:00:00 UTC 2027
ubuntu@artful-teal:~$ date
Fri Sep 24 00:00:07 UTC 2027
ubuntu@artful-teal:~$ minikube start
😄  minikube v1.34.0 on Ubuntu 24.04 (kvm/amd64)
✨  Using the docker driver based on existing profile
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.45 ...
🔄  Restarting existing docker container for "minikube" ...
❗  Failing to connect to https://registry.k8s.io/ from both inside the minikube container and host machine
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.31.0 on Docker 27.2.0 ...
❗  kubeadm certificates have expired. Generating new ones...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
❗  Enabling 'default-storageclass' returned an error: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.31.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:

stderr:
error: error validating "/etc/kubernetes/addons/storageclass.yaml": error validating data: failed to download openapi: Get "https://localhost:8443/openapi/v2?timeout=32s": dial tcp [::1]:8443: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
]
❗  Enabling 'storage-provisioner' returned an error: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.31.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:

stderr:
error: error validating "/etc/kubernetes/addons/storage-provisioner.yaml": error validating data: failed to download openapi: Get "https://localhost:8443/openapi/v2?timeout=32s": dial tcp [::1]:8443: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
]
🌟  Enabled addons:

❌  Exiting due to K8S_APISERVER_MISSING: wait 6m0s for node: wait for apiserver proc: apiserver process never appeared
💡  Suggestion: Check that the provided apiserver flags are valid, and that SELinux is disabled
🍿  Related issues:
    ▪ https://github.com/kubernetes/minikube/issues/4536
    ▪ https://github.com/kubernetes/minikube/issues/6014

I've attached the log. After this I reset the time and started the cluster again.

ubuntu@artful-teal:~$ sudo timedatectl set-ntp true
ubuntu@artful-teal:~$ date
Tue Sep 24 22:29:51 UTC 2024
ubuntu@artful-teal:~$ minikube start
😄  minikube v1.34.0 on Ubuntu 24.04 (kvm/amd64)
✨  Using the docker driver based on existing profile
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.45 ...
    > gcr.io/k8s-minikube/kicbase...:  487.90 MiB / 487.90 MiB  100.00% 9.93 Mi
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.31.0 on Docker 27.2.0 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
❌  Problems detected in etcd [932d25a8fc89]:
    {"level":"warn","ts":"2024-09-24T22:31:47.195964Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:53902","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:48.241567Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37658","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:53.033950Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37660","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:53.413810Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37668","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:55.038471Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37682","server-name":"","error":"remote error: tls: bad certificate"}
❌  Problems detected in etcd [932d25a8fc89]:
    {"level":"warn","ts":"2024-09-24T22:31:47.195964Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:53902","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:48.241567Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37658","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:53.033950Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37660","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:53.413810Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37668","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:55.038471Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37682","server-name":"","error":"remote error: tls: bad certificate"}
❌  Problems detected in etcd [932d25a8fc89]:
    {"level":"warn","ts":"2024-09-24T22:31:47.195964Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:53902","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:48.241567Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37658","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:53.033950Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37660","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:53.413810Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37668","server-name":"","error":"remote error: tls: bad certificate"}
    {"level":"warn","ts":"2024-09-24T22:31:55.038471Z","caller":"embed/config_logging.go:170","msg":"rejected connection on client endpoint","remote-addr":"127.0.0.1:37682","server-name":"","error":"remote error: tls: bad certificate"}
❗  Enabling 'default-storageclass' returned an error: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.31.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:

stderr:
error: error validating "/etc/kubernetes/addons/storageclass.yaml": error validating data: failed to download openapi: Get "https://localhost:8443/openapi/v2?timeout=32s": dial tcp [::1]:8443: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false
]
❗  Enabling 'storage-provisioner' returned an error: running callbacks: [sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.31.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:

stderr:
error: error validating "/etc/kubernetes/addons/storage-provisioner.yaml": error validating data: failed to download openapi: Get "https://localhost:8443/openapi/v2?timeout=32s": dial tcp [::1]:8443: connect: connection refused - error from a previous attempt: read tcp [::1]:59104->[::1]:8443: read: connection reset by peer; if you choose to ignore these errors, turn validation off with --validate=false
]

log.txt

kubernetes / minikube

After updating the certificates, all of my deployments were lost: #19134