search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.51k stars 4.89k forks source link

error execution phase kubeconfig/admin: a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert #4287

Closed dineshtailor closed 5 years ago

dineshtailor commented 5 years ago

The exact command to reproduce the issue:

The full output of the command that failed:

The output of the minikube logs command:

The operating system version:

tstromberg commented 5 years ago

Thank you for sharing your experience! If you don't mind, could you please provide:

This will help us isolate the problem further. Thank you!

vikaschoudhary16 commented 5 years ago

Version: 0.30.0 and 1.1.0 I am also hitting this. First time, command ran fine:

 minikube start --vm-driver=none --extra-config=kubelet.cgroup-driver=systemd --kubernetes-version v1.12.0 --v 5  --alsologtostderr

But when i retried after minikube delete, always getting this error. I have tried even removing all the state:

rm  -rf /etc/kubernetes
rm -rf ~/.kube && rm -rf ~/.minikube && rm -rf /var/lib/minikube

Still no luck.

full output:

[Install]
💾  Downloading kubeadm v1.12.0
💾  Downloading kubelet v1.12.0
I0530 07:21:56.875116   23331 exec_runner.go:39] Run: 
sudo systemctl daemon-reload &&
sudo systemctl start kubelet

I0530 07:21:57.326687   23331 certs.go:46] Setting up certificates for IP: 192.168.1.103
I0530 07:21:58.664432   23331 interface.go:360] Looking for default routes with IPv4 addresses
I0530 07:21:58.664449   23331 interface.go:365] Default route transits interface "wlp4s0"
I0530 07:21:58.664619   23331 interface.go:174] Interface wlp4s0 is up
I0530 07:21:58.664684   23331 interface.go:222] Interface "wlp4s0" has 2 addresses :[192.168.1.103/24 fe80::c78a:deb4:c5d7:d13b/64].
I0530 07:21:58.664700   23331 interface.go:189] Checking addr  192.168.1.103/24.
I0530 07:21:58.664712   23331 interface.go:196] IP found 192.168.1.103
I0530 07:21:58.664723   23331 interface.go:228] Found valid IPv4 address 192.168.1.103 for interface "wlp4s0".
I0530 07:21:58.664732   23331 interface.go:371] Found active IP 192.168.1.103 
I0530 07:21:58.664781   23331 kubeconfig.go:127] Using kubeconfig:  /etc/kubernetes/admin.conf
🚜  Pulling images ...
I0530 07:21:58.665151   23331 exec_runner.go:39] Run: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
🚀  Launching Kubernetes ... 
I0530 07:22:35.472607   23331 kubeadm.go:221] Older Kubernetes release detected (1.12.0), disabling SystemVerification check.
I0530 07:22:35.472643   23331 exec_runner.go:50] Run with output: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification
I0530 07:22:39.680352   23331 exec_runner.go:50] Run with output: docker ps -a --filter="name=kube-apiserver" --format="{{.ID}}"
I0530 07:22:39.739817   23331 logs.go:158] 0 containers: []
W0530 07:22:39.739868   23331 logs.go:160] No container was found matching "kube-apiserver"
I0530 07:22:39.739885   23331 exec_runner.go:50] Run with output: docker ps -a --filter="name=coredns" --format="{{.ID}}"
I0530 07:22:39.788627   23331 logs.go:158] 0 containers: []
W0530 07:22:39.788657   23331 logs.go:160] No container was found matching "coredns"
I0530 07:22:39.788673   23331 exec_runner.go:50] Run with output: docker ps -a --filter="name=kube-scheduler" --format="{{.ID}}"
I0530 07:22:39.834988   23331 logs.go:158] 0 containers: []
W0530 07:22:39.835131   23331 logs.go:160] No container was found matching "kube-scheduler"
I0530 07:22:39.835209   23331 exec_runner.go:50] Run with output: docker ps -a --filter="name=kube-proxy" --format="{{.ID}}"
I0530 07:22:39.882639   23331 logs.go:158] 0 containers: []
W0530 07:22:39.882803   23331 logs.go:160] No container was found matching "kube-proxy"
I0530 07:22:39.882908   23331 exec_runner.go:50] Run with output: docker ps -a --filter="name=kube-addon-manager" --format="{{.ID}}"
I0530 07:22:39.927728   23331 logs.go:158] 0 containers: []
W0530 07:22:39.927750   23331 logs.go:160] No container was found matching "kube-addon-manager"
I0530 07:22:39.927789   23331 exec_runner.go:50] Run with output: docker ps -a --filter="name=kubernetes-dashboard" --format="{{.ID}}"
I0530 07:22:39.974652   23331 logs.go:158] 0 containers: []
W0530 07:22:39.974677   23331 logs.go:160] No container was found matching "kubernetes-dashboard"
I0530 07:22:39.974688   23331 exec_runner.go:50] Run with output: docker ps -a --filter="name=storage-provisioner" --format="{{.ID}}"
I0530 07:22:40.021866   23331 logs.go:158] 0 containers: []
W0530 07:22:40.021906   23331 logs.go:160] No container was found matching "storage-provisioner"
I0530 07:22:40.021925   23331 logs.go:76] Gathering logs for kubelet ...
I0530 07:22:40.021950   23331 exec_runner.go:50] Run with output: journalctl -u kubelet -n 200
I0530 07:22:40.090523   23331 logs.go:76] Gathering logs for dmesg ...
I0530 07:22:40.090569   23331 exec_runner.go:50] Run with output: sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 200
W0530 07:22:40.207338   23331 exit.go:100] Error starting cluster: cmd failed: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification

: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification
 output: [init] using Kubernetes version: v1.12.0
[preflight] running pre-flight checks
    [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
    [WARNING Swap]: running with swap on is not supported. Please disable swap
    [WARNING FileExisting-socat]: socat not found in system path
    [WARNING Hostname]: hostname "minikube" could not be reached
    [WARNING Hostname]: hostname "minikube" lookup minikube on 127.0.0.1:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [minikube localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [minikube localhost] and IPs [192.168.1.103 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/var/lib/minikube/certs/"
[certificates] Generated sa key and public key.
a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification
.: exit status 1

💣  Error starting cluster: cmd failed: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification

: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification
 output: [init] using Kubernetes version: v1.12.0
[preflight] running pre-flight checks
    [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
    [WARNING Swap]: running with swap on is not supported. Please disable swap
    [WARNING FileExisting-socat]: socat not found in system path
    [WARNING Hostname]: hostname "minikube" could not be reached
    [WARNING Hostname]: hostname "minikube" lookup minikube on 127.0.0.1:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [minikube localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [minikube localhost] and IPs [192.168.1.103 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] valid certificates and keys now exist in "/var/lib/minikube/certs/"
[certificates] Generated sa key and public key.
a kubeconfig file "/etc/kubernetes/admin.conf" exists already but has got the wrong CA cert
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--data-minikube,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,SystemVerification
.: exit status 1

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new
sharifelgamal commented 5 years ago

Looks like #2857 reported the same issue. I'm not sure if it's been fixed in the meantime.

vikaschoudhary16 commented 5 years ago

unfortunately, work-around/solution suggested there, does not work for me.

Thanks a lot for replying. Looking forward for further inputs/hints.

If i run kubeadm init ... command directly it works. But when i run minikube start ... after cleaning up all the cache like /etc/kubernetes, /var/lib/minikube, ~/.minikube, it fails at kubeadm init with the error in subject.

vikaschoudhary16 commented 5 years ago

I have figured out solution for this problem. I had to run:

minikube config set embed-certs true

NOTE: This is supported only post 0.30.0

WHAT WAS THE ISSUE: kubeconfig file created by minikube was not having client-certificate-data: and kubeadm was expecting this to be non-empty and was failing from here https://github.com/kubernetes/kubernetes/blob/release-1.12/cmd/kubeadm/app/phases/kubeconfig/kubeconfig.go#L271

Instead, minikube was setting certificate-authority in the kubeconfig.admin, which is actually file path.

By running above command, minikube initializes certificate-authority-data:

I am not sure why others are not facing this issue

fejta-bot commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

tstromberg commented 5 years ago

@dineshtailor - Hopefully it's OK if I close this - there wasn't enough information to make it actionable, and some time has already passed. If you are able to provide additional details, you may reopen it at any point by adding /reopen to your comment.

Here is additional information that may be helpful to us:

Thank you for sharing your experience!