Minikube start does not work.

[romu42]

What Happened?

Minikube will not start despite multiple attempts. [ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': executable file not found in $PATH

I have tested the recommendations in: https://github.com/kubernetes/minikube/issues/18939 and https://github.com/kubernetes/minikube/issues/18884

without success.

Attach the log file

logs.txt

Operating System

macOS (Default)

Driver

Docker

[drozd-A]

I am in the same boat. Followed the suggested issues and still unable to start mini-kube.

[spowelljr]

Hi @romu42, thanks for reporting your issue with minikube. We have seen this occasionally but have so far been unable to reproduce this ourselves. The current theory is that the kubelet binary is either missing from the cluster, doesn't have executable permissions, or somehow makes it's way onto the machine too late in the start process.

After you attempt to start minikube and it fails, could you run minikube ssh it should hopefully allow you to do so, then could you run ls -la /var/lib/minikube/binaries/v1.30.0 and upload the output. That will help us see if our theory is correct.

[romu42]

ran minikube delete --all --purge ; minikube start . and then:

❯ minikube ssh
docker@minikube:~$ ls -la /var/lib/minikube/binaries/v1.30.0
total 190780
drwxr-xr-x 2 root root     4096 Apr 18 05:50 .
drwxr-xr-x 3 root root     4096 Apr 18 05:50 ..
-rwxr-xr-x 1 1001 1001 48955544 Apr 18 05:50 kubeadm
-rwxr-xr-x 1 1001 1001 49938584 Apr 18 05:50 kubectl
-rwxr-xr-x 1 1001 1001 96446456 Apr 18 05:50 kubelet
docker@minikube:~$

[spowelljr]

Interesting, so the binaries are there, and that correlates to your logs:

I0620 13:15:06.804729   14001 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.30.0
I0620 13:15:06.883552   14001 binaries.go:44] Found k8s binaries, skipping transfer

And immediately after that the kubectl service is started:

I0620 13:15:07.770471   14001 ssh_runner.go:195] Run: sudo systemctl start kubelet

Which fails due to the binaries not being present:

Jun 20 11:15:07 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jun 20 11:15:07 minikube systemd[1576]: kubelet.service: Failed to execute /var/lib/minikube/binaries/v1.30.0/kubelet: No such file or directory
Jun 20 11:15:07 minikube systemd[1576]: kubelet.service: Failed at step EXEC spawning /var/lib/minikube/binaries/v1.30.0/kubelet: No such file or directory
Jun 20 11:15:07 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Jun 20 11:15:07 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jun 20 11:15:08 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1.
Jun 20 11:15:08 minikube systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Jun 20 11:15:08 minikube systemd[1]: Started kubelet: The Kubernetes Node Agent.
Jun 20 11:15:08 minikube systemd[1578]: kubelet.service: Failed to execute /var/lib/minikube/binaries/v1.30.0/kubelet: No such file or directory
Jun 20 11:15:08 minikube systemd[1578]: kubelet.service: Failed at step EXEC spawning /var/lib/minikube/binaries/v1.30.0/kubelet: No such file or directory
Jun 20 11:15:08 minikube systemd[1]: kubelet.service: Main process exited, code=exited, status=203/EXEC
Jun 20 11:15:08 minikube systemd[1]: kubelet.service: Failed with result 'exit-code'.
Jun 20 11:15:09 minikube systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 2.

So I'm not sure how they can be there but not be there at the same time, my current guess is maybe something is wrong with the preload side of things, if you do a delete and then try starting with minikube start --driver=docker --preload=false that should acquire the binaries a different way.

[romu42]

I tried that. Seems to still be a problem. I am uploading the logs from that run. Let me know if there are any other tests I can do. logs_preload_false.txt

Just a side note: I installed file and ran it on the binaries. It is returning the following:

docker@minikube:/var/lib/minikube/binaries/v1.30.0$ file *
kubeadm: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=KQNbLlw3GyqDpcCOq9IM/_Yc1x2_sW-ezJJ4buJsE/Ir594eVRkzMuVjvpq3-V/A_mlKXzzhfL33u3-REer, stripped
kubectl: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=VzRAnGXpLlQctbNaLnKl/oZDpJF6tO440o8MZX2jG/RvCjxRr54MHOuPRJSrwS/dYEmNm_CirHEAgYUSdff, stripped
kubelet: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=756ad8e3e80107a65118e072c146ee5a6050c964, for GNU/Linux 3.7.0, stripped 

I think it is a little odd that it is ARM instead of x86_64

Uname returns x86_64

docker@minikube:/var/lib/minikube/binaries/v1.30.0$ uname -m
x86_64

I am not sure if it is significant or not, or if that is the way it should be?

[spowelljr]

That's not good...

I can see everything with your setup is arm64

Binary: Built with gc go1.22.3 for darwin/arm64
...
hostinfo: "kernelArch":"arm64"
...
OperatingSystem:Docker Desktop OSType:linux Architecture:aarch64

So it should be using an arm64 kicbase image, but uname -m says otherwise.

This is why you're getting kubelet errors, I'm assuming if you sshed in and tried to execute the binary you're get a exec error due to mismatching arch between the container and the binaries.

The question now is why is Docker pulling the amd64 version of the kicbase image instead of arm64?

Have you configured Docker Desktop at all? Any experimental settings enabled?

To see if this is a Docker Desktop issue could you try the following:

$ docker pull alpine:latest
$ docker image inspect alpine:latest | grep Architecture

If you get amd64 then there's something wrong with your Docker Desktop that's causing it to not pull arm64 images.

When minikube pulls the image it specifically sets the platform based on the binary arch, so it should pull the arm64 image regardless unless something in Docker is overriding this.

        defaultPlatform = v1.Platform{
                Architecture: runtime.GOARCH,
                OS:           "linux",
        }

I see you ran minikube delete --all --purge, you still had two images in your docker daemon (Images:2), could one of them possibly been the kicbase image?

Just to make sure we get rid of all possible causes could you run minikube delete --all --purge again, then run docker images and if any of the images are kicbase images could you docker rmi them so we have a completely clean slate and then try starting again.

I created a PR (https://github.com/kubernetes/minikube/pull/19205) to check the arch of the kicbase image on start and will redownload if the arch doesn't match. I can provide the binary from the PR once its built if you want to test that as well.

[spowelljr]

Here's an arm64 macOS binary with the kicbase arch check if you want to download it and try it after trying the above steps.

https://storage.googleapis.com/minikube-builds/19205/minikube-darwin-arm64

[romu42]

This could have been entirely my fault. I had an export in my shell that set: DOCKER_DEFAULT_PLATFORM=linux/amd64

To quiet The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested from another setup.

I removed the export and retested and it is now working as expected. Minikube pulls one image ( 648c6affc8e4 ) which is arm64

❯ docker image inspect 648c6affc8e4 | grep Arch
        "Architecture": "arm64",

With DOCKER_DEFAULT_PLATFORM=linux/amd64 set, then minikube pulls two images:

gcr.io/k8s-minikube/kicbase                                         v0.0.44   648c6affc8e4   3 months ago        1.17GB
gcr.io/k8s-minikube/kicbase                                         <none>    5a6e59a9bdc0   3 months ago        1.26GB

one (648c6affc8e4) is arm64 and one (5a6e59a9bdc0) is amd64.

Sorry for the slow turn-around time. Hope this helps and if there is anything more I can do please let me know. Thank you for your time and assistance.