Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

roy-work commented 8 months ago

What Happened?

» minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /home/roy/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster.

» minikube start 
😄  minikube v1.32.0 on Ubuntu 23.04 (amd64)
✨  Automatically selected the docker driver. Other choices: none, ssh
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.28.3 preload ...
    > preloaded-images-k8s-v18-v1...:  403.35 MiB / 403.35 MiB  100.00% 260.90 
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🔥  Deleting "minikube" in docker ...
🤦  StartHost failed, but will try again: creating host: create host timed out in 360.000000 seconds
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
😿  Failed to start docker container. Running "minikube delete" may fix it: creating host: create host timed out in 360.000000 seconds

❌  Exiting due to DRV_CREATE_TIMEOUT: Failed to start host: creating host: create host timed out in 360.000000 seconds
💡  Suggestion: Try 'minikube delete', and disable any conflicting VPN or firewall software
🍿  Related issue: https://github.com/kubernetes/minikube/issues/7072

The logs are full of:

I0213 14:52:59.107708   17969 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:32792 SSHKeyPath:/home/roy/.minikube/machines/minikube/id_rsa Username:docker}
W0213 14:52:59.130977   17969 sshutil.go:64] dial failure (will retry): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
I0213 14:52:59.130996   17969 retry.go:31] will retry after 300.690662ms: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
W0213 14:52:59.455324   17969 sshutil.go:64] dial failure (will retry): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
I0213 14:52:59.455339   17969 retry.go:31] will retry after 250.410719ms: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
W0213 14:52:59.729781   17969 sshutil.go:64] dial failure (will retry): ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
I0213 14:52:59.729802   17969 retry.go:31] will retry after 646.185592ms: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
I0213 14:52:59.734559   17969 main.go:141] libmachine: Error dialing TCP: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

I can't SSH either (the port here is the port from docker ps),

» ssh docker@127.0.0.1 -p 32792 -i ~/.minikube/machines/minikube/id_rsa
The authenticity of host '[127.0.0.1]:32792 ([127.0.0.1]:32792)' can't be established.
ED25519 key fingerprint is SHA256:/GNLGkp0oz8keBudJAQktFdHwAE0H4AT+VPjBotw3Ik.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[127.0.0.1]:32792' (ED25519) to the list of known hosts.
docker@127.0.0.1's password: 
Permission denied, please try again.
docker@127.0.0.1's password: 
(last command got signal SIGINT)

The reason minikube can't SSH is that the authorized_keys file is unusable, due to bad permissions; we can see this if we docker exec into the container:

root@minikube:/home/docker# ls -la
total 12
drwxr-x--- 1 root root    8 Feb 12 22:29 .
drwxr-xr-x 1 root root   12 Feb 12 22:29 ..
-rw-r--r-- 1 root root  220 Feb 12 22:29 .bash_logout
-rw-r--r-- 1 root root 3771 Feb 12 22:29 .bashrc
-rw-r--r-- 1 root root  807 Feb 12 22:29 .profile
drwxr-xr-x 1 root root   30 Feb 13 14:47 .ssh

~docker should be owned by docker, not by root, and similarly, ~docker/.ssh should too. sshd simply can't get to authorized_keys.

If we correct that:

root@minikube:/home/docker# chown docker: . .ssh

Then we're able to SSH:

» ssh docker@127.0.0.1 -p 32792 -i ~/.minikube/machines/minikube/id_rsa
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

docker@minikube:~$

But why is ~docker inside the container so messed up to begin with?

Attach the log file

minikube logs fails

minikube.log minikube_logs_05c6d87097f2294684e6847624a5fb0bff018ece_0.log

Operating System

Ubuntu

Driver

None

roy-work commented 8 months ago

Even if I attempt to race minikube start & chown docker: … while it is running, it still fails with:

🤦  StartHost failed, but will try again: creating host: create: provisioning: Temporary Error: sudo mkdir -p /etc/docker /etc/docker /etc/docker: Process exited with status 1
stdout:

stderr:
sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set

Indeed, sudo does not have the setuid bit set.

roy-work commented 8 months ago

I eventually found minikube delete takes flags?!

» minikube delete --all --purge
🔥  Successfully deleted all profiles
💀  Successfully purged minikube directory located at - [/home/roy/.minikube]
📌  Kicbase images have not been deleted. To delete images run:
    ▪ docker rmi gcr.io/k8s-minikube/kicbase:v0.0.42

I also ran the docker rmi.

That worked.

I still have no idea why that worked, or even how I got into this state in the first place.

…why does a normal unadorned minikube delete say,

💀 Removed all traces of the "minikube" cluster.

… when that's clearly not true?

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 3 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/minikube/issues/18162#issuecomment-2226000861): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

kubernetes / minikube