pause/suspend/resume/unpause: dial tcp 192.168.39.x:22: connect: no route to host

[prgaly]

Steps to reproduce the issue:

1. minikube pause
2. suspend host
3. resume host
4. minikube unpause

⏸️ Unpausing node minikube ...

❌ Exiting due to GUEST_UNPAUSE: list paused: docker: NewSession: new client: new client: dial tcp 192.168.39.155:22: connect: no route to host

😿 If the above advice does not help, please let us know: 👉 https://github.com/kubernetes/minikube/issues/new/choose

➜ examples git:(master) ✗ minikube status E0926 20:06:27.696075 14354 status.go:329] failed to get storage capacity of /var: NewSession: new client: new client: dial tcp 192.168.39.155:22: connect: no route to host E0926 20:06:27.696692 14354 status.go:216] status error: NewSession: new client: new client: dial tcp 192.168.39.155:22: connect: no route to host minikube type: Control Plane host: Error kubelet: Nonexistent apiserver: Nonexistent kubeconfig: Configured

Optional: Full output of minikube logs command:

minikube logs E0926 20:07:45.520025 14634 logs.go:203] Failed to list containers for "kube-apiserver": docker: NewSession: new client: new client: dial tcp 192.168.39.155:22: connect: no route to host E0926 20:07:48.592041 14634 logs.go:203] Failed to list containers for "etcd": docker: NewSession: new client: new client: dial tcp 192.168.39.155:22: connect: no route to host E0926 20:07:51.664044 14634 logs.go:203] Failed to list containers for "coredns": docker: NewSession: new client: new client: dial tcp 192.168.39.155:22: connect: no route to host E0926 20:07:54.739989 14634 logs.go:203] Failed to list containers for "kube-scheduler": docker: NewSession: new client: new client: dial tcp 192.168.39.155:22: connect: no route to host

[tstromberg]

What driver is this with?

It seems like suspend/resume of the host caused the VM environment to break in some way. I suspect you would have a similar experience with suspend host/resume host/minikube status.

[tstromberg]

One possibility is that suspend/resume may have changed the IP of your VM environment. We'll be able to investigate more once we know what driver we are dealing with.

[prgaly]

What driver is this with?

This is on Linux with libvirt/kvm2 driver - I will check if the IP is changing but if that's the case I guess there's nothing much that can be done?

It seems like suspend/resume of the host caused the VM environment to break in some way. I suspect you would have a similar experience with suspend host/resume host/minikube status.

No idea yet, will test. Thanks.

[tstromberg]

Let's start by seeing if this works:

minikube ip
minikube ssh uptime
<suspend>
<resume>
minikube ip --alsologtostderr -v=1
minikube ssh uptime

If it fails, share the output of

sudo virsh list
sudo virsh domifaddr minikube
sudo virsh domiflist minikube

[prgaly]

Unable to reproduce easily - will keep trying - it may need a longer sleep time - enough to let the DHCP lease expire on minikube-net.

[prgaly]

After overnight suspend resume I was able to reproduce the issue - it looks like the minikube VM has no IP after resume - not sure if that's an libvirt issue or minikube - I will check if my other VMs retain the IP but I am fairly certain they do.

This looks like it will never work for pause/unpause if the minikube VM isn't able to renew the IP as it is paused and minikube needs to talk over the IP channel to resume it. So assuming libvirt is not at fault this is matter of making sure the minikube VM renews its IP after lease expiry - which should solve the normal (non pause/unpause) suspend / resume case? Below are the logs after resume when it doesn't work.

➜ ~ sudo virsh list
Id Name State

1 minikube running

sudo virsh domifaddr minikube

Name MAC address Protocol Address

sudo virsh domiflist minikube

Interface Type Source Model MAC

vnet0 network default virtio 24:c1:4a:45:21:b9 vnet1 network minikube-net virtio 90:f4:a7:ed:dc:91

sudo virsh net-dhcp-leases minikube-net Expiry Time MAC address Protocol IP address Hostname Client ID or DUID

[medyagh]

@prgaly I am working on a PR to implement static IP for docker driver I wonder if the binary in this PR works for your scenario with docker driver?

http://storage.googleapis.com/minikube-builds/9294/minikube-linux-amd64
http://storage.googleapis.com/minikube-builds/9294/minikube-darwin-amd64
http://storage.googleapis.com/minikube-builds/9294/minikube-windows-amd64.exe

do you mind tryiung the same thing with docker driver in that PR ? (https://github.com/kubernetes/minikube/pull/9294)

[prgaly]

@prgaly I am working on a PR to implement static IP for docker driver I wonder if the binary in this PR works for your scenario with docker driver?

http://storage.googleapis.com/minikube-builds/9294/minikube-linux-amd64

do you mind trying the same thing with docker driver in that PR ? (#9294)

Hello Medya. Thanks - I will try it out and report.

[prgaly]

Tested couple times and found no issues after suspend/resume - pause before suspend and unpause after resume worked fine with the new build. Thanks Thomas and Medya.

[tstromberg]

Great news! Thank you.