search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.52k stars 4.89k forks source link

Minikube tunnel is not working on Windows (endessly trying to add a route) #11645

Open sm4sh1k opened 3 years ago

sm4sh1k commented 3 years ago

Minikube tunnel is not working on Windows because of minikube can not recognize the correct answer (or status code) of the route command.

Steps to reproduce the issue:

  1. minikube start --driver vmware
  2. minikube tunnel --cleanup --alsologtostderr

Full output of failed command:

PS C:\Windows\system32> minikube tunnel --cleanup --alsologtostderr
I0614 02:41:08.169600    3528 out.go:291] Setting OutFile to fd 84 ...
I0614 02:41:08.178573    3528 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0614 02:41:08.178573    3528 out.go:304] Setting ErrFile to fd 88...
I0614 02:41:08.178573    3528 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0614 02:41:08.185616    3528 root.go:291] Error reading config file at C:\Users\User\.minikube\config\config.json: open C:\Users\User\.minikube\config\config.json: The system cannot find the file specified.
I0614 02:41:08.186129    3528 mustload.go:65] Loading cluster: minikube
I0614 02:41:08.191755    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:08.191755    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:08.204499    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60699
I0614 02:41:08.205496    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:08.206051    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:08.206051    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:08.206564    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:08.206564    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:08.207075    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:08.417717    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:08.417717    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:08.426165    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:08.431377    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:08.431377    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:08.444381    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60702
I0614 02:41:08.444381    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:08.444891    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:08.444891    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:08.445405    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:08.445405    3528 main.go:128] libmachine: (minikube) Calling .DriverName
I0614 02:41:08.445917    3528 api_server.go:148] Checking apiserver status ...
I0614 02:41:08.452561    3528 ssh_runner.go:149] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0614 02:41:08.453071    3528 main.go:128] libmachine: (minikube) Calling .GetSSHHostname
I0614 02:41:08.454096    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:08.663265    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:08.663265    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:08.671568    3528 main.go:128] libmachine: (minikube) DBG | MAC address in VMX: 00:0c:29:ca:93:41
I0614 02:41:08.671568    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in configuration file: C:\ProgramData\VMware\vmnetdhcp.conf
I0614 02:41:08.672079    3528 main.go:128] libmachine: (minikube) DBG | Following IPs found map[00:50:56:c0:00:01:192.168.106.1 00:50:56:c0:00:08:192.168.198.1]
I0614 02:41:08.672591    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in leases file: C:\ProgramData\VMware\vmnetdhcp.leases
I0614 02:41:08.673102    3528 main.go:128] libmachine: (minikube) DBG | IP found in DHCP lease table: 192.168.198.128
I0614 02:41:08.673102    3528 main.go:128] libmachine: (minikube) Calling .GetSSHPort
I0614 02:41:08.673615    3528 main.go:128] libmachine: (minikube) Calling .GetSSHKeyPath
I0614 02:41:08.673615    3528 main.go:128] libmachine: (minikube) Calling .GetSSHUsername
I0614 02:41:08.673615    3528 sshutil.go:53] new ssh client: &{IP:192.168.198.128 Port:22 SSHKeyPath:C:\Users\User\.minikube\machines\minikube\id_rsa Username:docker}
I0614 02:41:08.768157    3528 ssh_runner.go:149] Run: sudo egrep ^[0-9]+:freezer: /proc/6269/cgroup
I0614 02:41:08.771472    3528 api_server.go:164] apiserver freezer: "6:freezer:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88b6effc84e76606921f1b221ad131c.slice/docker-5149535924cb43a882b3a4bcd18074fb7844c7e0f76c4234fbb9e9f0dd958119.scope"
I0614 02:41:08.778203    3528 ssh_runner.go:149] Run: sudo cat /sys/fs/cgroup/freezer/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88b6effc84e76606921f1b221ad131c.slice/docker-5149535924cb43a882b3a4bcd18074fb7844c7e0f76c4234fbb9e9f0dd958119.scope/freezer.state
I0614 02:41:08.781719    3528 api_server.go:186] freezer state: "THAWED"
I0614 02:41:08.781719    3528 api_server.go:223] Checking apiserver healthz at https://192.168.198.128:8443/healthz ...
I0614 02:41:08.786036    3528 api_server.go:249] https://192.168.198.128:8443/healthz returned 200:
ok
I0614 02:41:08.786551    3528 tunnel.go:57] Checking for tunnels to cleanup...
I0614 02:41:08.794762    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:08.800968    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:08.800968    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:08.813239    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60706
I0614 02:41:08.813749    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:08.813749    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:08.813749    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:08.814742    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:08.814742    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:08.815302    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:09.021094    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:09.021094    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:09.029606    3528 main.go:128] libmachine: (minikube) Calling .GetIP
I0614 02:41:09.030118    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:09.237544    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:09.238580    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:09.246815    3528 main.go:128] libmachine: (minikube) DBG | MAC address in VMX: 00:0c:29:ca:93:41
I0614 02:41:09.246815    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in configuration file: C:\ProgramData\VMware\vmnetdhcp.conf
I0614 02:41:09.247328    3528 main.go:128] libmachine: (minikube) DBG | Following IPs found map[00:50:56:c0:00:01:192.168.106.1 00:50:56:c0:00:08:192.168.198.1]
I0614 02:41:09.247328    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:09.247841    3528 main.go:128] libmachine: (minikube) DBG | Trying to find IP address in leases file: C:\ProgramData\VMware\vmnetdhcp.leases
I0614 02:41:09.247841    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:09.247841    3528 main.go:128] libmachine: (minikube) DBG | IP found in DHCP lease table: 192.168.198.128
I0614 02:41:09.248352    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:09.248352    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:09.248868    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:09.248868    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:09.249384    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:09.249384    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:09.249384    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:09.249897    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:09.249897    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:09.250411    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:09.250411    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:09.250411    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:09.250942    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:09.251455    3528 tunnel_manager.go:71] Setting up tunnel...
I0614 02:41:09.251455    3528 tunnel_manager.go:81] Started minikube tunnel.
I0614 02:41:14.251829    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:14.256747    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:14.257256    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:14.270187    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60719
I0614 02:41:14.270700    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:14.270700    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:14.270700    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:14.271210    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:14.271210    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:14.272235    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:14.479222    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:14.479222    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:14.602249    3528 route_windows.go:47] Adding route for CIDR 10.96.0.0/12 to gateway 192.168.198.128
I0614 02:41:14.602249    3528 route_windows.go:49] About to run command: [route ADD 10.96.0.0 MASK 255.240.0.0 192.168.198.128]
Status:
        machine: minikube
        pid: 3528
        route: 10.96.0.0/12 -> 192.168.198.128
        minikube: Running
        services: []
    errors:
                minikube: no errors
                router: error adding route:  ��
, 2
                loadbalancer emulator: no errors
I0614 02:41:14.618056    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:14.618056    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:14.618569    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:14.618569    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:14.619082    3528 main.go:128] libmachine: Making call to close connection to plugin binary
I0614 02:41:19.619835    3528 host.go:66] Checking if "minikube" exists ...
I0614 02:41:19.625308    3528 main.go:128] libmachine: Found binary path at C:\Program Files (x86)\VMware\VMware Workstation\bin\docker-machine-driver-vmware.exe
I0614 02:41:19.625371    3528 main.go:128] libmachine: Launching plugin server for driver vmware
I0614 02:41:19.638659    3528 main.go:128] libmachine: Plugin server listening at address 127.0.0.1:60739
I0614 02:41:19.638659    3528 main.go:128] libmachine: () Calling .GetVersion
I0614 02:41:19.639172    3528 main.go:128] libmachine: Using API Version  1
I0614 02:41:19.639172    3528 main.go:128] libmachine: () Calling .SetConfigRaw
I0614 02:41:19.639683    3528 main.go:128] libmachine: () Calling .GetMachineName
I0614 02:41:19.639683    3528 main.go:128] libmachine: (minikube) Calling .GetState
I0614 02:41:19.640708    3528 main.go:128] libmachine: (minikube) DBG | executing: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe list
I0614 02:41:19.847709    3528 main.go:128] libmachine: (minikube) DBG | Total running VMs: 1
I0614 02:41:19.847709    3528 main.go:128] libmachine: (minikube) DBG | C:\Users\User\.minikube\machines\minikube\minikube.vmx
I0614 02:41:19.976476    3528 route_windows.go:47] Adding route for CIDR 10.96.0.0/12 to gateway 192.168.198.128
I0614 02:41:19.976476    3528 route_windows.go:49] About to run command: [route ADD 10.96.0.0 MASK 255.240.0.0 192.168.198.128]
Status:
        machine: minikube
        pid: 3528
        route: 10.96.0.0/12 -> 192.168.198.128
        minikube: Running
        services: []
    errors:
                minikube: no errors
                router: error adding route: ���� ���������� ��������: ���� ��ꥪ� 㦥 ����������.

, 3
                loadbalancer emulator: no errors
I0614 02:41:19.992076    3528 main.go:128] libmachine: Making call to close driver server
I0614 02:41:19.992076    3528 main.go:128] libmachine: (minikube) Calling .Close
I0614 02:41:19.992588    3528 main.go:128] libmachine: (minikube) DBG | Closing plugin on server side
I0614 02:41:19.992588    3528 main.go:128] libmachine: Successfully made call to close driver server
I0614 02:41:19.993098    3528 main.go:128] libmachine: Making call to close connection to plugin binary

The first time the route is successfully created, command route ADD 10.96.0.0 MASK 255.240.0.0 192.168.198.128 returns OK (on russian). All next tries to add a route are ended with error The route exists (on russian too). So, in spite of route is created minikube thinks that is not and endlessly tries to create it again and again.

joseluisgomes commented 3 years ago

Currently I have the same problem

ziemerz commented 3 years ago

I am facing similar issues on Mac with the hyperkit driver

Nirvandil commented 3 years ago

Same here with VirtualBox backend

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

sathiralamal commented 2 years ago

Same here with Docker

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

sharifelgamal commented 2 years ago

I'm not sure what exactly is causing this issue, I would love someone to take a closer look.

Yjq14114 commented 2 years ago

I have the same problem PS C:\Windows\system32> minikube.exe tunnel Status: machine: minikube pid: 7464 route: 10.96.0.0/12 -> 172.20.126.198 minikube: Running services: [] errors: minikube: no errors router: error adding route: ·������ʧ��: �����Ѵ��ڡ�

, 3 loadbalancer emulator: no errors

sathiralamal commented 2 years ago

run minikube tunnel in separate terminal window

ghost commented 2 years ago

Change windows language to en-US will solve this problem. Currently, minikube adds/checks route by compare route command's output, which will changes based on system language.

We need a better way to examine the execution status of route command.

medyagh commented 2 years ago

seems like an easy fix, I would accept any PR that fixes this relying on English text for the tunnel command

Rishit-dagli commented 2 years ago

I'd love to take this up but had a hard time figuring out the part introducing the reliance on English text, could you help a bit on this front @medyagh ?

favadi commented 2 years ago

I'll give this issue a try, downloading Windows 11 Russian ISO file.