hyperkit: VM is unable to access k8s.gcr.io (when VPN is in use)

[massenz]

Starting minikube with the default VM driver (Hyperkit) makes the external network unreachable:

minikube -p test start
😄  [test] minikube v1.6.2 on Darwin 10.15.2
✨  Automatically selected the 'hyperkit' driver (alternates: [virtualbox vmwarefusion])
🔥  Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository
🐳  Preparing Kubernetes v1.17.0 on Docker '19.03.5' ...
🚜  Pulling images ...
❌  Unable to pull images, which may be OK: running cmd: "/bin/bash -c \"sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml\"": /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm config images pull --config /var/tmp/minikube/kubeadm.yaml": Process exited with 
status 1

stdout: stderr: W0113 20:47:22.189603 2751 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version. W0113 20:47:22.190387 2751 common.go:77] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta1". Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version. W0113 20:47:22.192219 2751 validation.go:28] Cannot validate kube-proxy config - no validator is available W0113 20:47:22.192254 2751 validation.go:28] Cannot validate kubelet config - no validator is available failed to pull image "k8s.gcr.io/kube-apiserver:v1.17.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) , error: exit status 1 To see the stack trace of this error execute with --v=5 or higher

And then trying to deploy any pod fails.

Starting with --vmdriver virtualbox works just fine.

The exact command to reproduce the issue:

kube apply -f utils.yaml

utils.yaml pulls in massenz/dnsutils:1.1 image (but this is reproducible with any Docker image).

The full output of the command that failed:

Normal BackOff 30s kubelet, minikube Back-off pulling image "massenz/dnsutils:1.1" Warning Failed 30s kubelet, minikube Error: ImagePullBackOff Normal Pulling 16s (x2 over 46s) kubelet, minikube Pulling image "massenz/dnsutils:1.1" Warning Failed 1s (x2 over 31s) kubelet, minikube Failed to pull image "massenz/dnsutils:1.1": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Failed 1s (x2 over 31s) kubelet, minikube Error: ErrImagePull

The output of the minikube logs command:

Jan 11 14:19:23 minikube kubelet[4757]: W0111 14:19:23.514145 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for Jan 11 14:19:30 minikube kubelet[4757]: W0111 14:19:30.612595 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for Jan 11 14:19:31 minikube kubelet[4757]: W0111 14:19:31.754128 4757 docker_sandbox.go:394] failed to read pod IP from plugin/docker: Couldn't find network status for default/utils through plugin: invalid network status for

The operating system version: MacOS 10.15.2

$ minikube version      
minikube version: v1.6.2
commit: 54f28ac5d3a815d1196cd5d57d707439ee4bb392

[medyagh]

@massenz sorry that you face this issue, I am curious does docker pull massenz/dnsutils:1.1 work on your own machine without problem ?

and also do you happen to use VPN or corp proxy? and last question, does it make a difference if you do docker login ?

[massenz]

Yes, Docker works just fine (with/without login) Yes, I use a VPN (hopefully everyone does, these days...) but the problem persists with/without VPN.

Yep, the docker login was the first thing I tried as I assumed that would have been the case; but really, the problem must be around the egress network for the hyperkit VM: as mentioned, when I use --vm-driver virtualbox it works just fine; VBox, by default, creates a bridge and allows egress to the Internet for the VM, I'm guessing Hyperkit doesn't, and as the VM is on a separate subnet than the host network, then it doesn't know how to reach the external network.

BTW - this seems to be a good place to remark how awesome Minikube is - I absolutely love it, it has helped me immensely in exploring K8s, developing a whole stack of services, and generally making my developer's life so much easier: thank you deeply to the contributors!

[raghvendra1218]

At last it seems this issue is with the hyperkitVM, because It works fine with the virtual box. Those who want their pod to be up and running, without worrying about the type of VM, Can follow these steps

1. Delete the existing minikube by running minikube delete
2. Install the virtual box brew cask install virtualbox , make sure you do the installation proper way, if you are using macos it will ask for permission to be enabled from firevault, once enabled re-run the command until you see virtual box is installed successfully.
3. once all is set up finally start the minikube using the virtual box minikube start --vm-driver=virtualbox
4. At last you can run the pod using your usual yaml file _kubectl create -f "yourfilename.yaml"
5. run kubectl get pods
6. in case of error you can check kubectl describe pods to check the detailed errors. If your yml is correct and path for the Image is correct, you should be good and will just require till step 5.

[arvtiwar]

With minikube on virtualbox, all good till the deployment. My Problem is, pods running NodeJS service can't connect to the external services (e.g. MongoDB Atlas). Note: I have deployed on DigitalOcen, the same docker build work perfectly fine.

2020-02-24 17:05:02 [ info ] : server started on port 3333 (local) Error: Error: querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net at NativeConnection. (/usr/src/app/services/mongoose.js:57:19) at NativeConnection.emit (events.js:196:13) at NativeConnection.EventEmitter.emit (domain.js:471:20) at /usr/src/app/node_modules/mongoose/lib/connection.js:817:37 at processTicksAndRejections (internal/process/task_queues.js:82:9) 2020-02-24 17:05:02 [ error ] : Error connecting to DB : querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net Error: querySrv ENOTFOUND _mongodb._tcp.twrdev-hokqm.mongodb.net at QueryReqWrap.onresolve [as oncomplete] (dns.js:203:19) { errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'querySrv', hostname: '_mongodb._tcp.twrdev-hokqm.mongodb.net' } [Unhandled Rejection] Node NOT Exiting... Error: Could not connect to database at NativeConnection. (/usr/src/app/services/mongoose.js:70:19) at NativeConnection.emit (events.js:196:13) at NativeConnection.EventEmitter.emit (domain.js:471:20) at NativeConnection.set (/usr/src/app/node_modules/mongoose/lib/connection.js:125:12) at /usr/src/app/node_modules/mongoose/lib/connection.js:711:26 at executeCallback (/usr/src/app/node_modules/mongodb/lib/operations/execute_operation.js:74:5) at /usr/src/app/node_modules/mongodb/lib/operations/connect.js:209:23 at /usr/src/app/node_modules/mongodb/lib/operations/connect.js:299:21 at QueryReqWrap.callback (/usr/src/app/node_modules/mongodb/lib/core/uri_parser.js:56:21) at QueryReqWrap.onresolve [as oncomplete] (dns.js:203:10) 2020-02-24 17:05:02 [ error ] : Disconnected from Database! Any suggestion?? Thanks in advance

[tstromberg]

For hyperkit, VM is unable to access k8s.gcr.io says to me that this is likely a DNS issue, and likely #3036

Do you mind sharing the output of the following two commands for me?

• sudo lsof -i4UDP:53 -P -n
• ps -afe | grep dns

Thank you!

[ashleyconnor]

dnsmasq was causing issues for me:

brew uninstall dnsmasq
sudo lsof -i4UDP:53 -P -n

COMMAND PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
dnsmasq 123 nobody    4u  IPv4 0xb150fdd17fe0cd81      0t0  UDP 127.0.0.1:53

sudo kill -9 123

Minikube then runs without issues.

If you want to keep dnsmasq installed then I would try these instructions: https://github.com/kubernetes/minikube/issues/3036#issuecomment-423150899

[chrber]

Same problem here, no dnsmasq installed, when using HyperKit.

The question is, which application does interfere on my machine?

MacOs Mojave: 10.14.6 (18G2022)

minikube version
minikube version: v1.7.3 commit: 436667c819c324e35d7e839f8116b968a2d0a3ff

kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-13T18:06:54Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

[riklopfer]

I'm also running into this issue. Note: I also have AnyConnect running -- seems like this could be a common denominator.

😄  minikube v1.9.2 on Darwin 10.15.4
✨  Automatically selected the hyperkit driver
👍  Starting control plane node m01 in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

minikube version: v1.9.2 commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

I've tried re-installing with,

minikube stop 
minikube delete
brew reinstall minikube

I'm running Docker Desktop 2.2.0.5 No dnsmasq

sudo lsof -i4UDP:53 -P -n
<yields nothing> 

Following DNS Debugging, I see the following.

$ for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done
E0415 18:58:18.027806       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:18.028150       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:18.028245       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b
E0415 18:58:17.954733       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:17.954834       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
E0415 18:58:17.954901       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: connection refused
.:53
[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
CoreDNS-1.6.7
linux/amd64, go1.13.6, da7f65b

[majinghe]

Hi team I met the same error. Mac Mojave , version 10.14.6. run the command minikube start --vm-driver=hyperkit to start the minikube, below is the logs

😄  minikube v1.9.2 on Darwin 10.14.6
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node m01 in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

I am using Ciso VPN, no matter i set the proxy or not, the result is the same.

kubectl -n kube-system get pods
NAME                               READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-fnxht           1/1     Running   5          56m
coredns-66bff467f8-vxr5s           1/1     Running   5          56m
etcd-minikube                      1/1     Running   5          56m
kube-apiserver-minikube            1/1     Running   5          56m
kube-controller-manager-minikube   1/1     Running   5          56m
kube-proxy-pt5js                   1/1     Running   5          56m
kube-scheduler-minikube            1/1     Running   5          56m
storage-provisioner                1/1     Running   8          56m

Can get the dns log

kubectl -n kube-system logs coredns-66bff467f8-fnxht 
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0423 14:01:35.230402       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):
Trace[2019727887]: [30.002647734s] [30.002647734s] END
E0423 14:01:35.230463       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.230738       1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):
Trace[1427131847]: [30.004178291s] [30.004178291s] END
E0423 14:01:35.230753       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.235236       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):
Trace[939984059]: [30.002522331s] [30.002522331s] END
E0423 14:01:35.235287       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I can get the some docker images inside the VM

minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                     v1.18.0             43940c34f24f        4 weeks ago         117MB
k8s.gcr.io/kube-apiserver                 v1.18.0             74060cea7f70        4 weeks ago         173MB
k8s.gcr.io/kube-controller-manager        v1.18.0             d3e55153f52f        4 weeks ago         162MB
k8s.gcr.io/kube-scheduler                 v1.18.0             a31f78c7c8ce        4 weeks ago         95.3MB
kubernetesui/dashboard                    v2.0.0-rc6          cdc71b5a8a0e        5 weeks ago         221MB
k8s.gcr.io/pause                          3.2                 80d28bedfe5d        2 months ago        683kB
k8s.gcr.io/coredns                        1.6.7               67da37a9a360        2 months ago        43.8MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        5 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        6 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        6 months ago        40.1MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

But when i try to login the dockerhub, failed

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: dllbh
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some article said it has relationship with the DNS

sudo lsof -i4UDP:53 -P -n
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 93271 _mdnsresponder   33u  IPv4 0x1154dc77691251b5      0t0  UDP *:53

After kill the process, error still occurs. It seems a bug for minikube with hyperkit driver. How to fix this?

By the way,

minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

[crankbird]

I'm also seeing this, trying to run on a Mac that has a "Cisco Anconnect" VPN software and hyperkit .. the easiest workaround is to use the --vm-driver=virtualbox option. I'm happy to provide config information if anyone really wants the details however my gut feeling is that the corporate installed Cisco VPN software is the culprit, it futzes with DNS even when its not "turned on" to ensure I'm not accessing "inappropriate" websites like say .. urban dictionary (I know, it seems unreasonable, but thats just collateral damage for a decent security posture so I put up with it).

[mjm19091979]

I'm also seeing this, trying to run on a Mac that has a "Cisco Anconnect" VPN software and hyperkit .. the easiest workaround is to use the --vm-driver=virtualbox option. I'm happy to provide config information if anyone really wants the details however my gut feeling is that the corporate installed Cisco VPN software is the culprit, it futzes with DNS even when its not "turned on" to ensure I'm not accessing "inappropriate" websites like say .. urban dictionary (I know, it seems unreasonable, but thats just collateral damage for a decent security posture so I put up with it).

---

I am using win10 pro and I have default hyper-v and cisco anyconnect while running minikube it shows the exactly same error however, if I turn-off the cisco anyconnect, it still shows the same error " "! This VM is having trouble accessing https://k8s.gcr.io"

how can we turn off hyper-v on windows and start using --vm-driver=virtualbox

[mjm19091979]

Hi team I met the same error. Mac Mojave , version 10.14.6. run the command minikube start --vm-driver=hyperkit to start the minikube, below is the logs

😄  minikube v1.9.2 on Darwin 10.14.6
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node m01 in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🐳  Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

I am using Ciso VPN, no matter i set the proxy or not, the result is the same.

kubectl -n kube-system get pods
NAME                               READY   STATUS    RESTARTS   AGE
coredns-66bff467f8-fnxht           1/1     Running   5          56m
coredns-66bff467f8-vxr5s           1/1     Running   5          56m
etcd-minikube                      1/1     Running   5          56m
kube-apiserver-minikube            1/1     Running   5          56m
kube-controller-manager-minikube   1/1     Running   5          56m
kube-proxy-pt5js                   1/1     Running   5          56m
kube-scheduler-minikube            1/1     Running   5          56m
storage-provisioner                1/1     Running   8          56m

Can get the dns log

kubectl -n kube-system logs coredns-66bff467f8-fnxht 
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0423 14:01:35.230402       1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):
Trace[2019727887]: [30.002647734s] [30.002647734s] END
E0423 14:01:35.230463       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.230738       1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):
Trace[1427131847]: [30.004178291s] [30.004178291s] END
E0423 14:01:35.230753       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
I0423 14:01:35.235236       1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):
Trace[939984059]: [30.002522331s] [30.002522331s] END
E0423 14:01:35.235287       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I can get the some docker images inside the VM

minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-proxy                     v1.18.0             43940c34f24f        4 weeks ago         117MB
k8s.gcr.io/kube-apiserver                 v1.18.0             74060cea7f70        4 weeks ago         173MB
k8s.gcr.io/kube-controller-manager        v1.18.0             d3e55153f52f        4 weeks ago         162MB
k8s.gcr.io/kube-scheduler                 v1.18.0             a31f78c7c8ce        4 weeks ago         95.3MB
kubernetesui/dashboard                    v2.0.0-rc6          cdc71b5a8a0e        5 weeks ago         221MB
k8s.gcr.io/pause                          3.2                 80d28bedfe5d        2 months ago        683kB
k8s.gcr.io/coredns                        1.6.7               67da37a9a360        2 months ago        43.8MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        5 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        6 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        6 months ago        40.1MB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

But when i try to login the dockerhub, failed

$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: dllbh
Password:
Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some article said it has relationship with the DNS

sudo lsof -i4UDP:53 -P -n
COMMAND     PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
mDNSRespo 93271 _mdnsresponder   33u  IPv4 0x1154dc77691251b5      0t0  UDP *:53

After kill the process, error still occurs. It seems a bug for minikube with hyperkit driver. How to fix this?

By the way,

minikube version
minikube version: v1.9.2
commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

I have the probelm, can you please tell me how to get inside minikube vm?

[medyagh]

You can use minikube ssh

On Thu, Apr 30, 2020, 7:36 PM mjm19091979 notifications@github.com wrote:

Hi team I met the same error. Mac Mojave , version 10.14.6. run the command minikube start --vm-driver=hyperkit to start the minikube, below is the logs

😄 minikube v1.9.2 on Darwin 10.14.6

✨ Using the hyperkit driver based on existing profile

👍 Starting control plane node m01 in cluster minikube

🔄 Restarting existing hyperkit VM for "minikube" ...

🐳 Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...

❗ This VM is having trouble accessing https://k8s.gcr.io

💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

🌟 Enabling addons: default-storageclass, storage-provisioner

🏄 Done! kubectl is now configured to use "minikube"

I am using Ciso VPN, no matter i set the proxy or not, the result is the same.

kubectl -n kube-system get pods

NAME READY STATUS RESTARTS AGE

coredns-66bff467f8-fnxht 1/1 Running 5 56m

coredns-66bff467f8-vxr5s 1/1 Running 5 56m

etcd-minikube 1/1 Running 5 56m

kube-apiserver-minikube 1/1 Running 5 56m

kube-controller-manager-minikube 1/1 Running 5 56m

kube-proxy-pt5js 1/1 Running 5 56m

kube-scheduler-minikube 1/1 Running 5 56m

storage-provisioner 1/1 Running 8 56m

Can get the dns log

kubectl -n kube-system logs coredns-66bff467f8-fnxht

[INFO] plugin/ready: Still waiting on: "kubernetes"

[INFO] plugin/ready: Still waiting on: "kubernetes"

I0423 14:01:35.230402 1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.226674406 +0000 UTC m=+0.085300319) (total time: 30.002647734s):

Trace[2019727887]: [30.002647734s] [30.002647734s] END

E0423 14:01:35.230463 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I0423 14:01:35.230738 1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.22650071 +0000 UTC m=+0.085126637) (total time: 30.004178291s):

Trace[1427131847]: [30.004178291s] [30.004178291s] END

E0423 14:01:35.230753 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I0423 14:01:35.235236 1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105 (started: 2020-04-23 14:01:05.232689948 +0000 UTC m=+0.091315902) (total time: 30.002522331s):

Trace[939984059]: [30.002522331s] [30.002522331s] END

E0423 14:01:35.235287 1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

I can get the some docker images inside the VM

minikube ssh

                     _             _

        _         _ ( )           ( )

(_) _ ()| |/') | |

/' ` \| |/' _| || , < ( ) ( )| '_\ /'__\

| ( ) ( ) || || ( ) || || |`\ | () || |) )( ___/

() () ()()() ()()() (_)\___/'(_,__/'____)

$ docker images

REPOSITORY TAG IMAGE ID CREATED SIZE k8s.gcr.io/kube-proxy v1.18.0 43940c34f24f 4 weeks ago 117MB k8s.gcr.io/kube-apiserver v1.18.0 74060cea7f70 4 weeks ago 173MB k8s.gcr.io/kube-controller-manager v1.18.0 d3e55153f52f 4 weeks ago 162MB k8s.gcr.io/kube-scheduler v1.18.0 a31f78c7c8ce 4 weeks ago 95.3MB

kubernetesui/dashboard v2.0.0-rc6 cdc71b5a8a0e 5 weeks ago 221MB k8s.gcr.io/pause 3.2 80d28bedfe5d 2 months ago 683kB k8s.gcr.io/coredns 1.6.7 67da37a9a360 2 months ago 43.8MB

kindest/kindnetd 0.5.3 aa67fec7d7ef 5 months ago 78.5MB k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 6 months ago 288MB

kubernetesui/metrics-scraper v1.0.2 3b08661dc379 6 months ago 40.1MB gcr.io/k8s-minikube/storage-provisioner v1.8.1 4689081edb10 2 years ago 80.8MB

But when i try to login the dockerhub, failed

$ docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.

Username: dllbh

Password:

Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Some article said it has relationship with the DNS

sudo lsof -i4UDP:53 -P -n

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

mDNSRespo 93271 _mdnsresponder 33u IPv4 0x1154dc77691251b5 0t0 UDP *:53

After kill the process, error still occurs. It seems a bug for minikube with hyperkit driver. How to fix this?

By the way,

minikube version

minikube version: v1.9.2

commit: 93af9c1e43cab9618e301bc9fa720c63d5efa393

I have the probelm, can you please tell me how to get inside minikube vm?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/minikube/issues/6296#issuecomment-622220331, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABC2KAZUETHTFA3DC4KGUPLRPIYRZANCNFSM4KGIXIMQ .

[tstromberg]

We should update the error string to mention trying --driver=docker when a VPN is in use.

[medyagh]

we need a solution message, if can't pull images due to network issues, only on windows and macos, if they are not already using docker driver they should use docker driver.

[sharifelgamal]

Still an issue.

[vignesh-subburaj]

| =>minikube delete

| =>minikube start --driver=docker
😄  minikube v1.11.0 on Darwin 10.15.4
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=1989MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

worked for me

OS: MacOs Catalina 10.15.4 (19E287)

[priyawadhwa]

@vignesh-subburaj great. I'm not sure what the cause of this issue is, so for now it seems our recommended advice is to try to use the docker driver instead of the hyperkit driver.

If anyone would be interested in updating our warning message from:

This VM is having trouble accessing https://k8s.gcr.io

to something like

This VM is having trouble accessing https://k8s.gcr.io -- consider using the `docker` driver by running `minikube start --driver docker`

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[kylemclaren]

Still an issue for me:

😄  minikube v1.15.1 on Darwin 10.15.7
✨  Using the hyperkit driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

minikube version: v1.15.1
commit: 23f40a012abb52eff365ff99a709501a61ac5876

EDIT: I should add that no VPN is in use

[bmariesan]

same issue for me, similarly to @kylemclaren I have no VPN at all,

[marcusthelin]

I'm on mac and have to use the hyperkit driver for Ingress to work. I solved this VPN issue by adding the --hyperkit-vpnkit-sock=auto flag to minikube start. This way it uses the VPNKit that ships with Docker for Mac. I found it here: https://minikube.sigs.k8s.io/docs/drivers/hyperkit/#special-features

Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if ‘auto’ uses Docker for Mac VPNKit connection, otherwise uses the specified VSock

[sharifelgamal]

We should add an integration test that tests minikube behind a VPN.

[gongpengjun]

I'm on mac and have to use the hyperkit driver for Ingress to work. I solved this VPN issue by adding the --hyperkit-vpnkit-sock=auto flag to minikube start. This way it uses the VPNKit that ships with Docker for Mac. I found it here: https://minikube.sigs.k8s.io/docs/drivers/hyperkit/#special-features

Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if ‘auto’ uses Docker for Mac VPNKit connection, otherwise uses the specified VSock

minikube start --hyperkit-vpnkit-sock=auto works for me. I found when close Cisco VPN client and minikube hyperkit VM can't access internet. I try to turn off and then of Mac network, the VM can access internet again.

[ctoomey]

I'm using the Global Protect VPN by Palo Alto Networks and was having the issue that while I could connect to external servers accessible outside our VPN, I couldn't connect to those inside our VPN. @gongpengjun's solution fixed it for me.

[mbateman-vz]

Using --hyperkit-vpnkit-sock=auto tries to use the VPNKit included with Docker Desktop. But the whole reason to go to minikube/hyperkit was to remove the dependency on Docker Desktop. Is there any way to use VPNKit without Desktop?

[ipv1337]

Using --hyperkit-vpnkit-sock=auto tries to use the VPNKit included with Docker Desktop. But the whole reason to go to minikube/hyperkit was to remove the dependency on Docker Desktop. Is there any way to use VPNKit without Desktop?

I've yet to find an answer to this as well. Why do people like macs again? =) I hope one day our company opens up other options other than forcing macs, but for now I still have my trusty linux desktop that just works.

[pbabbott]

Using --hyperkit-vpnkit-sock=auto tries to use the VPNKit included with Docker Desktop. But the whole reason to go to minikube/hyperkit was to remove the dependency on Docker Desktop. Is there any way to use VPNKit without Desktop?

I've yet to find an answer to this as well. Why do people like macs again? =) I hope one day our company opens up other options other than forcing macs, but for now I still have my trusty linux desktop that just works.

For those of us who have to use a Mac and Cisco AnyConnect- are there any options to resolve this? Perhaps installing vpn kit from source somehow?

[ar2pi]

Running minikube alongside vpnkit on Mac seems to work, with a couple of gotchas.

Building vpnkit from source fails on original's repo (moby/vpnkit) and latest binaries are not available anywhere. In short, the Makefile for Mac build needs a bit of tweaking for opam dependencies. So I forked the original repo to build from source:

# install build dependencies
brew install opam gpatch pkg-config dune dylibbundler libtool automake

# build vpnkit
git clone git@github.com:ar2pi/vpnkit.git
cd vpnkit
make -f Makefile.darwin ocaml
make -f Makefile.darwin depends
make -f Makefile.darwin build
cp ~/.opam/4.12.0/bin/vpnkit /usr/local/bin/vpnkit

Then hyperkit Homebrew's install also has a known issue, so we need to build that from source as well.

# build hyperkit
brew uninstall hyperkit
git clone git@github.com:moby/hyperkit.git
cd hyperkit
make
cp build/hyperkit /usr/local/bin/hyperkit

Once you have vpnkit and hyperkit, you can run:

# terminal 1
vpnkit --ethernet=/tmp/vpnkit.eth.sock
# terminal 2
minikube start --driver hyperkit --hyperkit-vpnkit-sock=/tmp/vpnkit.eth.sock --memory 8192 --cpus 4
eval $(minikube -p minikube docker-env)

# [...] your docker commands

And voilà! Docker will run within minikube's hyperkit VM, through vpnkit.

But there's still a couple of connection error messages that have been bugging me for a few days when starting / restarting a new VM:

[...]
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
❗  This VM is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
[...]

And on vpnkit's output we can see:

time="2021-11-27T02:53:59Z" level=warning msg="arp: ARP table has no entry for 192.168.64.26"
time="2021-11-27T02:53:59Z" level=info msg="ipv4: IP.output: could not determine link-layer address for local network (0.0.0.0/0) ip 192.168.64.26"
time="2021-11-27T02:53:59Z" level=warning msg="ipv4: Could not find 192.168.64.26 on the local network"
time="2021-11-27T02:53:59Z" level=warning msg="Wire: Error sending TCP packet via IP: no route to destination: no response for IP on local network"

Once VM is started everything appears to be fine though, could pull a few images and run basic docker commands. Haven't yet tested container to container networking, nor file mounts.

[semistone]

I just workaround by installing proxyman in my local laptop and add HTTP_PROXY settings when launching minikube

[eberlou]

I just workaround by installing proxyman in my local laptop and add HTTP_PROXY settings when launching minikube

What are you setting your HTTP_PROXY to? The Proxyman "listening on" address and port?

[semistone]

I just workaround by installing proxyman in my local laptop and add HTTP_PROXY settings when launching minikube

What are you setting your HTTP_PROXY to? The Proxyman "listening on" address and port?

After turning on VPN, I can't pull image because can't connect to the internet so I just run by

export HTTPS_PROXY=192.168.64.1:9090 export HTTP_PROXY=192.168.64.1:9090 export NO_PROXY=localhost,127.0.0.1,10.96.0.0/12,192.168.59.0/24,192.168.39.0/24,192.168.64.0/24 minikube start

and the postman will start a proxy and listen on port 9090

Reference to https://minikube.sigs.k8s.io/docs/reference/networking/proxy/

[eberlou]

Thanks! It worked. Not the solution I hoped for, but it enables me to remove docker desktop for now.

[ewan-chalmers]

I tried long and hard to get minikube with hyperkit to work on macos

I couldn't get past

⚠️  VM is unable to access k8s.gcr.io, you may need to configure a proxy or set --image-repository

... and related, in a persistent way.

My corporate VPN means I have a bunch of cisco processes - even when the VPN is not running

$ sudo lsof -i4UDP:53 -P -n
COMMAND    PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
com.cisco  430   root   10u  IPv4 0x8f3cc549208e6f87      0t0  UDP 192.168.1.6:63839->8.8.8.8:53
com.cisco  430   root   13u  IPv4 0x8f3cc5491e00b607      0t0  UDP 192.168.1.6:50014->8.8.8.8:53
com.cisco  430   root   14u  IPv4 0x8f3cc549208ead27      0t0  UDP 192.168.1.6:62140->8.8.8.8:53
com.cisco  430   root   15u  IPv4 0x8f3cc5491d646867      0t0  UDP 192.168.1.6:60980->8.8.8.8:53
com.cisco  430   root   16u  IPv4 0x8f3cc549208eb037      0t0  UDP 192.168.1.6:59493->8.8.8.8:53
com.cisco  430   root   17u  IPv4 0x8f3cc549208eb347      0t0  UDP 192.168.1.6:62816->8.8.8.8:53
com.cisco  430   root   18u  IPv4 0x8f3cc549208eb657      0t0  UDP 192.168.1.6:54240->8.8.8.8:53
com.cisco  430   root   19u  IPv4 0x8f3cc549208eb967      0t0  UDP 192.168.1.6:58697->8.8.8.8:53
com.cisco  430   root   20u  IPv4 0x8f3cc5491d6477b7      0t0  UDP 192.168.1.6:64878->8.8.8.8:53
com.cisco  430   root   21u  IPv4 0x8f3cc5491d647ac7      0t0  UDP 192.168.1.6:64042->8.8.8.8:53
com.cisco  430   root   22u  IPv4 0x8f3cc5491d6480e7      0t0  UDP 192.168.1.6:63625->8.8.8.8:53
com.cisco  430   root   23u  IPv4 0x8f3cc5491d6433f7      0t0  UDP 192.168.1.6:53050->8.8.8.8:53
dnscrypt- 1610 nobody   46u  IPv4 0x8f3cc5491d11f347      0t0  UDP 127.0.0.1:53

I tried unsuccessfully using File Sync to copy an /etc/resolv.conf with nameserver 8.8.8.8 and other things

I found that /etc/resolv.conf on the minikube VM got regularly overridden. I tried unsuccessfully to use a systemd dropin.

I gave up and installed virtualbox instead. It worked out of the box, with and without VPN connected.

[dhlpj]

I want to use minikube without docker desktop on mac by hyperkit vm. Also, I'm using VPN. I have tried setting proxy and VPN, refer to https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/, but not working. Any update on this issue? The minikube version: v1.25.2. Thanks

[craftzneko]

I dont have a VPN (no vpn installed ever). I get This VM is having trouble accessing https://k8s.gcr.io using hyper kit driver, any hope this can be resolved or is hyperkit driver no longer supported?

[r4j4h]

Could https://github.com/containers/gvisor-tap-vsock be of help here?

edit:

A replacement for libslirp and VPNKit, written in pure Go. It is based on the network stack of gVisor.
Compared to libslirp, gvisor-tap-vsock brings a configurable DNS server and dynamic port forwarding.
It can be used with Qemu, Hyperkit, Hyper-V and User Mode Linux.

[sftim]

BTW k8s.gcr.io is deprecated as a source of container images

[lukasfrank]

On my machine is some corporate software (cisco security, vpn) running and I got it working by starting vpnkit along with minikube:

vpnkit --ethernet /tmp/vpn.socket

minikube start --driver hyperkit --hyperkit-vpnkit-sock=/tmp/vpn.socket

[gsmethells]

This is still an issue in April 2023. I do not have a VPN running (direct internet connection) and I see:

😄 minikube v1.30.1 on Darwin 12.6.5 ✨ Automatically selected the hyperkit driver. Other choices: parallels, virtualbox, ssh 👍 Starting control plane node minikube in cluster minikube 🔥 Creating hyperkit VM (CPUs=2, Memory=6000MB, Disk=20000MB) ... ❗ This VM is having trouble accessing https://registry.k8s.io 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ 🐳 Preparing Kubernetes v1.26.3 on Docker 20.10.23 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔗 Configuring bridge CNI (Container Networking Interface) ... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🔎 Verifying Kubernetes components... 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

[sftim]

k8s.gcr.io is deprecated - see https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/ and other announcements.

/remove-help (pending triage)

[sftim]

@massenz would you be willing to revise this to provide an updated steps-to-reproduce, covering the new registry etc?