RA489
commented
3 years ago /kind support
Closed OkayJosh closed 1 year ago
RA489
commented
3 years ago /kind support
bkahlerventer
commented
3 years ago I do not think minikube handles/likes the redirect:
~> curl https://k8s.gcr.io
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://cloud.google.com/container-registry/">here</A>.
</BODY></HTML>
cmarquezrusso
commented
3 years ago Same issue here. I tried 1.15.1 and 1.12.3
I am not behind a proxy. I sshed into the minikube instance and I am not even able to resolve / connect to google. Resolvconf is pointing to 192.168.64.1
kubectl get events
Failed to pull image "jettech/kube-webhook-certgen:v1.3.0": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.64.1:53: read udp 192.168.64.44:60147->192.168.64.1:53: i/o timeoutOS: MacOs Mojave
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/[...]
I1203 13:30:00.069998 22265 preload.go:105] Found local preload: /Users/cristian/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4
I1203 13:30:00.070126 22265 ssh_runner.go:148] Run: docker images --format {{.Repository}}:{{.Tag}}
I1203 13:30:10.091456 22265 ssh_runner.go:188] Completed: curl -sS -m 2 https://k8s.gcr.io/: (10.062157611s)
I1203 13:30:10.091487 22265 ssh_runner.go:188] Completed: docker images --format {{.Repository}}:{{.Tag}}: (10.021275564s)
W1203 13:30:10.091500 22265 start.go:505] [curl -sS -m 2 https://k8s.gcr.io/] failed: curl -sS -m 2 https://k8s.gcr.io/: Process exited with status 28
stdout:
stderr:
curl: (28) Resolving timed out after 2000 milliseconds
I1203 13:30:10.091515 22265 docker.go:381] Got preloaded images:
I1203 13:30:10.091523 22265 docker.go:386] k8s.gcr.io/kube-proxy:v1.18.3 wasn't preloaded
W1203 13:30:10.091603 22265 out.go:151] ❗ This VM is having trouble accessing https://k8s.gcr.io
❗ This VM is having trouble accessing https://k8s.gcr.io
W1203 13:30:10.091634 22265 out.go:151] 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/./minikube-darwin-amd64 ssh
_ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$ curl -sS -m 2 https://k8s.gcr.io/
curl: (28) Resolving timed out after 2000 millisecondsI solved my problem by stopping minikube, delete the minikube docker container, delete the minikube image, remove the minikube commandline app, then starting fresh with the same versions, then problem was solved.
Looks like some configuration that got saved to the minikube container.
cmarquezrusso
commented
3 years ago I am leaving a recording of my terminal showing the error. I think it's related to the DNS configuration on the Hyperkit VM.
https://asciinema.org/a/377638
It works fine with both Docker and Virtualbox drivers.
cmarquezrusso
commented
3 years ago This seems to be related with #3036
cmarquezrusso
commented
3 years ago The issue was related to a VPN Kernel extension for MacOs that disables communication with the Hyperkit endpoint (192.168.64.1)
If you need to troubleshoot this, try opening a port on your host machine nc -l 8080 and try connecting to that port from the minikube instance. curl -v 192.168.64.1:8080
My fix was to uninstall the VPN Client.
I think this can be closed
cmarquezrusso
commented
3 years ago I had the same issue on Windows using hyper-v. The connection between minikube and the hyper-v eth interface was blocked by a firewall. To fix the issue, disable the firewall and test running a webserver on the windows host and try connecting to it using the hyper-v ip address (it should be same as the nameserver on /etc/resolv.conf)
Workaround: Uninstall hyper-v and use virtualbox.
k8s-ci-robot
commented
3 years ago @cristian04: The label(s) kind/hyper-v cannot be applied, because the repository doesn't have them
santhoshcameo
commented
3 years ago having the same when tried to run with Gvisor runtime via minikube
ubuntu@gvisor-san:~$ minikube start --container-runtime=containerd \
> --docker-opt containerd=/var/run/containerd/containerd.sock
😄 minikube v1.18.1 on Ubuntu 18.04 (amd64)
✨ Using the docker driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing docker container for "minikube" ...
🌐 Found network options:
▪ NO_PROXY=localhost,127.0.0.1,169.254.169.254,dkfz-heidelberg.de,192.168.49.2,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24
▪ http_proxy=http://www-int2.dkfz-heidelberg.de:80
▪ https_proxy=http://www-int2.dkfz-heidelberg.de:80
▪ no_proxy=localhost,127.0.0.1,169.254.169.254,dkfz-heidelberg.de,192.168.49.2,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24
❗ This container is having trouble accessing https://k8s.gcr.io
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
📦 Preparing Kubernetes v1.20.2 on containerd 1.4.3 ...
▪ opt containerd=/var/run/containerd/containerd.sock
▪ env NO_PROXY=localhost,127.0.0.1,169.254.169.254,dkfz-heidelberg.de,192.168.49.2,10.96.0.0/12,192.168.99.0/24,192.168.39.0/24
▪ env HTTP_PROXY=http://www-int2.dkfz-heidelberg.de:80
▪ env HTTPS_PROXY=http://www-int2.dkfz-heidelberg.de:80
🔗 Configuring CNI (Container Networking Interface) ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/gvisor-addon:3
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v4
🔎 Verifying gvisor addon...
🌟 Enabled addons: storage-provisioner, default-storageclass, gvisor
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
adsilva-ivre
commented
3 years ago I0421 16:17:31.481481 46649 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0421 16:17:31.508473 46649 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49182 SSHKeyPath:/home/aviadmin/.minikube/machines/minikube/id_rsa Username:docker}
I0421 16:17:31.508473 46649 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49182 SSHKeyPath:/home/aviadmin/.minikube/machines/minikube/id_rsa Username:docker}
/ I0421 16:17:41.606540 46649 ssh_runner.go:189] Completed: systemctl --version: (10.1251636s)
I0421 16:17:41.606693 46649 ssh_runner.go:149] Run: sudo systemctl is-active --quiet service containerd
I0421 16:17:41.606562 46649 ssh_runner.go:189] Completed: curl -sS -m 2 https://k8s.gcr.io/: (10.1250974s)
W0421 16:17:41.606874 46649 start.go:627] [curl -sS -m 2 https://k8s.gcr.io/] failed: curl -sS -m 2 https://k8s.gcr.io/: Process exited with status 28
stdout:
stderr:
curl: (28) Resolving timed out after 2000 milliseconds
W0421 16:17:41.607062 46649 out.go:222] ❗ This container is having trouble accessing https://k8s.gcr.io
❗ This container is having trouble accessing https://k8s.gcr.io
W0421 16:17:41.607240 46649 out.go:222] 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
I0421 16:17:41.617908 46649 ssh_runner.go:149] Run: sudo systemctl cat docker.service
I0421 16:17:41.624199 46649 cruntime.go:219] skipping containerd shutdown because we are bound to it
I0421 16:17:41.624259 46649 ssh_runner.go:149] Run: sudo systemctl is-active --quiet service crio
I0421 16:17:41.630036 46649 ssh_runner.go:149] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
image-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"same issue with Docker installed in Ubuntu 20.04 in WSL2(Windows)
timscottbell
commented
3 years ago I just started experiencing this problem this morning on Ubuntu 20.04 using vm-driver=docker. What is the workaround?
andriyDev
commented
3 years ago @timscottbell From other comments, this seems to be caused by a VPN extension. Do you have any VPN installed?
danstancu
commented
3 years ago check /etc/docker/daemon.json. if { "iptables": false } set it to true. More info: https://www.mkubaczyk.com/2017/09/05/force-docker-not-bypass-ufw-rules-ubuntu-16-04/
sbollap1
commented
3 years ago I just started experiencing this problem this morning on Ubuntu 20.04 using
vm-driver=docker. What is the workaround?
I am facing the same issue
shivanshuraj1333
commented
3 years ago Same issue for me as well on macos
ILyaCyclone
commented
3 years ago minikube delete minikube start
solved it for me. Apparently, this solution is not very suitable. (Win 10, hyperv)
SanjayDatta
commented
2 years ago Yes. delete minikube container --> remove minikube image --> minikube start made the problem go away. So if there is an existing image, update may be a problem. Hence the error message.
jcostom
commented
2 years ago I was just experiencing this exact problem on macOS 11.6.4 using the freshly installed combination of:
All straight out of Homebrew. The fix? I went into the Security & Privacy System Preferences Panel, Firewall Tab, Firewall Options. I deactivated "Stealth Mode". Everything worked like a charm from that point forward.
(I did try deleting and re-creating the Minikube instance, as suggested above, which didn't do a blessed thing)
teja463
commented
2 years ago Same issue running minikube on Windows 10 with hyperv driver. Below is the Warning I got on starting of the cluster
PS C:\WINDOWS\system32> minikube start minikube v1.25.2 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042 Using the hyperv driver based on existing profile Starting control plane node minikube in cluster minikube Updating the running hyperv "minikube" VM ... This VM is having trouble accessing https://k8s.gcr.io To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ Preparing Kubernetes v1.23.3 on Docker 20.10.12 ... kubelet.housekeeping-interval=5m
medyagh
commented
2 years ago @OkayJosh do you still have this issue?
RichardLee0211
commented
2 years ago I am having the same issue with Ubuntu, couldn't pass get start page and pull any image due to this network issue.
$ minikube start
😄 minikube v1.25.2 on Ubuntu 22.04
✨ Automatically selected the docker driver
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=3900MB) ...
❗ This container is having trouble accessing https://k8s.gcr.io
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
▪ kubelet.housekeeping-interval=5m
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ minikube ssh
docker@minikube:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12287msI used docker driver and it worked fine.
JoshTheDeveloperr
commented
2 years ago I'm on the docker driver and I'm getting this exact issue right now.
rhythmshandlya
commented
2 years ago 
Facing similar issue: Docker : 20.10.17 minikube : 1.26.0 OS: Ubuntu 22.04 LTS
mtsgrd
commented
2 years ago Since there's a few people posting here I'd like to quickly share that I've encountered this error a few times.
ro0NL
commented
2 years ago 🚜 Pulling base image ...
> gcr.io/k8s-minikube/kicbase: 0 B [___________________________] ?% ? p/s ?BUT
$ docker pull gcr.io/k8s-minikube/kicbase:v0.0.32
v0.0.32: Pulling from k8s-minikube/kicbase:shrug:
<insert you had one job here>
tAnboyy
commented
2 years ago Since there's a few people posting here I'd like to quickly share that I've encountered this error a few times.
- A couple of times I didn't notice I had OpenVPN trying to connect to unresponsive server (hidden terminal window)
- Once the problem seemed to be docker, restarting the service helped
- Never happened because of Minikube being broken
Restarting docker did the trick for me. Thanks!
yitzikz
commented
2 years ago Since there's a few people posting here I'd like to quickly share that I've encountered this error a few times.
- A couple of times I didn't notice I had OpenVPN trying to connect to unresponsive server (hidden terminal window)
- Once the problem seemed to be docker, restarting the service helped
- Never happened because of Minikube being broken
Restarting docker did the trick for me. Thanks!
Do you use docker minikube or latest minikube from kubernetes site?
mavericks013
commented
2 years ago I have given up on minikube. Installed 'kind' and everything works like a charm.
tAnboyy
commented
2 years ago Since there's a few people posting here I'd like to quickly share that I've encountered this error a few times.
- A couple of times I didn't notice I had OpenVPN trying to connect to unresponsive server (hidden terminal window)
- Once the problem seemed to be docker, restarting the service helped
- Never happened because of Minikube being broken
Restarting docker did the trick for me. Thanks!
Do you use docker minikube or latest minikube from kubernetes site?
the latter
tankilo
commented
2 years ago I also came across this problem today!
I find that i can pull image on the host machine, but i can't in the minikube docker container.
I didn't find the way to pass the dns settings via minikube start command, so i mannualy change the content of the /etc/resolv.conf in the conatiner to thoses on the host machine. And the problem is solved.
ozymandias89
commented
2 years ago I resolved in Windows with Docker Desktop. Procedure: 1) minikube stop 2) delete minikube docker container 3) minikube start
JustinKirkJohnson
commented
2 years ago I experienced this issue, the root cause was security software (zscalar) running on my workstation that was injecting it's certificate as the root CA for TLS certificates causing a certificate chain validation failure. I was able to determine this by:
docker exec -it 8988ee1de268 "bash"openssl s_client -connect k8s.gcr.io:443The certificate chain that was presented showed the CA and Intermediate CA as being CN = Zscaler Root CA and CN = Zscaler Intermediate Root CA respectively.
I disabled the service on my workstation and voilà, everything worked.
Hondough
commented
2 years ago Since there's a few people posting here I'd like to quickly share that I've encountered this error a few times.
- A couple of times I didn't notice I had OpenVPN trying to connect to unresponsive server (hidden terminal window)
- Once the problem seemed to be docker, restarting the service helped
- Never happened because of Minikube being broken
I had this issue today and it was due to VPN running. I turned off the VPN and stopped and started the cluster. Everything went smoothly. Thank you for saving me much time. I recommend to everybody to try this fix first before rebuilding your installations.
yitzikz
commented
2 years ago I resolved in Windows with Docker Desktop. Procedure:
- minikube stop
- delete minikube docker container
- minikube start
didn't help to me
amitmavgupta
commented
2 years ago Use docker as the driver and it works seamlessly. No point using Hyper-V.
s4pfyr
commented
1 year ago Just found this, for me I could resolve the problem by removing the protocol from the proxy. I.e. I changes http://proxy:80 to just proxy:80
shashwata27
commented
1 year ago I have given up on minikube
brandontan
commented
1 year ago So what's the solution? I am still getting when I run minikube --vm-driver=docker --cni=calico
This container is having trouble accessing https://registry.k8s.io
josvazg
commented
1 year ago I was suffering this error, although I suspect this same error has many variants in different setups. In my case I use: Ubuntu 22.04 with docker 23.0.0 with minikube 1.26.1
I think many of the problems people face here are related to the DNS resolution not working from the minikube container to its gateway in a custom docker network. At least that was what I was seeing, network and internet were reachable using bare IP addresses, but DNS was not.
After lots of Googling, including finding this page, I gave up that but came across this error message in my /var/log/syslog:
dockerd[155770]: time="2023-02-04T22:32:14.303114536+01:00" level=warning msg="[resolver] failed to read from DNS server: 172.17.0.1:53, query: ;k8s.gcr.io.\tIN\t A" error="read udp 172.17.0.1:50487->172.17.0.1:53: read: connection refused"That meant that docker was trying to ask my system-resolved about that DNS resolution and was getting "connection refused".
BTW, my docker + systemd-resolved setup is also a bit special because I use a VPN and I need docker containers to use systemd-resolved to reach the VPN network when available. I managed to trick docker into using systemd-resolved by following this setup:
https://github.com/cohoe/workstation/issues/105
Then I realized my other Ubuntu machine that was working fine had a slightly different config at /etc/docker/daemon.json, using a custom bip:
{
"bip": "172.20.0.1/16",
"dns": ["172.20.0.1"]
}I changed systemd-resolved to do DNSStubListenerExtra=172.20.0.1 instead of 172.17.0.1 and restarted it, then changed /etc/docker/daemon.json as per above and got minikube 1.21.1 working!
If your issue seems also DNS related, it is worth a try to change the docker bip even if you are running on another different Linux or even a Mac. In my case, I did not identify where the conflict with IP 172.17.0.1 came from, but changing to 172.20.0.1 made the trick anyways.
Still, I suspect some other issues with this same error message might differ.
k8s-triage-robot
commented
1 year ago The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/remove-lifecycle stale/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
keithhays69
commented
1 year ago On Ubuntu 22.04, this issue is caused by the Ubuntu firewall being on. I was not unable to figure out exactly what settings it should have to make the deploy work with it on, so I turned it off long enough to do the deploy and then turned it back on and it all worked fine.
k8s-triage-robot
commented
1 year ago The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/remove-lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
k8s-triage-robot
commented
1 year ago The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/reopen/remove-lifecycle rottenPlease send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
k8s-ci-robot
commented
1 year ago @k8s-triage-robot: Closing this issue, marking it as "Not Planned".
sawantnitesh76ns
commented
1 year ago Machine - Ubuntu 20.04 I was also facing same issue. Tried to setup HTTP_PROXY, HTPS_PROXY, NO_PROXY
But then by restarting docker.socket and docke.service it worked fine
Steps:
minikube stop
minikube delete
sudo systemctl restart docker.socket docker.service
cwegener
commented
8 months ago In case anyone is running into this issue with the podman rootless driver:
You'll need to make sure you have aardvark-dns installed.
Minutis
commented
8 months ago In case anyone is running into this issue with the Ubuntu WSL on Windows: Adding these two lines in /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4And then:
sudo systemctl daemon-reload
sudo systemctl restart docker
pratikparshetti
commented
7 months ago Machine - Windows driver - docker
I'm getting this as a warning - "This container is having trouble accessing https://registry.k8s.io" when I run minikube start but minikube is getting started. But I'm not able to enable the ingress addons and I suspect that its because I'm not able to pull the image from registry.
I even tried to pull the images manually and then try further but still it did not help.
Can someone suggest any possible solution?
Steps to reproduce the issue:
Full output of failed command:
Full output of
minikube startcommand used, if not already included:Deleting "minikube" in docker ... 🔥 Deleting container "minikube" ... 🔥 Removing /home/cloudsigma/.minikube/machines/minikube ... 💀 Removed all traces of the "minikube" cluster. [cloudsigma@Fedora-32 django]$ minikube start --driver=docker --image-repository=auto 😄 minikube v1.15.1 on Fedora 32 ✨ Using the docker driver based on user configuration ✅ Using image repository 👍 Starting control plane node minikube in cluster minikube 🔥 Creating docker container (CPUs=2, Memory=2200MB) ... ❗ This container is having trouble accessing https://k8s.gcr.io 💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ 🐳 Preparing Kubernetes v1.19.4 on Docker 19.03.13 ... 🔎 Verifying Kubernetes components... 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
Optional: Full output of
minikube logscommand: