Open makcumka2000 opened 6 years ago
I have exactly the same issue
Same here, had that issue a couple month ago alrady and just tried again after upgrading all components it is still there:
Interesting - it sounds like kube-dns is effectively broken within minikube. https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ has some tips on how to debug internal DNS service issues. Do you mind sharing the output of:
kubectl get svc --namespace=kube-system
kubectl get ep kube-dns --namespace=kube-system
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c kubedns
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c dnsmasq
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c sidecar
Thanks!
Also, please check if the host (Mac OS) has anything listening on port 53 already, or specifically, if dnsmasq is running.
In my case, I have something listening on :53
and this process running:
/usr/local/opt/dnsmasq/sbin/dnsmasq --keep-in-foreground
Update, fix to my issue: dnsmasq was running and ignoring the dns request, since it was not listening to the right local addresses
If you are using dnsmasq in your setup and cluster creation fails (stuck at kube-dns initialization) you might need to add
listen-address=127.0.0.1,192.168.64.1
todnsmasq.conf
.
Note: If dnsmasq.conf
contains listen-address=127.0.0.1
kubernetes discovers dns at 127.0.0.1:53 and tries to use it using bridge ip address, but dnsmasq replies only to reqests from 127.0.0.1
My /usr/local/etc/dnsmasq.conf
looks like that:
address=/kube.local/192.168.64.1
listen-address=127.0.0.1,192.168.64.1
Also, please check if the host (Mac OS) has anything listening on port 53 already, or specifically, if dnsmasq is running.
Yep, I am using dnsmasq on local host, it is neccesary for routing *.kube hosts to minikube
It seems there is progress ongoing .. there is an open PR with a workaround: https://github.com/kubernetes/minikube/pull/3003/files
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver
there is a problem with the current solution, dnsmasq will fail to start when minikube is not running, 192.168.64.1 does not exist
finally I decided to stop my dnsmasq when running minikube, this is a easy way
I am using minikube using the hyperkit driver on macOS and experiencing the same problems with dnscrypt-proxy, which listens on 127.0.0.1:53 (udp+tcp). This binary is part of the Cisco Umbrella (corporate OpenDNS) roaming client. I cannot modify the configuration used by dnscrypt-proxy. The dnsmasq.conf changes linked are insufficient to resolve the issue for others.
Is there no way to apply a persistent DNS configuration change to minikube and the system pods?
~
β― sudo lsof -i :53
Password:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dnscrypt- 44992 nobody 7u IPv4 0x94dd190be9fa26c5 0t0 UDP localhost:domain
dnscrypt- 44992 nobody 8u IPv4 0x94dd190bff44e9ed 0t0 TCP localhost:domain (LISTEN)
~
β― ps ax 44992
PID TT STAT TIME COMMAND
44992 ?? Ss 0:02.69 /Library/Application Support/OpenDNS Roaming Client/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/Library/Application Support/OpenDNS Roaming Client/libdcplugin_erc.so -d
~
β― minikube ssh -- 'cat /etc/resolv.conf; echo "Ping by Name:" && ping -v -c 4 google.com; echo "Ping by IP:" && ping -c 4 8.8.8.8'
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 192.168.64.1
Ping by Name:
ping: bad address 'google.com'
Ping by IP:
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=51 time=23.489 ms
64 bytes from 8.8.8.8: seq=1 ttl=51 time=25.994 ms
64 bytes from 8.8.8.8: seq=2 ttl=51 time=27.537 ms
64 bytes from 8.8.8.8: seq=3 ttl=51 time=25.740 ms
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 23.489/25.690/27.537 ms
DNSMASQ. followed recommendations: https://github.com/kubernetes/minikube/issues/2456 https://minikube.sigs.k8s.io/docs/reference/drivers/hyperkit/
I have disabled dnsmasq and deleted and started minikube over..
I get the same logging issue.
It seems like the bridge that is reachable from outside the minikube:
64 bytes from 192.168.64.1: icmp_seq=0 ttl=64 time=0.052 ms
is not reachable from inside
minikube ssh
#traceroute 192.168.64.1
traceroute to 192.168.64.1 (192.168.64.1), 30 hops max, 46 byte packets
1 * * *
this is also what coredns sees.
$ kubectl logs --namespace=kube-system coredns-5c98db65d4-9vpk8
.:53
2019-09-10T23:29:49.664Z [INFO] CoreDNS-1.3.1
2019-09-10T23:29:49.665Z [INFO] linux/amd64, go1.11.4, 6b56a9c
CoreDNS-1.3.1
linux/amd64, go1.11.4, 6b56a9c
2019-09-10T23:29:49.665Z [INFO] plugin/reload: Running configuration MD5 = 5d5369fbc12f985709b924e721217843
2019-09-10T23:29:55.669Z [ERROR] plugin/errors: 2 6233357733929179938.7507841895618823623. HINFO: read udp 172.17.0.2:33644->192.168.64.1:53: i/o timeout
2019-09-10T23:29:58.669Z [ERROR] plugin/errors: 2 6233357733929179938.7507841895618823623. HINFO: read udp 172.17.0.2:48448->192.168.64.1:53: i/o timeout
ram9:etc/ $ minikube version [17:01:12]
minikube version: v1.3.1
commit: ca60a424ce69a4d79f502650199ca2b52f29e631
ram9:etc/ $ kubectl version [17:01:15]
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
ls -la /usr/local/bin/*hyperkit* [17:18:01]
-rwsr-xr-x 1 root wheel 26811068 Apr 20 2018 /usr/local/bin/docker-machine-driver-hyperkit
lrwxr-xr-x 1 root admin 67 Sep 10 15:05 /usr/local/bin/hyperkit -> /Applications/Docker.app/Contents/Resources/bin/com.docker.hyperkit
$ sudo lsof -i :53 [17:18:34]
Password:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mDNSRespo 205 _mdnsresponder 64u IPv6 0x1e8f1c64183077e5 0t0 UDP *:domain
mDNSRespo 205 _mdnsresponder 66u IPv6 0x1e8f1c6416ac90cd 0t0 TCP *:domain (LISTEN)
trying: new version of hyperkit
brew reinstall hyperkit # force link instructions
curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit && sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/
Using the newest version of the hyper kit the dns resolution is solved enough that i can launch busybox. I was also able to install tiller. I am not able to re-enable dnsmasq.
hyperkit: 0.20190802
This is still a thing, but at least it's documented.
I suppose we could have a driver pre-run check that sniffs out if something is listening on the port already. It'd be nice to compare the lsof output for a working vs non-working configuration.
I haven't heard anyone running into this issue in a while. Anyone else?
I has run into this issue and it take me a while to fix that. This happends actually only with dnsmasq (or any other dns service) locally installed.
Here is my working dnsmasq.conf for anyone who may get into this error, too. Ofc you need to restart dnsmasq after adding the listen-address.
address=/.localhost/127.0.0.1
listen-address=127.0.0.1
listen-address=192.168.64.1
We're hitting this issue and also have an Umbrella / dnscrypt-proxy setup we can't change.
As a possible workaround, it appears that using Docker Desktop's vpnkit
, e.g. minikube start ... --hyperkit-vpnkit-sock=/Users/zarenner/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock
works, correctly forwarding DNS requests to dnscrypt-proxy. But it's not really ideal since we'd rather not also have Docker Desktop installed and running in addition to minikube. I'm not sure if running a separate vpnkit instance would make more sense.
Discovered that while possible, a separate vpnkit instance is difficult to set up without Docker Desktop due to lack of standalone vpnkit and vmnetd binaries.
Currently facing the same issue on linux with dnsmasq
Currently facing the same issue in macOs 18.7.0 Darwin Kernel Version 18.7.0: Sun Dec 1 18:59:03 PST 2019; root:xnu-4903.278.19~1/RELEASE_X86_64 x86_64
Hey folks, you need to check the IP Adresses and the DNS resolution. This issue is mostly based on wrong proxy or dns settings. Just make sure, that the routing and dns is correct and everything will be fine.
If you have dnsmasq, you need to extend your configuration like here
Any other issue should be traced back.
I can confirm that @zarenner's solution works with adding a line for listen-address for 192.168.64.1
I has run into this issue and it take me a while to fix that. This happends actually only with dnsmasq (or any other dns service) locally installed.
Here is my working dnsmasq.conf for anyone who may get into this error, too. Ofc you need to restart dnsmasq after adding the listen-address.
address=/.localhost/127.0.0.1 listen-address=127.0.0.1 listen-address=192.168.64.1
I did that and also had to restart dnsmasq
to make it work (for macOS):
sudo launchctl stop homebrew.mxcl.dnsmasq
sudo launchctl start homebrew.mxcl.dnsmasq
Had the same issue when starting minikube with hyperkit. There was a dnscrypt-proxy, also corporate solution, running on 53 port
/Library/Application Support/OpenDNS Roaming Client/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/Library/Application Support/OpenDNS Roaming Client/libdcplugin_erc.so -d
Restarting after
sudo launchctl unload /Library/LaunchDaemons/com.opendns.osx.RoamingClientConfigUpdater.plist
works just fine for me, however, not ideal still
We're hitting this issue and also have an Umbrella / dnscrypt-proxy setup we can't change.
As a possible workaround, it appears that using Docker Desktop's
vpnkit
, e.g.minikube start ... --hyperkit-vpnkit-sock=/Users/zarenner/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock
works, correctly forwarding DNS requests to dnscrypt-proxy. But it's not really ideal since we'd rather not also have Docker Desktop installed and running in addition to minikube. I'm not sure if running a separate vpnkit instance would make more sense.
we should add this to our hyperkit driver documentation
I am looking for someone to evaluate all these workarrounds and put them in the hyperkit driver documentation.
@medyagh Can I do this issue? I guess I can contribute for editing documentation even if I'm a newbie with k8s and minikube.
But I'm wondering is it a good way to add zarenner's comment in the documentation since I encountered this problem with cloudflared DNS and it's not a solution for cloudflared. (minikube start ... --hyperkit-vpnkit-sock=/Users/zarenner/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock
)
We'd be happy to review any PR you submit.
In my case, I have something listening on
:53
and this process running:/usr/local/opt/dnsmasq/sbin/dnsmasq --keep-in-foreground
Update, fix to my issue: dnsmasq was running and ignoring the dns request, since it was not listening to the right local addresses
If you are using dnsmasq in your setup and cluster creation fails (stuck at kube-dns initialization) you might need to add
listen-address=127.0.0.1,192.168.64.1
todnsmasq.conf
.Note: If
dnsmasq.conf
containslisten-address=127.0.0.1
kubernetes discovers dns at 127.0.0.1:53 and tries to use it using bridge ip address, but dnsmasq replies only to reqests from 127.0.0.1My
/usr/local/etc/dnsmasq.conf
looks like that:address=/kube.local/192.168.64.1 listen-address=127.0.0.1,192.168.64.1
Unfortunately these lines were not helpful in my case. But slightly modify them it is:
server=/kube.local/192.168.64.1
listen-address=127.0.0.1,192.168.64.1
Both directive are important. The docs say that listen-address
directive is enough but in my case it start working only with server
directive.
Hope it will be helpful to someone.
This seems to help in the meantime:
sudo socat UDP4-RECVFROM:53,bind=192.168.64.1,fork UDP4-SENDTO:127.0.0.1:53
Apart from having that line listen-address=127.0.0.1,192.168.64.1
the second one also seems to be crucial address=/.localhost/127.0.0.1
If that does not work check what network is really being used under the hood by minikube (via ifconfig)
Hey @bartkim0426! Are you still interested in this issue?
I was just investigating another DNS-related issue on my colleagues OS X High Sieera, and it turns out, it's really terrible.
He has a mDNSresponder service occupying *:53/UDP. It does not actually respond to any of the requests made by the minikube VM. So, DNS is effectively broken for the VM, using HyperKit. We did not know about the DNSresponder thingie, since we installed dnsmasq, and began using it. Eventually, we realized it's failing to start exactly because it cannot bind.
I think this is yet one more example of horrific engineering on Apple's side. I am not saying everything they do is garbag, but WHY on earth would you have a service that (according to Google) is there to support Bonjour (kind of a discorery protocol) occupiying *:53???
If there was at least a way how to tell it not to occupy ALL the ip addresses, such that actually useful software could listen on them instead :)
EDIT: You could possibly maybe tell Minikube to use a different IP for the DNS resolution, as routing seems to be working, but if you need that IP to be dnsmasq, you will have to host ti likely on a different VM, not the OS X :(((
@petr-motejlek How can you be wrong so much? My mac runs Catalina!
our current integration machines are out of work due to covid (they were in the office) we are looking into adding mac integration tests, once we have integration back, I would accept a PR that adds integration tests that fixes this issue
I've upped the priority based on the number of votes. Help wanted!
I imagine a lot of people will have this type of problem now that the docker desktop issue is forcing our hands. Here's my error to add to the list; hope its helpful. It seems like its' saying the same thing everyone else is saying. minikube start --driver=hyperkit --container-runtime=docker π minikube v1.24.0 on Darwin 11.4 β¨ Using the hyperkit driver based on existing profile π Starting control plane node minikube in cluster minikube π Updating the running hyperkit "minikube" VM ... β The image 'k8s.gcr.io/kube-proxy:v1.23.2' was not found; unable to add it to cache. β The image 'k8s.gcr.io/kube-controller-manager:v1.23.2' was not found; unable to add it to cache. β The image 'k8s.gcr.io/kube-scheduler:v1.23.2' was not found; unable to add it to cache. β The image 'k8s.gcr.io/kube-apiserver:v1.23.2' was not found; unable to add it to cache. β This VM is having trouble accessing https://k8s.gcr.io π‘ To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/ π³ Preparing Kubernetes v1.23.2 on Docker 20.10.8 ... β Unable to load cached images: loading cached images: stat /Users/kobebryant/.minikube/cache/images/k8s.gcr.io/kube-controller-manager_v1.23.2: no such file or directory
β Exiting due to K8S_INSTALL_FAILED: updating control plane: downloading binaries: downloading kubeadm: download failed: https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm.sha256: getter: &{Ctx:context.Background Src:https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm?checksum=file:https://storage.googleapis.com/kubernetes-release/release/v1.23.2/bin/linux/amd64/kubeadm.sha256 Dst:/Users/kobebryant/.minikube/cache/linux/v1.23.2/kubeadm.download Pwd: Mode:2 Umask:---------- Detectors:[0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8 0x70f4bb8] Decompressors:map[bz2:0x70f4bb8 gz:0x70f4bb8 tar:0x70f4bb8 tar.bz2:0x70f4bb8 tar.gz:0x70f4bb8 tar.xz:0x70f4bb8 tar.zst:0x70f4bb8 tbz2:0x70f4bb8 tgz:0x70f4bb8 txz:0x70f4bb8 tzst:0x70f4bb8 xz:0x70f4bb8 zip:0x70f4bb8 zst:0x70f4bb8] Getters:map[file:0xc001177190 http:0xc000435900 https:0xc000435920] Dir:false ProgressListener:0x70b6540 Insecure:false Options:[0x5452be0]}: invalid checksum: Error downloading checksum file: bad response code: 404
In my case I had unused bind9 aside dnsmasq coming with NetworkManager
To solve it, did:
sudo apt purge bind9
sudo systemctl restart network-manager
sudo systemctl restart Network-Manager
xxx:~ $ sudo lsof -i :53
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mDNSRespo 70594 _mdnsresponder 52u IPv6 0x33aa4c2d2a962605 0t0 UDP *:domain
mDNSRespo 70594 _mdnsresponder 53u IPv4 0x33aa4c2d2a5abe4d 0t0 TCP *:domain (LISTEN)
mDNSRespo 70594 _mdnsresponder 54u IPv6 0x33aa4c2d244e9b7d 0t0 TCP *:domain (LISTEN)
looks like in my case it's mDNSResponder
that's using port 53?
As a workaround, following change on running minikube
may be done:
minikube ssh -- 'echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf'
Instead of 1.1.1.1
could be any DNS server setup.
Update: above workaround looks temporary, more persistent workaround here: https://github.com/kubernetes/minikube/issues/13497#issuecomment-1022413061
I was having the same issue on OS X running minikube
with also having dnscrypt-proxy
which config I can't change.
Adding this to my /etc/pf.conf
seems to have fixed it (bridge100 is the hyperkit created bridge):
rdr on bridge100 proto {tcp,udp} from any to bridge100 port 53 -> 127.0.0.1 port 53
@nikolay-te Discovered that pf is angry if you don't put the rule in the right order:
sudo pfctl -f /etc/pf.conf
/etc/pf.conf:24: Rules must be in order: options, normalization, queueing, translation, filtering
So I: /etc/pf.conf
set limit {tables 10000, table-entries 400000}
scrub-anchor "cisco.anyconnect.vpn"
scrub-anchor "com.apple/*"
nat-anchor "com.apple/*"
rdr-anchor "com.apple/*"
rdr on bridge100 inet proto {tcp,udp} from any to bridge100 port 53 -> 127.0.0.1 port 53
dummynet-anchor "com.apple/*"
anchor "cisco.anyconnect.vpn"
anchor "com.apple/*"
load anchor "com.apple" from "/etc/pf.anchors/com.apple"
sudo pfctl -f /etc/pf.conf
I'm on a Mac that's smothered with Cisco Umbrella sauce. BTW, I can't run minikube AT ALL unless I disconnect from my corporate VPN
This allowed my minikube VM to hit my corporate private registry url via Umbrella's DNS
I also have dnscrypt-proxy running:
ps aux | grep crypt
nobody 78236 0.0 0.0 5928120 4492 ?? SN 11:53AM 0:00.78 /opt/cisco/anyconnect/bin/dnscrypt-proxy --user nobody --local-address=127.0.0.1:53 --plugin=/opt/cisco/anyconnect/lib/libdcplugin_erc.so -d
Can someone on this thread that doesn't have "Cisco Umbrella sauce" (sadly, that includes me) do a sudo lsof -i :53
, and post the output?
In particular, I'm wondering if minikube's DNS server, whatever that is, is trying to bind to all interfaces, instead of just the bridge. If I'm looking at this right, I'm wondering why there should be a conflict here: Umbrella only binds to localhost
, so a bind to the bridge shouldn't be affected by it. But, if minikube were instead trying to bind to all interfaces, then that would be a different story. The lsof
from a normal setup should help answer that.
Can make this work with driver 'hyperkit' on MAC
Post that in the minikube /etc/resolv.conf we will see a new entry of nameserver 8.8.8.8 along with 192.168.64.1
$ cat /etc/resolv.conf | grep name nameserver 8.8.8.8 nameserver 192.168.64.1
NOTE : If we directly edit file "/etc/resolv.conf" with 8.8.8.8, It will revert back to old setting since its been maintained by the systemd-resolved.service .
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT
Please provide the following details:
Environment: MacOS High Sierra
Minikube version (use
minikube version
): 0.25.0 and abovecat ~/.minikube/machines/minikube/config.json | grep DriverName
): hyperkitcat ~/.minikube/machines/minikube/config.json | grep -i ISO
orminikube ssh cat /etc/VERSION
): .minikube/cache/iso/minikube.25.0.0.isoWhat happened: dns in minikube does not work. DNS cluster can not start, because can not download images from registry. 192.168.64.1 - it is my host IP
What you expected to happen: DNS works. If I change DNS server in minikube to 8.8.8.8, it works.
How to reproduce it (as minimally and precisely as possible):
Output of
minikube logs
(if applicable):Anything else do we need to know: