Closed reinvantveer closed 3 years ago
reinvantveer
commented
3 years ago Just tried: fiddling with the Network Manager doesn't help unfortunately:
❯ echo "server=/test/$(minikube ip)" | sudo tee /etc/NetworkManager/dnsmasq.d/minikube.conf
server=/test/192.168.49.2
❯ sudo systemctl restart NetworkManager.service
❯ ping hello-john.test
ping: hello-john.test: Name or service not known
seanrmurphy
commented
3 years ago I've been looking at something similar this morning and managed to get the DNS resolution working (I think) but there is an issue with the port config as far as I can see - basically ping works but curl does not.
I'm on Ubuntu Focal, using minikube 1.17.1 with ingress-dns - I'm using systemd-resolved.
My approach was the following:
/etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf/etc/resolv.conf with contents below/run/systemd/resolve/resolv.conf to /etc/resolv.confContents of /etc/resolv.conf
ubuntu@test:/etc$ minikube ip
192.168.49.2
ubuntu@test:/etc$ cat /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search openstacklocal
search test
nameserver 192.168.49.2
timeout 5
I was then able to install the test application and do the following:
ubuntu@test:/etc$ nslookup hello-jane.test
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: hello-jane.test
Address: 192.168.49.2
ubuntu@test:/etc$ ping hello-jane.test
PING hello-jane.test (192.168.49.2) 56(84) bytes of data.
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=1 ttl=64 time=0.078 ms
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=2 ttl=64 time=0.096 ms
^C
--- hello-jane.test ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.078/0.087/0.096/0.009 ms
ubuntu@test:/etc$ curl hello-jane.test
curl: (7) Failed to connect to hello-jane.test port 80: Connection refused
ubuntu@test:/etc$
It seems I also needed to enable the ingress addon - with this, the example works in my environment.
ubuntu@test:~$ curl hello-jane.test
Hello, world!
Version: 1.0.0
Hostname: hello-world-app-7b9bf45d65-gvrtk
ubuntu@test:~$ Sorry @seanrmurphy I can't reproduce your steps, I'm on Focal as well. Could I trouble you for the commands you issued? Here's my attempt:
echo "Exposing apps to host"
LOCAL_DEV_DOMAIN=test
sudo service systemd-resolved stop
sudo rm /etc/resolv.conf
echo Adding ${LOCAL_DEV_DOMAIN} host to /etc/resolv.conf
echo nameserver 127.0.0.53 | sudo tee -a /etc/resolv.conf
echo options edns0 trust-ad | sudo tee -a /etc/resolv.conf
echo search openstacklocal | sudo tee -a /etc/resolv.conf
echo "" | sudo tee -a /etc/resolv.conf
echo "search ${LOCAL_DEV_DOMAIN}" | sudo tee -a /etc/resolv.conf
echo "nameserver $(minikube ip)" | sudo tee -a /etc/resolv.conf
echo "timeout 5" | sudo tee -a /etc/resolv.conf
sudo ln -nsf /run/systemd/resolve/resolv.conf /etc/resolv.conf
sudo service systemd-resolved start
echo deploying test application to verify ingress
kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/8209421c3e5ef0af4ca92a30da25b5e75c9255bc/deploy/addons/ingress-dns/example/example.yaml # Use a k8s 1.18 beta networking version
ping hello-john.test
kubectl delete -f https://raw.githubusercontent.com/kubernetes/minikube/8209421c3e5ef0af4ca92a30da25b5e75c9255bc/deploy/addons/ingress-dns/example/example.yamlBut this fails on ping hello-john.test, because when I list /etc/resolv.conf it just says:
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 192.168.1.1So on a sudo service systemd-resolved start it overwrites the configuration I just wrote. Even if I remove both...
PS I tried variations of
sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolvedfollowed by
sudo systemctl restart NetworkManager
sudo systemctl start systemd-resolved
sudo systemctl enable systemd-resolvedbut it just keeps the nameserver 192.168.1.1
seanrmurphy
commented
3 years ago Sure - this worked for me:
ubuntu@deploy:~$ minikube start
😄 minikube v1.17.1 on Ubuntu 20.04 (amd64)
✨ Using the docker driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing docker container for "minikube" ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.2 ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
ubuntu@deploy:~$ cd /etc/
ubuntu@deploy:/etc$ head resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
ubuntu@deploy:/etc$ sudo systemctl stop systemd-resolved
ubuntu@deploy:/etc$ sudo rm resolv.conf
sudo: unable to resolve host deploy: Temporary failure in name resolution
ubuntu@deploy:/etc$ minikube ip
192.168.49.2
ubuntu@deploy:/etc$ sudo vi resolv.conf
sudo: unable to resolve host deploy: Temporary failure in name resolution
ubuntu@deploy:/etc$ cat resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search openstacklocal
search test
nameserver 192.168.49.2
timeout 5
ubuntu@deploy:/etc$ sudo rm /run/systemd/resolve/resolv.conf
sudo: unable to resolve host deploy: Temporary failure in name resolution
ubuntu@deploy:/etc$ sudo ln -s /etc/resolv.conf /run/systemd/resolve/resolv.conf
sudo: unable to resolve host deploy: Temporary failure in name resolution
ubuntu@deploy:/etc$ sudo systemctl restart systemd-resolved
sudo: unable to resolve host deploy: Temporary failure in name resolution
ubuntu@deploy:/etc$ ping www.google.com
PING www.google.com (172.217.168.36) 56(84) bytes of data.
64 bytes from zrh04s14-in-f4.1e100.net (172.217.168.36): icmp_seq=1 ttl=112 time=3.93 ms
C64 bytes from zrh04s14-in-f4.1e100.net (172.217.168.36): icmp_seq=2 ttl=112 time=3.98 ms
64 bytes from zrh04s14-in-f4.1e100.net (172.217.168.36): icmp_seq=3 ttl=112 time=3.90 ms
^C
--- www.google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 3.899/3.935/3.978/0.032 ms
ubuntu@deploy:/etc$ minikube addons enable ingress
🔎 Verifying ingress addon...
🌟 The 'ingress' addon is enabled
ubuntu@deploy:/etc$ minikube addons enable ingress-dns
🌟 The 'ingress-dns' addon is enabled
ubuntu@deploy:/etc$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml
deployment.apps/hello-world-app created
ingress.networking.k8s.io/example-ingress created
service/hello-world-app created
service/hello-world-app created
ubuntu@deploy:/etc$ ping hello-jane.test
PING hello-jane.test (192.168.49.2) 56(84) bytes of data.
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=2 ttl=64 time=0.078 ms
^C
--- hello-jane.test ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.068/0.073/0.078/0.005 ms
ubuntu@deploy:/etc$ ^C
ubuntu@deploy:/etc$ ^C
ubuntu@deploy:/etc$ curl hello-jane.test
Hello, world!
Version: 1.0.0
Hostname: hello-world-app-7b9bf45d65-zfcs4
ubuntu@deploy:/etc$ Note that some of the content of resolv.conf above is picked up from our openstack cluster (the domain openstacklocal and the trust-ad stuff - you should basically copy what is there from before and add the clause around the new domain with the minikube ip acting as the nameserver.
Of course this would only work if you were using systemd-resolved for name resolution in the first instance.
Hope that helps!
reinvantveer
commented
3 years ago Yes!!! Thanks ever so much.
ping -c 5 hello-john.geodan.dev
PING hello-john.geodan.dev (192.168.49.2) 56(84) bytes of data.
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=1 ttl=64 time=0.141 ms
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=2 ttl=64 time=0.090 ms
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=3 ttl=64 time=0.088 ms
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=4 ttl=64 time=0.096 ms
64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=5 ttl=64 time=0.087 ms
--- hello-john.geodan.dev ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4068ms
rtt min/avg/max/mdev = 0.087/0.100/0.141/0.020 ms
curl hello-john.geodan.dev
Hello, world!
Version: 1.0.0
Hostname: hello-world-app-5f5d8b66bb-67pd7Now, for some documentation fixing of the ingress-dns plugin I guess...
reinvantveer
commented
3 years ago OK, fair warning: I rebooted my system this morning and my networking is broken, it seems I shouldn't have disabled the network manager. I need to do some more figuring out.
reinvantveer
commented
3 years ago OK, fair warning: I rebooted my system this morning and my networking is broken, it seems I shouldn't have disabled the network manager. I need to do some more figuring out.
Just restoring the network manager was enough, don't know exactly where I got the advice to disable it but it was a very bad idea :laughing:
shreyasshivakumara
commented
3 years ago Hello, I am facing the same issue as @reinvantveer while starting the ingress DNS. Is there anything wrong with the Minikube version or am I doing wrong? Seems like the /etc/resolv.conf is not updating after overwriting it.
`shreyas@ubuntu-machine:/etc$ minikube start
😄 minikube v1.23.2 on Ubuntu 20.04
✨ Automatically selected the docker driver. Other choices: none, ssh
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=7800MB) ...
🐳 Preparing Kubernetes v1.22.2 on Docker 20.10.8 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
(base) shreyas@ubuntu-machine:/etc$ head resolv.conf
This file is managed by man:systemd-resolved(8). Do not edit.
This is a dynamic resolv.conf file for connecting local clients directly to
all known uplink DNS servers. This file lists all configured search domains.
Third party programs must not access this file directly, but only through the
symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
replace this symlink by a static file or a different symlink.
See man:systemd-resolved.service(8) for details about the supported modes of
nameserver 8.8.8.8
nameserver 130.236.1.9
nameserver 130.236.1.10
nameserver 130.236.1.11
search test
nameserver 192.168.49.2
timeout 5
(base) shreyas@ubuntu-machine:/etc$ sudo systemctl stop systemd-resolved
(base) shreyas@ubuntu-machine:/etc$ sudo rm resolv.conf
(base) shreyas@ubuntu-machine:/etc$ sudo nano resolv.conf
(base) shreyas@ubuntu-machine:/etc$ sudo rm /run/systemd/resolve/resolv.conf
(base) shreyas@ubuntu-machine:/etc$ sudo ln -s /etc/resolv.conf /run/systemd/resolve/resolv.conf
(base) shreyas@ubuntu-machine:/etc$ sudo systemctl restart systemd-resolved
(base) shreyas@ubuntu-machine:/etc$ ping www.google.com
PING www.google.com (142.250.74.100) 56(84) bytes of data.
64 bytes from arn11s10-in-f4.1e100.net (142.250.74.100): icmp_seq=1 ttl=53 time=7.21 ms
64 bytes from arn11s10-in-f4.1e100.net (142.250.74.100): icmp_seq=2 ttl=53 time=7.85 ms
64 bytes from arn11s10-in-f4.1e100.net (142.250.74.100): icmp_seq=3 ttl=53 time=7.76 ms
(base) shreyas@ubuntu-machine:/etc$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 130.236.1.9
nameserver 130.236.1.10
nameserver 130.236.1.11
(base) shreyas@ubuntu-machine:/etc$ minikube addons enable ingress
▪ Using image k8s.gcr.io/ingress-nginx/controller:v1.0.0-beta.3
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
▪ Using image k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0
🔎 Verifying ingress addon...
🌟 The 'ingress' addon is enabled
(base) shreyas@ubuntu-machine:/etc$ minikube addons enable ingress-dns
▪ Using image gcr.io/k8s-minikube/minikube-ingress-dns:0.0.1
🌟 The 'ingress-dns' addon is enabled
(base) shreyas@ubuntu-machine:/etc$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml
deployment.apps/hello-world-app created
ingress.networking.k8s.io/example-ingress created
service/hello-world-app created
service/hello-world-app created
(base) shreyas@ubuntu-machine:/etc$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-world-app-7b9bf45d65-9hlxg 1/1 Running 0 18s
(base) shreyas@ubuntu-machine:/etc$ ping hello-jane.test
ping: hello-jane.test: Name or service not known
(base) shreyas@ubuntu-machine:/etc$ nslookup hello-jane.test
Server: 130.236.1.9
Address: 130.236.1.9#53
** server can't find hello-jane.test: NXDOMAIN
Hello, I am facing the same issue as @reinvantveer while starting the ingress DNS. Is there anything wrong with the Minikube version or am I doing wrong? Seems like the /etc/resolv.conf is not updating after overwriting it.
Yeah I discussed this with a few colleagues the other day and we agreed that messing with /etc/resolv.conf is not the way forward here. There has to be a better way. So I would advise against using resolv.conf as a method. Configuring minikube access should not have to include root-required operations on system resources that are supposed to be managed by other services.
reinvantveer
commented
3 years ago @shreyasshivakumara If you like, you can reopen the issue. I feel there should be something available that does not require root rights to and meddling with /etc/resolv.conf to make this work...
Nieto2018
commented
2 years ago Sure - this worked for me:
ubuntu@deploy:~$ minikube start 😄 minikube v1.17.1 on Ubuntu 20.04 (amd64) ✨ Using the docker driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🔄 Restarting existing docker container for "minikube" ... 🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.2 ... 🔎 Verifying Kubernetes components... 🌟 Enabled addons: default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default ubuntu@deploy:~$ cd /etc/ ubuntu@deploy:/etc$ head resolv.conf # This file is managed by man:systemd-resolved(8). Do not edit. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "resolvectl status" to see details about the uplink DNS servers # currently in use. # # Third party programs must not access this file directly, but only through the ubuntu@deploy:/etc$ sudo systemctl stop systemd-resolved ubuntu@deploy:/etc$ sudo rm resolv.conf sudo: unable to resolve host deploy: Temporary failure in name resolution ubuntu@deploy:/etc$ minikube ip 192.168.49.2 ubuntu@deploy:/etc$ sudo vi resolv.conf sudo: unable to resolve host deploy: Temporary failure in name resolution ubuntu@deploy:/etc$ cat resolv.conf nameserver 127.0.0.53 options edns0 trust-ad search openstacklocal search test nameserver 192.168.49.2 timeout 5 ubuntu@deploy:/etc$ sudo rm /run/systemd/resolve/resolv.conf sudo: unable to resolve host deploy: Temporary failure in name resolution ubuntu@deploy:/etc$ sudo ln -s /etc/resolv.conf /run/systemd/resolve/resolv.conf sudo: unable to resolve host deploy: Temporary failure in name resolution ubuntu@deploy:/etc$ sudo systemctl restart systemd-resolved sudo: unable to resolve host deploy: Temporary failure in name resolution ubuntu@deploy:/etc$ ping www.google.com PING www.google.com (172.217.168.36) 56(84) bytes of data. 64 bytes from zrh04s14-in-f4.1e100.net (172.217.168.36): icmp_seq=1 ttl=112 time=3.93 ms C64 bytes from zrh04s14-in-f4.1e100.net (172.217.168.36): icmp_seq=2 ttl=112 time=3.98 ms 64 bytes from zrh04s14-in-f4.1e100.net (172.217.168.36): icmp_seq=3 ttl=112 time=3.90 ms ^C --- www.google.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 3.899/3.935/3.978/0.032 ms ubuntu@deploy:/etc$ minikube addons enable ingress 🔎 Verifying ingress addon... 🌟 The 'ingress' addon is enabled ubuntu@deploy:/etc$ minikube addons enable ingress-dns 🌟 The 'ingress-dns' addon is enabled ubuntu@deploy:/etc$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml deployment.apps/hello-world-app created ingress.networking.k8s.io/example-ingress created service/hello-world-app created service/hello-world-app created ubuntu@deploy:/etc$ ping hello-jane.test PING hello-jane.test (192.168.49.2) 56(84) bytes of data. 64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=1 ttl=64 time=0.068 ms 64 bytes from 192.168.49.2 (192.168.49.2): icmp_seq=2 ttl=64 time=0.078 ms ^C --- hello-jane.test ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 0.068/0.073/0.078/0.005 ms ubuntu@deploy:/etc$ ^C ubuntu@deploy:/etc$ ^C ubuntu@deploy:/etc$ curl hello-jane.test Hello, world! Version: 1.0.0 Hostname: hello-world-app-7b9bf45d65-zfcs4 ubuntu@deploy:/etc$Note that some of the content of
resolv.confabove is picked up from our openstack cluster (the domainopenstacklocaland thetrust-adstuff - you should basically copy what is there from before and add the clause around the new domain with the minikube ip acting as the nameserver.Of course this would only work if you were using systemd-resolved for name resolution in the first instance.
Hope that helps!
Hi @seanrmurphy,
Thanks, this solution works temporarily for me, but If I reboot my computer, or the services NetWorkManage/systemd-resolved or change my wifi connection to a different network, the file /run/systemd/resolve/resolv.conf is overwritten and It doesn't work. I tried different solutions others don't work or work temporarily.
Finally, I found a definitive solution that works with Ubuntu 22.04 and continues working after rebooting too. The documentation for ingress-dns, the "Linux OS with Network Manager" section seems to need a first step, this step is to check the /etc/resolv.conf is a symbolic link to the /run/NetworkManager/resolv.conf file (In ubuntu 22.04, it is a symbolic link to /run/systemd/resolve/stub-resolv.conf file as default), if it isn't, you should change it.
Hope that helps!
(Sorry my English)
rodriguezc
commented
2 years ago Had the same issue with NetworkManager and I finally solved it by following the official ingress dns doc BUT had to delete the resolv.conf file before restarting the network service:
sudo mv /etc/resolv.conf /etc/resolv.conf.old
systemctl restart NetworkManager.service
bitemarcz
commented
6 months ago ems to need a fir
Honestly, I'm really confused, but somehow this worked for me. My confusion right now is that my resolv.conf was sym linked to /etc/resolvconf/resolv.conf not to the systemd stub file or resolv.conf file in /run/systemd.
I was still able to replicate what you provided only using the different path for the symlinks. Wondering why the documentation is different or what's the delta between this and what the official docs have.
I still haven't rebooted my workstation so I don't know exactly if this will persist. Will find out down the road.
I guess my only question, what's the point of configuring the base file if you can bypass that and add the nameserver minikube ip to the resolv.conf file?
It seems to me something is broken where any updates are not picked up from the base file properly or everything isn't mapped correctly by default and it requires manual intervention to fix the mappings.
Any advice?
Steps to reproduce the issue: I feel a bit silly but I can't get the https://github.com/kubernetes/minikube/blob/master/deploy/addons/ingress-dns/example/example.yaml to give me a ping and I don't know where to begin to troubleshoot.
For starters: we don't work on k8s 1.19+ so I need
kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/8209421c3e5ef0af4ca92a30da25b5e75c9255bc/deploy/addons/ingress-dns/example/example.yamlas a fallback option. But still, it doesn't provide me with a correct ping output.I ran minikube delete --all --purge beforehand
minikube start \ --cpus=2 \ --kubernetes-version=v1.18.15 \ --driver=docker \ --memory=2g \ --disk-size=2gb
minikube addons enable ingress-dns
sleep 10 # Allow $(minikube ip) to become available
echo "search test" | sudo tee -a /etc/resolvconf/resolv.conf.d/base echo "nameserver $(minikube ip)" | sudo tee -a /etc/resolvconf/resolv.conf.d/base echo "timeout 5" | sudo tee -a /etc/resolvconf/resolv.conf.d/base
sudo resolvconf -u sudo systemctl disable --now resolvconf.service
kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/\ 8209421c3e5ef0af4ca92a30da25b5e75c9255bc/deploy/addons/ingress-dns/\ example/example.yaml # Use a k8s 1.18 beta networking version
ping hello-john.test