pods cant resolve dns from external dns server

[agonzalezm]

Ubuntu Server 22.04 microk8s enable dns

pods can ping ips of the network but cant resolve dns names in the network.

from node host i can resolve external dns correctly.

-- inside pod---

(ping to external dns ip)
# ping 10.58.20.1
PING 10.58.20.1 (10.58.20.1) 56(84) bytes of data.
64 bytes from 10.58.20.1: icmp_seq=1 ttl=127 time=0.487 ms

(ping to external dns name)
# ping dc01
ping: dc01: Name or service not known

# ping dc01.lab.local
ping: dc01: Name or service not known

# cat /etc/resolv.conf
search app.svc.cluster.local svc.cluster.local cluster.local lab.local
nameserver 10.152.183.10
options ndots:5
root@app-api-6d7f865754-9zcdg:/opt/app/api#

[ktsakalozos]

Hi @agonzalezm, have you tried to set the forward dns to the dns of your lab? See: https://microk8s.io/docs/addon-dns

[agonzalezm]

yes i tried adding my dns to enable dns (microk8s enable dns:10.58.20.1 ) but the same result, this was working months ago in ubutu 20.04, now i installed new machine with 22.04 and dns is not working. any other idea why is not working?

when i leave default dns ( micok8s enable dns) I get ping dc01 ping: dc01: Name or service not know

when i configure my dns (microk8s enable dns:10.58.20.1 ) ping dc01 ping: dc01: Temporary failure in name resolution

Also when using my dns apt-get update doesnt work apt-get update Ign:1 http://security.ubuntu.com/ubuntu focal-security InRelease Ign:2 http://archive.ubuntu.com/ubuntu focal InRelease Err:3 http://security.ubuntu.com/ubuntu focal-security Release 404 Not Found [IP: 10.58.20.41 80]

so looks like specifying my dns breaks everything, and i can ping by ip my dns, and from host everything resolves fine, also did the iptables forward and disabled ufw

what is wrong?

[agonzalezm]

the strange this is if i deploy a dnsutils pod it can resolve dns correctly but my pods cant. any idea why my pods cant connect to dns or do apt-get update if i use my network dns?

[agonzalezm]

I tested this in Ubuntu Server 20.04.4 and no problem i can run apt-get update from my container and resolve correctly with my pods, looks like problem in 22.04 not mature enough

[manoj-bandara]

microk8s disable dns export dnss=cat /run/systemd/resolve/resolv.conf | grep nameserver | cut -d ' ' -f2 microk8s enable dns:$dnss

[olamy]

microk8s disable dns export dnss=cat /run/systemd/resolve/resolv.conf | grep nameserver | cut -d ' ' -f2 microk8s enable dns:$dnss

I did that for sure. But then:

microk8s kubectl run -i --tty --rm debug --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # nslookup github.com
Server:     10.152.183.10
Address:    10.152.183.10:53

Non-authoritative answer:

*** Can't find github.com: No answer

But more interesting my dns server is 10.0.0.1.

/ # nslookup github.com 10.0.0.1
Server:     10.0.0.1
Address:    10.0.0.1:53

Non-authoritative answer:
Name:   github.com
Address: 140.82.114.3

*** Can't find github.com: No answer

[rodrigorodrigues]

microk8s disable dns export dnss=cat /run/systemd/resolve/resolv.conf | grep nameserver | cut -d ' ' -f2 microk8s enable dns:$dnss

This solution works perfectly for me I wasn't able to connect to external datasource from local k8s cluster with raspberry, after run those commands now it's working.

Thanks

[neoaggelos]

Note that starting from 1.26, MicroK8s will now automatically try its best to pick up the upstream nameservers from the underlying host, see https://github.com/canonical/microk8s-core-addons/pull/111.

It would be great if you can test whether this helps you with your issue on a new install:

sudo snap install microk8s --classic --channel 1.26
sudo microk8s enable gpu

[Azbesciak]

I suppose not gpu but dns. I just updated our ubuntu 18 to 22, then it turned out that dns does not work at all. I got info that /etc/resolv.conf was not found.. of course, there was some, the default one I suppose, but it was not picked up anyway.

[AugustBrenner]

I'm having the same issue on Ubuntu 22

[teoxrr]

Same problem here, Jammy

[neoaggelos]

Hi @AugustBrenner @teoxrr @Azbesciak , would you mind sharing an inspection tarball (microk8s inspect) so that we can look into the issue further? Thanks!

[timbo-r]

I've the same issue ...

For the moment I can get around this by defining my dns config in the deployment:

    spec:
      dnsPolicy: "None"
      dnsConfig:
        nameservers:
          - 192.168.xx.x
      containers:

I configured the dns addon like mentioned above: microk8s enable dns:192.168.xx.x

Additionally the metallb addon is enabled.

cat /etc/os-release:

PRETTY_NAME="Ubuntu 22.04.3 LTS"
[...]

snap info microk8s:

[...]
installed:               v1.27.4             (5643) 177MB classic

[timbo-r]

After a weekend of reading, learning and debugging I was able to fix my issue and it was not related to microk8s:

I use a self hosted dns called adguard and configured a dns rewrite rule with a wildcard: e.g. *.home.net. This domain is forwarded to my metallb-ip ...

For testing in a pod i used the dnsutils pod described in the kubernetes docs.

When executing a ping in this pod ... kubectl exec -i -t dnsutils -- ping google.de

PING google.de.home.net (192.168.10.30) 56(84) bytes of data.

... google.de is suffixed with home.net

As I learned this is because the ping command will try every search domain listed in /etc/resolv.conf.

kubectl exec -i -t dnsutils -- cat /etc/resolv.conf

search default.svc.cluster.local svc.cluster.local cluster.local home.net
nameserver 10.152.183.10
options ndots:5

Because I used a wildcard domain in my adguard the home.net search domain is the first one which responded ... This explanation was the key for me.

So the solution for me is to specify every domain, in my adguard, as a whole, like traefik.home.net or heimdall.home.net and not use the wildcard option in my dns rewrites.

Best regards

[mvarchdev]

For me it is not working. Same issue

[mvarchdev]

This is what I am getting:

root@dnsutils:/# nslookup google.com
Server:         10.152.183.10
Address:        10.152.183.10#53

Non-authoritative answer:
Name:   google.com.sunray.sk
Address: 46.150.201.69

Any suggestions? Thanks in advance.

[mnbf9rca]

I performed a clean install of Ubuntu 22.04 minimal. I did not update any packages. I installed Microk8s with sudo snap install microk8s --classic. There are no other changes to the base installation.

i installed coredns with a specific DNS server which i know is good:

rob@k7:~$ microk8s enable dns:192.168.17.4
Infer repository core for addon dns
Enabling DNS
Will use  192.168.17.4 as upstream nameservers
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
CoreDNS service deployed with IP address 10.152.183.10
Restarting kubelet
DNS is enabled

I then edited the config with kubectl edit configmap coredns -n kube-system to increase the logging level for coredns, and deleted the pods with kubectl delete pod -n kube-system -l k8s-app=kube-dns. when i inspect the logs, i see it's only trying to resolve local addresses, which means it never resolves:

rob@k7:~$ kubectl logs -n kube-system -l k8s-app=kube-dns
[INFO] 10.1.170.77:48121 - 26887 "AAAA IN uksouth.obo.arc.azure.com.mydomain.net. udp 55 false 512" NOERROR qr,rd,ra 153 0.018593215s
[INFO] 10.1.170.77:48121 - 26735 "A IN uksouth.obo.arc.azure.com.mydomain.net. udp 55 false 512" NOERROR qr,rd,ra 153 0.033970645s
[INFO] 10.1.170.77:60249 - 40605 "AAAA IN mcr.microsoft.com.azure-arc-release.svc.cluster.local. udp 71 false 512" NXDOMAIN qr,aa,rd 164 0.000144352s
[INFO] 10.1.170.77:60249 - 40375 "A IN mcr.microsoft.com.azure-arc-release.svc.cluster.local. udp 71 false 512" NXDOMAIN qr,aa,rd 164 0.00014482s
[INFO] 10.1.170.77:53076 - 62574 "AAAA IN mcr.microsoft.com.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000080304s
[INFO] 10.1.170.77:53076 - 62296 "A IN mcr.microsoft.com.svc.cluster.local. udp 53 false 512" NXDOMAIN qr,aa,rd 146 0.000104837s
[INFO] 10.1.170.77:51504 - 118 "AAAA IN mcr.microsoft.com.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000066976s
[INFO] 10.1.170.77:51504 - 65389 "A IN mcr.microsoft.com.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000088796s
[INFO] 10.1.170.77:59900 - 11187 "AAAA IN mcr.microsoft.com.mydomain.net. udp 47 false 512" NOERROR qr,rd,ra 145 0.019024323s
[INFO] 10.1.170.77:59900 - 10958 "A IN mcr.microsoft.com.mydomain.net. udp 47 false 512" NOERROR qr,rd,ra 145 0.019595862s

interestingly, if i put a . on the end of the DNS name it works e.g.

rob@k7:~$ kubectl exec -i -t dnsutils -- nslookup www.google.com
Server:         10.152.183.10
Address:        10.152.183.10#53

Non-authoritative answer:
*** Can't find www.google.com: No answer

rob@k7:~$ kubectl exec -i -t dnsutils -- nslookup www.google.com.
Server:         10.152.183.10
Address:        10.152.183.10#53

Non-authoritative answer:
Name:   www.google.com
Address: 172.217.169.4

i found this: Debugging DNS Resolution > known issues but on my machine it's already set correctly:

rob@k9:~$ cat /var/snap/microk8s/current/args/kubelet
--resolv-conf=/run/systemd/resolve/resolv.conf
--kubeconfig=${SNAP_DATA}/credentials/kubelet.config
--cert-dir=${SNAP_DATA}/certs
--client-ca-file=${SNAP_DATA}/certs/ca.crt
--anonymous-auth=false
--root-dir=${SNAP_COMMON}/var/lib/kubelet
--fail-swap-on=false
--eviction-hard="memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi"
--container-runtime-endpoint=${SNAP_COMMON}/run/containerd.sock
--containerd=${SNAP_COMMON}/run/containerd.sock
--node-labels="microk8s.io/cluster=true,node.kubernetes.io/microk8s-controlplane=microk8s-controlplane"
--authentication-token-webhook=true
--read-only-port=0
--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
--serialize-image-pulls=false
--cluster-domain=cluster.local
--cluster-dns=10.152.183.10

lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:    22.04
Codename:   jammy

snap --version:

snap    2.60.4
snapd   2.60.4
series  16
ubuntu  22.04
kernel  5.15.0-88-generic

rob@k7:~$ microk8s version
MicroK8s v1.28.3 revision 6089
rob@k7:~$ microk8s status
microk8s is running
high-availability: no
  datastore master nodes: 127.0.0.1:19001
  datastore standby nodes: none
addons:
  enabled:
    dns                  # (core) CoreDNS
    ha-cluster           # (core) Configure high availability on the current node
    helm                 # (core) Helm - the package manager for Kubernetes
    helm3                # (core) Helm 3 - the package manager for Kubernetes
  disabled:
    cert-manager         # (core) Cloud native certificate management
    cis-hardening        # (core) Apply CIS K8s hardening
    community            # (core) The community addons repository
    dashboard            # (core) The Kubernetes dashboard
    gpu                  # (core) Automatic enablement of Nvidia CUDA
    host-access          # (core) Allow Pods connecting to Host services smoothly
    hostpath-storage     # (core) Storage class; allocates storage from host directory
    ingress              # (core) Ingress controller for external access
    kube-ovn             # (core) An advanced network fabric for Kubernetes
    mayastor             # (core) OpenEBS MayaStor
    metallb              # (core) Loadbalancer for your Kubernetes cluster
    metrics-server       # (core) K8s Metrics Server for API access to service metrics
    minio                # (core) MinIO object storage
    observability        # (core) A lightweight observability stack for logs, traces and metrics
    prometheus           # (core) Prometheus operator for monitoring and logging
    rbac                 # (core) Role-Based Access Control for authorisation
    registry             # (core) Private image registry exposed on localhost:32000
    rook-ceph            # (core) Distributed Ceph storage using Rook
    storage              # (core) Alias to hostpath-storage add-on, deprecated