search

containers / aardvark-dns

Authoritative dns server for A/AAAA container records. Forwards other request to host's /etc/resolv.conf
Apache License 2.0
176 stars 31 forks source link

Rootful containers on debian sid host unable to resolve DNS #453

Closed krysclarke closed 4 months ago

krysclarke commented 4 months ago

INFO

Host: VPS running Debian Sid Container: NextCloud:stable (from docker.io, but also affects all other containers on same host) I did use docker initially, but migrated to podman. I am using podman-compose up -d to run the containers, and they start on boot-up of the host aardvark-dns is running on 10.8.1.1 and, as shown below, is working correctly to resolve DNS requests from the host.

I have one container where I have managed to get DNS working (I can't remember exactly how I managed to get /etc/resolv.conf 'locked in' inside that container, but it's a custom docker container I have build), but I would like to resolve why my containers can't.

None of the containers have a 'network' stanza in the docker-compose.yml I have tried podman network update --dns-add, amongst numerous other things which I'm too tired right now to recall (it's 03:25 where I am) I'm running out of things I can think of to 'google' for.

FROM HOST:

$ dig apps.nextcloud.com @10.8.1.1

 ; <<>> DiG 9.19.21-1+b1-Debian <<>> apps.nextcloud.com @10.8.1.1
 ;; global options: +cmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62660
 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

 ;; OPT PSEUDOSECTION:
 ; EDNS: version: 0, flags:; udp: 1410
 ;; QUESTION SECTION:
 ;apps.nextcloud.com.            IN      A

 ;; ANSWER SECTION:
 apps.nextcloud.com.     2033    IN      A       176.9.217.53

 ;; Query time: 24 msec
 ;; SERVER: 10.8.1.1#53(10.8.1.1) (UDP)
 ;; WHEN: Tue Apr 23 16:22:06 UTC 2024
 ;; MSG SIZE  rcvd: 63

$ curl https://apps.nextcloud.com
<Lots of output>

FROM CONTAINER:

$ podman exec -it cloud cat /etc/resolv.conf
search dns.podman
nameserver 10.8.1.1

$ podman exec -it cloud curl https://apps.nextcloud.com
curl: (6) Could not resolve host: apps.nextcloud.com

FROM HOST during curl ...:

Apr 23 17:16:20 <REDACTED> podman[323035]: 2024-04-23 17:16:20.930295082 +0000 UTC m=+0.707901763 container exec 7b69bae96e2676244174cb39dc62bd2945690cadd0f4c89c1a22a56f8ae48941 (image=docker.io/library/nextcloud:stable, name=cloud, PODMAN_SYSTEMD_UNIT=podman-compose@docker.service, com.docker.compose.project=docker, io.podman.compose.project=docker, com.docker.compose.container-number=1, com.docker.compose.project.config_files=docker-compose.yml, io.podman.compose.config-hash=f82ac3baaa4440712fe5b223698bc15986b2531b21b4e3917da914b72df39c1a, io.podman.compose.version=1.0.6, com.docker.compose.project.working_dir=/docker, com.docker.compose.service=nextcloud)
Apr 23 17:16:40 <REDACTED> podman[323062]: 2024-04-23 17:16:40.952167881 +0000 UTC m=+0.064408201 container exec_died 7b69bae96e2676244174cb39dc62bd2945690cadd0f4c89c1a22a56f8ae48941 (image=docker.io/library/nextcloud:stable, name=cloud, com.docker.compose.service=nextcloud, com.docker.compose.container-number=1, io.podman.compose.version=1.0.6, com.docker.compose.project.working_dir=/docker, io.podman.compose.config-hash=f82ac3baaa4440712fe5b223698bc15986b2531b21b4e3917da914b72df39c1a, io.podman.compose.project=docker, com.docker.compose.project.config_files=docker-compose.yml, PODMAN_SYSTEMD_UNIT=podman-compose@docker.service, com.docker.compose.project=docker)
Apr 23 17:16:41 <REDACTED> podman[323035]: 2024-04-23 17:16:41.248378177 +0000 UTC m=+21.025984868 container exec_died 7b69bae96e2676244174cb39dc62bd2945690cadd0f4c89c1a22a56f8ae48941 (image=docker.io/library/nextcloud:stable, name=cloud, PODMAN_SYSTEMD_UNIT=podman-compose@docker.service, com.docker.compose.project=docker, io.podman.compose.config-hash=f82ac3baaa4440712fe5b223698bc15986b2531b21b4e3917da914b72df39c1a, com.docker.compose.project.config_files=docker-compose.yml, io.podman.compose.project=docker, io.podman.compose.version=1.0.6, com.docker.compose.container-number=1, com.docker.compose.project.working_dir=/docker, com.docker.compose.service=nextcloud)

Any, and all, assistance would be greatly appreciated.

Luap99 commented 4 months ago

What podman, netavark, aardvark-dns version are you using? Note we only support the latest versions so I suggest you update them first.

If it works from the host but not the container most likely you have some firewall rules dropping the traffic. You can do some package captures to see where the packages are lost.

krysclarke commented 4 months ago

Software versions installed & running on the host (These are the latest available in Debian Sid):

# dpkg -l *podman* *netavark* *aardvark-dns*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-===============================================
ii  aardvark-dns   1.4.0-5      amd64        Container-focused DNS server
ii  netavark       1.4.0-4      amd64        Rust based network stack for containers
ii  podman         4.9.4+ds1-1  amd64        tool to manage containers and pods
ii  podman-compose 1.0.6-1      all          Run docker-compose.yml using podman
ii  podman-docker  4.9.4+ds1-1  amd64        tool to manage containers and pods (Docker CLI)

Packet capture from host during podman exec -it cloud curl https://apps.nextcloud.com

# tcpdump -tttt -i any host 10.8.1.1
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
2024-04-25 03:24:39.435293 veth3 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:39.435299 podman1 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:39.435327 veth3 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:39.435328 podman1 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:39.610188 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610190 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610191 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610192 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610193 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612017 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612019 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612020 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612021 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612021 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615011 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615013 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615014 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615015 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615016 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:44.440472 veth3 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:44.440477 podman1 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:44.440510 veth3 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:44.440510 podman1 In  IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:44.491683 veth3 In  ARP, Request who-has 10.8.1.1 tell 10.8.1.12, length 28
2024-04-25 03:24:44.491689 podman1 In  ARP, Request who-has 10.8.1.1 tell 10.8.1.12, length 28
2024-04-25 03:24:44.491701 podman1 Out ARP, Reply 10.8.1.1 is-at 86:c2:e4:d9:11:3b (oui Unknown), length 28
2024-04-25 03:24:44.491704 veth3 Out ARP, Reply 10.8.1.1 is-at 86:c2:e4:d9:11:3b (oui Unknown), length 28
2024-04-25 03:24:44.633605 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633608 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633609 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633611 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633612 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678733 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678735 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678736 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678737 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678737 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680533 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680535 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680536 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680537 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680537 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682453 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682454 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682455 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682456 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682456 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:54.447560 veth3 In  IP 10.8.1.12.48390 > 10.8.1.1.domain: 3105+ A? apps.nextcloud.com.dns.podman. (47)
2024-04-25 03:24:54.447565 podman1 In  IP 10.8.1.12.48390 > 10.8.1.1.domain: 3105+ A? apps.nextcloud.com.dns.podman. (47)
2024-04-25 03:24:54.447597 veth3 In  IP 10.8.1.12.48390 > 10.8.1.1.domain: 22565+ AAAA? apps.nextcloud.com.dns.podman. (47)
2024-04-25 03:24:54.447598 podman1 In  IP 10.8.1.12.48390 > 10.8.1.1.domain: 22565+ AAAA? apps.nextcloud.com.dns.podman. (47)

So this confirms that the container is sending the packets. Not shown, but when I set tcpdump to listen to just the podman1 interface, it also captured the DNS packets.

NFT active rules

# nft list table ip filter
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain NETAVARK_FORWARD {
                ip daddr 10.8.1.0/28 ct state related,established counter packets 147983 bytes 23043160 accept
                ip saddr 10.8.1.0/28 counter packets 85272 bytes 22639658 accept
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                 counter packets 10236795 bytes 7180413560 jump NETAVARK_FORWARD
        }
}
# nft list table ip nat
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                counter packets 327921 bytes 20979404 jump NETAVARK-HOSTPORT-MASQ
                ip saddr 10.8.1.0/28 counter packets 11044 bytes 775651 jump NETAVARK-0353337B2E75F
        }

        chain NETAVARK-HOSTPORT-SETMARK {
                counter packets 120500 bytes 7230000 meta mark set mark or 0x2000
        }

        chain NETAVARK-HOSTPORT-MASQ {
                 meta mark & 0x00002000 == 0x00002000 counter packets 120500 bytes 7230000 masquerade
        }

        chain NETAVARK-HOSTPORT-DNAT {
                tcp dport 3306  counter packets 1205 bytes 71212 jump NETAVARK-DN-0353337B2E75F
                tcp dport 8080  counter packets 107 bytes 4828 jump NETAVARK-DN-0353337B2E75F
                tcp dport 25  counter packets 276 bytes 14520 jump NETAVARK-DN-0353337B2E75F
                tcp dport 465  counter packets 5431 bytes 325540 jump NETAVARK-DN-0353337B2E75F
                tcp dport 993  counter packets 355 bytes 20316 jump NETAVARK-DN-0353337B2E75F
                tcp dport 88  counter packets 1101 bytes 65664 jump NETAVARK-DN-0353337B2E75F
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 325648 bytes 18037817 jump NETAVARK-HOSTPORT-DNAT
        }

        chain OUTPUT {
                type nat hook output priority dstnat; policy accept;
                fib daddr type local counter packets 30239 bytes 1814433 jump NETAVARK-HOSTPORT-DNAT
        }

        chain NETAVARK-0353337B2E75F {
                ip daddr 10.8.1.0/28 counter packets 0 bytes 0 accept
                ip daddr != 224.0.0.0/4 counter packets 11039 bytes 775262 masquerade
        }

        chain NETAVARK-DN-0353337B2E75F {
                ip saddr 10.8.1.0/28 tcp dport 3306 counter packets 1111 bytes 66660 jump NETAVARK-HOSTPORT-SETMARK
                ip saddr 127.0.0.1 tcp dport 3306 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                tcp dport 3306 counter packets 1205 bytes 71212 dnat to 10.8.1.9:3306
                ip saddr 10.8.1.0/28 ip daddr 127.0.0.1 tcp dport 8080 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                ip saddr 127.0.0.1 ip daddr 127.0.0.1 tcp dport 8080 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                ip daddr 127.0.0.1 tcp dport 8080 counter packets 0 bytes 0 dnat to 10.8.1.10:80
                ip saddr 10.8.1.0/28 tcp dport 25 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                ip saddr 127.0.0.1 tcp dport 25 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                tcp dport 25 counter packets 276 bytes 14520 dnat to 10.8.1.11:25
                ip saddr 10.8.1.0/28 tcp dport 465 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                ip saddr 127.0.0.1 tcp dport 465 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                tcp dport 465 counter packets 5431 bytes 325540 dnat to 10.8.1.11:465
                ip saddr 10.8.1.0/28 tcp dport 993 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                ip saddr 127.0.0.1 tcp dport 993 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                tcp dport 993 counter packets 355 bytes 20316 dnat to 10.8.1.11:993
                ip saddr 10.8.1.0/28 tcp dport 88 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
                ip saddr 127.0.0.1 tcp dport 88 counter packets 898 bytes 53880 jump NETAVARK-HOSTPORT-SETMARK
                tcp dport 88 counter packets 1101 bytes 65664 dnat to 10.8.1.12:80
        }
}
# nft list table inet filter
table inet filter {
        chain INPUT {
                type filter hook input priority filter; policy accept;
                ip daddr <REDACTED> tcp dport 8080 counter packets 886 bytes 40392 drop
                iif "lo" accept
                ct state established,related accept
                tcp dport { 80, 443 } accept
                ip daddr <REDACTED> udp dport 1194 accept
                ip daddr 10.8.0.1 accept
                icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, 148, 149 } accept
                ip6 saddr fe80::/10 icmpv6 type { mld-listener-query, mld-listener-report, mld-listener-done, mld2-listener-report, 151, 152, 153 } accept
                drop
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                ip saddr != 10.8.1.0/28 tcp dport 8080 counter packets 0 bytes 0 drop
                ip saddr != 10.8.1.0/28 tcp dport 3306 drop
        }

        chain OUTPUT {
                type filter hook output priority filter; policy accept;
        }
}
Luap99 commented 4 months ago

You need to add a rule to allow port 53 input, you drop all unknown input so aardvark-dns never gets the packages.

netavark/aardvark-dns 1.4 are very old Newer versions of netavark (v1.8) create the dns accept rule automatically https://github.com/containers/netavark/pull/780/commits/3806d9a97dc912ae30827621abe78676802710ff

krysclarke commented 4 months ago

They may be (very) old but without me manually installing them on the server I have to get more recent versions as these are the latest available for Debian Sid, at the moment - see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1052432

However, adding a rule to nftables to allow connections to 10.8.1.1:53 did the trick.