Closed krysclarke closed 4 months ago
What podman, netavark, aardvark-dns version are you using? Note we only support the latest versions so I suggest you update them first.
If it works from the host but not the container most likely you have some firewall rules dropping the traffic. You can do some package captures to see where the packages are lost.
# dpkg -l *podman* *netavark* *aardvark-dns*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-===============================================
ii aardvark-dns 1.4.0-5 amd64 Container-focused DNS server
ii netavark 1.4.0-4 amd64 Rust based network stack for containers
ii podman 4.9.4+ds1-1 amd64 tool to manage containers and pods
ii podman-compose 1.0.6-1 all Run docker-compose.yml using podman
ii podman-docker 4.9.4+ds1-1 amd64 tool to manage containers and pods (Docker CLI)
podman exec -it cloud curl https://apps.nextcloud.com
# tcpdump -tttt -i any host 10.8.1.1
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
2024-04-25 03:24:39.435293 veth3 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:39.435299 podman1 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:39.435327 veth3 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:39.435328 podman1 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:39.610188 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610190 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610191 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610192 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:39.610193 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612017 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612019 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612020 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612021 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:40.612021 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615011 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615013 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615014 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615015 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:42.615016 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 1.1.8.10.in-addr.arpa. (39)
2024-04-25 03:24:44.440472 veth3 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:44.440477 podman1 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 49613+ A? apps.nextcloud.com. (36)
2024-04-25 03:24:44.440510 veth3 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:44.440510 podman1 In IP 10.8.1.12.52629 > 10.8.1.1.domain: 22223+ AAAA? apps.nextcloud.com. (36)
2024-04-25 03:24:44.491683 veth3 In ARP, Request who-has 10.8.1.1 tell 10.8.1.12, length 28
2024-04-25 03:24:44.491689 podman1 In ARP, Request who-has 10.8.1.1 tell 10.8.1.12, length 28
2024-04-25 03:24:44.491701 podman1 Out ARP, Reply 10.8.1.1 is-at 86:c2:e4:d9:11:3b (oui Unknown), length 28
2024-04-25 03:24:44.491704 veth3 Out ARP, Reply 10.8.1.1 is-at 86:c2:e4:d9:11:3b (oui Unknown), length 28
2024-04-25 03:24:44.633605 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633608 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633609 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633611 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:44.633612 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 12.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678733 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678735 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678736 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678737 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:49.678737 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680533 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680535 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680536 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680537 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:50.680537 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682453 podman1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682454 veth3 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682455 veth2 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682456 veth1 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:52.682456 veth0 Out IP 10.8.1.1.mdns > mdns.mcast.net.mdns: 0 PTR (QM)? 11.1.8.10.in-addr.arpa. (40)
2024-04-25 03:24:54.447560 veth3 In IP 10.8.1.12.48390 > 10.8.1.1.domain: 3105+ A? apps.nextcloud.com.dns.podman. (47)
2024-04-25 03:24:54.447565 podman1 In IP 10.8.1.12.48390 > 10.8.1.1.domain: 3105+ A? apps.nextcloud.com.dns.podman. (47)
2024-04-25 03:24:54.447597 veth3 In IP 10.8.1.12.48390 > 10.8.1.1.domain: 22565+ AAAA? apps.nextcloud.com.dns.podman. (47)
2024-04-25 03:24:54.447598 podman1 In IP 10.8.1.12.48390 > 10.8.1.1.domain: 22565+ AAAA? apps.nextcloud.com.dns.podman. (47)
So this confirms that the container is sending the packets.
Not shown, but when I set tcpdump
to listen to just the podman1
interface, it also captured the DNS packets.
# nft list table ip filter
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
chain NETAVARK_FORWARD {
ip daddr 10.8.1.0/28 ct state related,established counter packets 147983 bytes 23043160 accept
ip saddr 10.8.1.0/28 counter packets 85272 bytes 22639658 accept
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
counter packets 10236795 bytes 7180413560 jump NETAVARK_FORWARD
}
}
# nft list table ip nat
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
chain POSTROUTING {
type nat hook postrouting priority srcnat; policy accept;
counter packets 327921 bytes 20979404 jump NETAVARK-HOSTPORT-MASQ
ip saddr 10.8.1.0/28 counter packets 11044 bytes 775651 jump NETAVARK-0353337B2E75F
}
chain NETAVARK-HOSTPORT-SETMARK {
counter packets 120500 bytes 7230000 meta mark set mark or 0x2000
}
chain NETAVARK-HOSTPORT-MASQ {
meta mark & 0x00002000 == 0x00002000 counter packets 120500 bytes 7230000 masquerade
}
chain NETAVARK-HOSTPORT-DNAT {
tcp dport 3306 counter packets 1205 bytes 71212 jump NETAVARK-DN-0353337B2E75F
tcp dport 8080 counter packets 107 bytes 4828 jump NETAVARK-DN-0353337B2E75F
tcp dport 25 counter packets 276 bytes 14520 jump NETAVARK-DN-0353337B2E75F
tcp dport 465 counter packets 5431 bytes 325540 jump NETAVARK-DN-0353337B2E75F
tcp dport 993 counter packets 355 bytes 20316 jump NETAVARK-DN-0353337B2E75F
tcp dport 88 counter packets 1101 bytes 65664 jump NETAVARK-DN-0353337B2E75F
}
chain PREROUTING {
type nat hook prerouting priority dstnat; policy accept;
fib daddr type local counter packets 325648 bytes 18037817 jump NETAVARK-HOSTPORT-DNAT
}
chain OUTPUT {
type nat hook output priority dstnat; policy accept;
fib daddr type local counter packets 30239 bytes 1814433 jump NETAVARK-HOSTPORT-DNAT
}
chain NETAVARK-0353337B2E75F {
ip daddr 10.8.1.0/28 counter packets 0 bytes 0 accept
ip daddr != 224.0.0.0/4 counter packets 11039 bytes 775262 masquerade
}
chain NETAVARK-DN-0353337B2E75F {
ip saddr 10.8.1.0/28 tcp dport 3306 counter packets 1111 bytes 66660 jump NETAVARK-HOSTPORT-SETMARK
ip saddr 127.0.0.1 tcp dport 3306 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
tcp dport 3306 counter packets 1205 bytes 71212 dnat to 10.8.1.9:3306
ip saddr 10.8.1.0/28 ip daddr 127.0.0.1 tcp dport 8080 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
ip saddr 127.0.0.1 ip daddr 127.0.0.1 tcp dport 8080 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
ip daddr 127.0.0.1 tcp dport 8080 counter packets 0 bytes 0 dnat to 10.8.1.10:80
ip saddr 10.8.1.0/28 tcp dport 25 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
ip saddr 127.0.0.1 tcp dport 25 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
tcp dport 25 counter packets 276 bytes 14520 dnat to 10.8.1.11:25
ip saddr 10.8.1.0/28 tcp dport 465 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
ip saddr 127.0.0.1 tcp dport 465 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
tcp dport 465 counter packets 5431 bytes 325540 dnat to 10.8.1.11:465
ip saddr 10.8.1.0/28 tcp dport 993 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
ip saddr 127.0.0.1 tcp dport 993 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
tcp dport 993 counter packets 355 bytes 20316 dnat to 10.8.1.11:993
ip saddr 10.8.1.0/28 tcp dport 88 counter packets 0 bytes 0 jump NETAVARK-HOSTPORT-SETMARK
ip saddr 127.0.0.1 tcp dport 88 counter packets 898 bytes 53880 jump NETAVARK-HOSTPORT-SETMARK
tcp dport 88 counter packets 1101 bytes 65664 dnat to 10.8.1.12:80
}
}
# nft list table inet filter
table inet filter {
chain INPUT {
type filter hook input priority filter; policy accept;
ip daddr <REDACTED> tcp dport 8080 counter packets 886 bytes 40392 drop
iif "lo" accept
ct state established,related accept
tcp dport { 80, 443 } accept
ip daddr <REDACTED> udp dport 1194 accept
ip daddr 10.8.0.1 accept
icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, 148, 149 } accept
ip6 saddr fe80::/10 icmpv6 type { mld-listener-query, mld-listener-report, mld-listener-done, mld2-listener-report, 151, 152, 153 } accept
drop
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
ip saddr != 10.8.1.0/28 tcp dport 8080 counter packets 0 bytes 0 drop
ip saddr != 10.8.1.0/28 tcp dport 3306 drop
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
}
}
You need to add a rule to allow port 53 input, you drop all unknown input so aardvark-dns never gets the packages.
netavark/aardvark-dns 1.4 are very old Newer versions of netavark (v1.8) create the dns accept rule automatically https://github.com/containers/netavark/pull/780/commits/3806d9a97dc912ae30827621abe78676802710ff
They may be (very) old but without me manually installing them on the server I have to get more recent versions as these are the latest available for Debian Sid, at the moment - see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1052432
However, adding a rule to nftables to allow connections to 10.8.1.1:53 did the trick.
INFO
Host: VPS running Debian Sid Container: NextCloud:stable (from docker.io, but also affects all other containers on same host) I did use docker initially, but migrated to podman. I am using
podman-compose up -d
to run the containers, and they start on boot-up of the host aardvark-dns is running on 10.8.1.1 and, as shown below, is working correctly to resolve DNS requests from the host.I have one container where I have managed to get DNS working (I can't remember exactly how I managed to get /etc/resolv.conf 'locked in' inside that container, but it's a custom docker container I have build), but I would like to resolve why my containers can't.
None of the containers have a 'network' stanza in the docker-compose.yml I have tried
podman network update --dns-add
, amongst numerous other things which I'm too tired right now to recall (it's 03:25 where I am) I'm running out of things I can think of to 'google' for.FROM HOST:
FROM CONTAINER:
FROM HOST during
curl ...
:Any, and all, assistance would be greatly appreciated.