Fix DNS resolution

[wanno-drijfhout]

Affects #3 / #28

• [x] Fix or disable IPv6 (the reason for the double nslookup response)
• [x] Fix configuration of /etc/resolv.conf in docker containers
• [x] Design DNS configuration of hypervisor/app-server/swarm

[wanno-drijfhout]

The problem is visible on Docker containers:

bash-5.1# nslookup pihole.keizerlijk.eu
Server:         127.0.0.11
Address:        127.0.0.11#53

Name:   pihole.keizerlijk.eu
Address: 10.69.10.255
** server can't find pihole.keizerlijk.eu: NXDOMAIN

The first response (Name + Address) is the A-record. The second response (NXDOMAIN) is the AAAA-record query.

This is a problem, because this causes DNS lookups to generally fail, even though there's an IPv4 route available. Curl would return "bad address", for example.

[wanno-drijfhout]

Another difficulty is the configuration of the host system w.r.t. networking and DNS. By default, the Docker daemon relies on /etc/resolve.conf on the host for configuring the DNS servers in the children.

DNS dependencies between host system and docker containers is incredibly difficult. DNS responses from Pi-Hole and Fritz!Box can be combined, leading to weird hybrids of public and private IPs/DNS records. This is also caused by IPv4 and IPv6 responses being treated separately. There is also a bootstrapping issue: the docker host system can't download (pihole) images without DNS.

Decision: let the hypervisor and Docker host OS just use DNS at 10.69.1.1 (Fritz!Box). They don't generally connect to keizerlijk.eu sub domains, which couldn't be served by Fritz!Box.

The hypervisor and Docker host should connect primarily to 10.69.10.255 and secondary to 10.69.1.1. The Ansible script does connect to keizerlijk.eu domains, so it needs to work there as well.

Configure the Docker daemon to use the DNS on its internal docker_gwbridge network (172.18.0.1). This resolves to the Pi-Hole, which is accessible for the rest of the network at 10.69.10.255.

[wanno-drijfhout]

Example of problematic dig:

; <<>> DiG 9.16.1-Ubuntu <<>> pihole.keizerlijk.eu
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3611
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;pihole.keizerlijk.eu.          IN      A

;; ANSWER SECTION:
pihole.keizerlijk.eu.   2       IN      A       10.69.10.255

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Aug 08 17:11:32 UTC 2021
;; MSG SIZE  rcvd: 65

Good dig:

; <<>> DiG 9.16.15-Debian <<>> pihole.keizerlijk.eu 10.69.10.255
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62757
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;pihole.keizerlijk.eu.          IN      A

;; ANSWER SECTION:
pihole.keizerlijk.eu.   67485   IN      CNAME   keizerlijk.eu.
keizerlijk.eu.          76991   IN      A       45.88.9.121

;; Query time: 0 msec
;; SERVER: 10.69.1.1#53(10.69.1.1)
;; WHEN: Sun Aug 08 22:43:38 CEST 2021
;; MSG SIZE  rcvd: 79

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 23300
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;10.69.10.255.                  IN      A

;; AUTHORITY SECTION:
.                       3600    IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2021080801 1800 900 604800 86400

;; Query time: 19 msec
;; SERVER: 10.69.1.1#53(10.69.1.1)
;; WHEN: Sun Aug 08 22:43:38 CEST 2021
;; MSG SIZE  rcvd: 116

[wanno-drijfhout]

Configure the Docker host system (Ubuntu) has some quirks too. Dropping some data for posterity:

/etc/systemd/resolved.conf:

[Resolve]
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
#DNSStubListener=yes
#ReadEtcHosts=yes
DNSStubListener=no
#DNS=127.0.0.1
DNS=10.69.10.255
Domains=keizerlijk.eu fritz.box
FallbackDNS=10.69.1.1

https://github.com/pi-hole/docker-pi-hole#installing-on-ubuntu

https://docs.docker.com/network/bridge/#enable-forwarding-from-docker-containers-to-the-outside-world