Clients using WAN DNS resolver not NextDNS

[mojo333]

After implementing the Conditional Forwarding fix, all my VLANs and my main (untagged) LAN use my WAN configured resolver (Cloudflared) instead of the NextDNS client on 10.0.5.3

This can be seen by looking on the NextDNS Setup page which shows the following: "This device is not using NextDNS. This device is currently using ”Cloudflare” as DNS resolver."

Also, I enabled logging within NextDNS and no DNS queries are being logged. When I force a query to NextDNS directly on my UDM (using "nslookup bbc.com 10.0.5.3") then I can see a NextDNS log entry so I know logging is enabled.

All clients are correctly pointing to 10.0.5.3 (unchanged from before the fix).

My nextdns.conf before I unwound it was as follows:

listen :53

report-client-info yes
cache-size=10MB
max-ttl=5s
discovery-dns 10.0.5.1

forwarder 240.168.192.in-addr.arpa=192.168.240.1
forwarder 220.168.192.in-addr.arpa=192.168.220.1
forwarder 210.168.192.in-addr.arpa=192.168.210.1
forwarder 200.168.192.in-addr.arpa=192.168.200.1
bogus-priv false

# guest network
config 192.168.220.0/24=xxxxxx
# everyone else
config yyyyyy

UDM Information

Variant: Dream Machine PRO Firmware Version: 1.8.0-rc7 Controller Version: 6.0.4

I unwound the changes to 10-dns.sh and nextdns.conf and unfortunately this has not fixed the issue.

I have also restarted the NextDns container as well as a full reboot of my UDM Pro - neither fixed the issue.

I think there's also an issue with the fix in 10-dns.sh in that the file /run/dnsmasq.conf.d/custom.conf doesn't exist so this part of the script fails. I worked around this by adding "touch /run/dnsmasq.conf.d/custom.conf"

It seems that ether DNSMASQ has been misconfigured (10-dns.sh script does mess with dnsmasq) or my NextDNS container is forwarding DNS requests to my WAN DNS resolver and I cannot figure out why when I've unwound the changes the issue still persists...

[mojo333]

I've tried stripping the config of 10-dns.sh and nextdns.conf down to the bare minimum and rebooting my UDM but still no DNS traffic is sent to 10.0.5.3.

10-dns.sh

#!/bin/sh

mkdir -p /opt/cni
rm -f /opt/cni/bin
ln -s /mnt/data/podman/cni /opt/cni/bin
ln -s /mnt/data/podman/cni/20-dns.conflist  /etc/cni/net.d/20-dns.conflist

# Assumes your Podman network made in the controller is on VLAN 5
# Adjust the IP to match the address in your cni configuration
ip link set br5 promisc on

ip link add br5.mac link br5 type macvlan mode bridge
ip addr add 10.0.5.1/24 dev br5.mac noprefixroute
ip link set br5.mac promisc on
ip link set br5.mac up

ip route add 10.0.5.3/32 dev br5.mac

# Remove the # on the line below when Docker container is deployed.
podman start nextdns

nextdns.conf

listen :53
config yyyyyy

The only way to force DNS traffic to NextDNS is to ssh onto my UDM and run the command "nslookup bbc.com 10.0.5.3" - when i do this i see entries in the NextDNS log (via their dashboard).

[mojo333]

I dug around a bit more... it would seem dnsmasq on my UDM PRO is listening on port 53 of my macvlan gateway (10.0.5.1)... should this be the case? The UDM is spawing one dnsmasq process per network on my UDM, even though DNS Filter is disabled (set to None) for all networks.

Partial output of ss -nlput | grep 53

udp    UNCONN     0      0      127.0.0.1:53                 0.0.0.0:*                   users:(("dnsmasq",pid=19159,fd=18))
udp    UNCONN     0      0      192.168.200.1:53                 0.0.0.0:*                   users:(("dnsmasq",pid=19159,fd=16))
udp    UNCONN     0      0      192.168.220.1:53                 0.0.0.0:*                   users:(("dnsmasq",pid=19159,fd=14))
udp    UNCONN     0      0      10.0.5.1:53                 0.0.0.0:*                   users:(("dnsmasq",pid=19159,fd=12))
udp    UNCONN     0      0      192.168.210.1:53                 0.0.0.0:*                   users:(("dnsmasq",pid=19159,fd=10))
udp    UNCONN     0      0      192.168.240.1:53                 0.0.0.0:*                   users:(("dnsmasq",pid=19159,fd=8))
udp    UNCONN     0      0      203.0.113.1:53                 0.0.0.0:*                   users:(("dnsmasq",pid=19159,fd=6))

Output of ps | grep dnsmasq

 3400 nobody   /usr/sbin/dnsmasq -r /run/dnsfilter/dns-10.0.5.1-resolv.conf -C /run/dnsfilter/dns-10.0.5.1-conf.conf --pid-file=/run/dnsfilter/dns-10.0.5.1.pid
 3408 nobody   /usr/sbin/dnsmasq -r /run/dnsfilter/dns-192.168.240.1-resolv.conf -C /run/dnsfilter/dns-192.168.240.1-conf.conf --pid-file=/run/dnsfilter/dns-192.168.240.1.pid
 3415 nobody   /usr/sbin/dnsmasq -r /run/dnsfilter/dns-192.168.200.1-resolv.conf -C /run/dnsfilter/dns-192.168.200.1-conf.conf --pid-file=/run/dnsfilter/dns-192.168.200.1.pid
 3434 nobody   /usr/sbin/dnsmasq -r /run/dnsfilter/dns-192.168.210.1-resolv.conf -C /run/dnsfilter/dns-192.168.210.1-conf.conf --pid-file=/run/dnsfilter/dns-192.168.210.1.pid
 3448 nobody   /usr/sbin/dnsmasq -r /run/dnsfilter/dns-192.168.220.1-resolv.conf -C /run/dnsfilter/dns-192.168.220.1-conf.conf --pid-file=/run/dnsfilter/dns-192.168.220.1.pid
 8254 root     grep dnsmasq
19159 nobody   /usr/sbin/dnsmasq --conf-dir=/run/dnsmasq.conf.d/
19160 root     /usr/sbin/dnsmasq --conf-dir=/run/dnsmasq.conf.d/

[nschemel]

Hi!

That sound a lot like my problem Link to my bug.

I've checked on my udm and can see the same behavior with the started dnsmask in my network. I've looked into the used dnsmasq conf. And there seems to be the problem. The nameserver there is '203.0.113.1', which ist the dnsfilter nameserver as far as I know.

Furthermore that's the content of of the configuration file.

$cat /run/dnsfilter/dns-192.168.2.1-conf.conf

# Configuration of DNS Forwarder
interface=dnsfilter0
no-dhcp-interface=dnsfilter0
conf-file=/run/dnsfilter/dns-192.168.2.1-black.list
conf-file=/run/dnsfilter/dns-192.168.2.1-white.list

That's more evidence the dnsfilter is activated.

It seems like there's a bug in the udm firmware with the dnsfilter. Perhaps it's activated although it's deactivates in the UI.

We have the same environment (udm 1.8.0 and controller 6.0.4). I'am not at home this evening so I cannot ask in the forums. Perhaps you've time to do it?

[nschemel]

I've digged into iptables NAT rules. While I'm definitely no iptables expert at all, I think I found another problem.

There are dnat rules for forwarding all dns requests to '203.0.113.3. And this chain is used for all traffic in PREROUTING, before the new rules from this tutorial, if I understand it correctly.

$ iptables -t nat -L -n -v

Chain PREROUTING
 6569  839K DNSFILTER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
 5551  764K UBIOS_PREROUTING_JUMP  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DNAT       udp  --  br0    *      !192.168.2.2         !192.168.2.2          udp dpt:53 to:192.168.2.2
    0     0 DNAT       tcp  --  br0    *      !192.168.2.2         !192.168.2.2          tcp dpt:53 to:192.168.2.2

Chain DNSFILTER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  *      *       192.168.2.0/24       0.0.0.0/0            tcp dpt:53 to:203.0.113.3:53
   12   835 DNAT       udp  --  *      *       192.168.2.0/24       0.0.0.0/0            udp dpt:53 to:203.0.113.3:53
    0     0 DNAT       tcp  --  *      *       192.168.1.0/24       0.0.0.0/0            tcp dpt:53 to:203.0.113.2:53
 1006 74484 DNAT       udp  --  *      *       192.168.1.0/24       0.0.0.0/0            udp dpt:53 to:203.0.113.2:53

If I delete the PREROUTING rule by the following command the dns queries arrive at my pihole.

# First get rule number
iptables -t nat -L -n -v --line-numbers

# Delete rule by number
iptables -t nat -D PREROUTING 1

WARNING: That's feels not good at all and I don't know if anything is breaking with that change. I only wanted to test it and share my learnings.

[mojo333]

So I'll start off with saying problem SOLVED! In a nutshell it's an issue with v6.0.4 of the Unifi Controller as you suspected @nschemel. I am still running firmware v1.8.0-rc7

I did a painful factory reset of my UDM PRO which then downgraded my Unifi Controller to v5.14.18. Fortunately I had a backup for that version and restored it, then following @boostchicken instructions once i had the base config setup.

As soon as I pointed my DNS to 10.0.5.3 DNS traffic flowed through to NextDNS.

There's definitely a problem with v6.0.4 of the Unifi Controller and it's implementation of DNS filtering which is affecting this solution.

For info @nschemel this is what i now see...

# ps | grep dnsmasq
13560 root     grep dnsmasq
15448 nobody   /usr/sbin/dnsmasq --conf-dir=/run/dnsmasq.conf.d/
15449 root     /usr/sbin/dnsmasq --conf-dir=/run/dnsmasq.conf.d/

and iptables...

# iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 59384 packets, 3494K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1    75505 4794K DNSFILTER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
2    75625 4808K UBIOS_PREROUTING_JUMP  all  --  *      *       0.0.0.0/0            0.0.0.0/0
3    75426 4788K HONEYPOT   all  --  *      *       0.0.0.0/0            0.0.0.0/0
4      125  8160 DNAT       udp  --  br0    *      !10.0.5.3            !10.0.5.3             udp dpt:53 to:10.0.5.3
5        0     0 DNAT       tcp  --  br0    *      !10.0.5.3            !10.0.5.3             tcp dpt:53 to:10.0.5.3
6        0     0 DNAT       udp  --  br30   *      !10.0.5.3            !10.0.5.3             udp dpt:53 to:10.0.5.3
7        0     0 DNAT       tcp  --  br30   *      !10.0.5.3            !10.0.5.3             tcp dpt:53 to:10.0.5.3
8       25  1621 DNAT       udp  --  br50   *      !10.0.5.3            !10.0.5.3             udp dpt:53 to:10.0.5.3
9        0     0 DNAT       tcp  --  br50   *      !10.0.5.3            !10.0.5.3             tcp dpt:53 to:10.0.5.3
10    2426  170K DNAT       udp  --  br99   *      !10.0.5.3            !10.0.5.3             udp dpt:53 to:10.0.5.3
11       0     0 DNAT       tcp  --  br99   *      !10.0.5.3            !10.0.5.3             tcp dpt:53 to:10.0.5.3
Chain DNSFILTER (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Not sure where the issue is as my IPTABLES looks similar to yours, but the DNSFILTER chain is empty - is yours?

Either way, reverting the Unifi Controller to v5.14.18 solved the problem. Let me know if you want any further info.

[boostchicken]

So the problem is the DNSFILTER jump point before. If you remove that it won't intercept the traffic. That being said, I don't know if that is going to have any other consequences. What does the DNSFilter chain have?

[boostchicken]

I dug a bit more, even on 5.x they always have dnsfilter there. I just has no rules. To be clear, on 6.x even if you disable it, it does not work? I would report that to UI. If you specify your own DNS server via DHCP does that fail to work as well?

[boostchicken]

@mojo333 and @nschemel try changing the prerouting rules from -A to -I. It should move our rules above anything else. https://github.com/boostchicken/udm-utilities/blob/c36ba5f51c1126e533989f6c30c906079476f2bc/dns-common/on_boot.d/10-dns.sh#L89 Should become

iptables -t nat -C ${prerouting_rule} || iptables -t nat -I ${prerouting_rule}

[nschemel]

So the problem is the DNSFILTER jump point before. If you remove that it won't intercept the traffic. That being said, I don't know if that is going to have any other consequences. What does the DNSFilter chain have?

My DNSFILTER Chain looks like that:

Chain DNSFILTER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  *      *       192.168.2.0/24       0.0.0.0/0            tcp dpt:53 to:203.0.113.3:53
   12   835 DNAT       udp  --  *      *       192.168.2.0/24       0.0.0.0/0            udp dpt:53 to:203.0.113.3:53
    0     0 DNAT       tcp  --  *      *       192.168.1.0/24       0.0.0.0/0            tcp dpt:53 to:203.0.113.2:53
 1006 74484 DNAT       udp  --  *      *       192.168.1.0/24       0.0.0.0/0            udp dpt:53 to:203.0.113.2:53

So for each of my networks the dns traffic is redirected to 203.0.113.3.

[nschemel]

I dug a bit more, even on 5.x they always have dnsfilter there. I just has no rules. To be clear, on 6.x even if you disable it, it does not work? I would report that to UI. If you specify your own DNS server via DHCP does that fail to work as well?

After a reset the DNSFILTER Chain was there and filled, although dns filtering was disabled in the ui. I enabled it, waited a some time and disabled it again to check if anything is changing. The iptables rules didn't change. but the content of the blacklist configs etc.

It seems like the iptables rules are created always, but the underlaying filtering rules are changed. So with disabled dns filtering the dns traffic is redirected to another dnsmasq instance which just redirects the request to the internal dnsmasq if disabled. Feels a little bit hacky what UI has done there.

[nschemel]

@mojo333 and @nschemel try changing the prerouting rules from -A to -I. It should move our rules above anything else. https://github.com/boostchicken/udm-utilities/blob/c36ba5f51c1126e533989f6c30c906079476f2bc/dns-common/on_boot.d/10-dns.sh#L89

Should become

iptables -t nat -C ${prerouting_rule} || iptables -t nat -I ${prerouting_rule}

That does work and redirects all dns traffic through my pihole.

But if one does not want to redirect all traffic and just wants to use the pihole manually, it gets intercepted by the dnsfilter rule.

So I integrated a rule bases on your prerouting rule explicitly allowing dns traffic to pihole. Once more: I'm not an iptables expert, but that does seem to work.

PREROUTING -i br0 -p udp ! -s 192.168.2.2 -d 192.168.2.2 --dport 53 -j DNAT --to 192.168.2. PREROUTING -i br0 -p tcp ! -s 192.168.2.2 -d 192.168.2.2 --dport 53 -j DNAT --to 192.168.2.

With that rules in place and deactivated redirect of all traffic, I can use the pihole manually on my clients or distribute it via dhcp.

[boostchicken]

All that does is insert those rules before DNSFilter. -A is append -I is insert.

[boostchicken]

As far as the DNSFilter intercepting all traffic even you have it disabled, take that up Ubiquiti. I haven't tried myself but I would be furious if they forced you to use dnsfilter even with it disabled.

[nschemel]

I've upgraded to controller 6.0.8 and it seems that the problem was fixed by Ubiquiti. The DNSFILTER chain is now empty, if the content is deactivated. So nothing special has to be done and the dns script works without customization for me.

[mojo333]

I've upgraded to controller 6.0.8 and it seems that the problem was fixed by Ubiquiti.

Excellent, thanks @nschemel for the update!