DietPi-Software | Unbound: Fixes and enhancements

Gill-Bates commented 5 years ago

Hi There, I would appreachiate the cloudflare deamon in dietpi. Right now, I have to set it up manually:

https://docs.pi-hole.net/guides/dns-over-https/

Could you incorporate the cloudflare deamon into the software catalouge of Dietpi?

MichaIng commented 5 years ago

@WTFMaster Thanks for your request.

There is another tool that does that thing, if I get it right: https://dnscrypt.info/

Related request: https://github.com/Fourdee/DietPi/issues/1245

Maybe someone can explain what those tools do, what I get is:

They will accept incoming DNS requests.
They will proxy them to an external DNS provider, but verify it's identity by checking SSL certificates.
So you need to configure your system (and/or Pi-hole, if in use) to send DNS requests to the tool (loopback IP + configured port).
I guess you cannot simply use any DNS provider you want then, but it needs to support this. Since usually HTTPS certificates are only made/valid for domains, e.g. plain Google DNS IP will not work? Although cloudflared suggests plain CloudFlare DNS server IP 🤔.

In cloudflared you can configure the upstream DNS: https://docs.pi-hole.net/guides/dns-over-https/ DNSCrypt allows that as well, although the guide suggests dnscrypt.nl-ns0: https://itchy.nl/raspberry-pi-3-with-openvpn-pihole-dnscrypt

Further reading: https://community.cloudflare.com/t/dnscrypt-great-proxy-alternative-to-cloudflared/14650

So generally I am opened to implement both of them. It doesn't seem too much work, and from system side, only the DNS server needs to be changed, really no big deal. Although we have DNSCrypt in combination with Pi-hole, I think it is worth to add independently. On Pi-hole install simply check for running DNS proxies (cloudflared/DNSCrypt) and set Pi-hole to use them. Actually, I believe if the system (resolv.conf) is already configured to use it, Pi-hole will automatically use the same upstream DNS on install?

MindTooth commented 5 years ago

I've used DNSCrypt-Proxy for several years. And the latest rewrite includes caching, meaning no need for DNSMasq any longer. Would love to see it include by default, and maybe a supplied interface to do changes.

MichaIng commented 5 years ago

Just to mention, there are several ideas/ways to mask your DNS requests:

DNS-over-HTTPS (DoH), currently forced by browser developers
DNS-over-TLS (DoT), which has less overhead but does not allow to use the same connection for multiple requests (DNS and others) natively (?)
DNSCrypt: Another separate protocol for encrypted DNS, that was believed to be not continued anymore but gained new development meanwhile.
Running your own local DNS server, e.g. via unbound.

A list of public free DNS providers and which supports which encryption protocol is available here: https://en.wikipedia.org/wiki/Public_recursive_name_server

It would be wonderful if we could implement all of them, however a client is always required since since Linux/Debian itself does not support this natively. DNSCrypt-proxy is known of course, cloudflared AFAIK uses DoH, for DoT I'm not sure currently.

MichaIng commented 4 years ago

Unbound supports upstream DoT and downstream DoT and DoH, so covers most reasonable cases. AFAIK there is not really an upstream DoH "resolver", meaning a node that can be connected to via any method to resolve hostnames and itself resolves them via DoH at public upstream DoH DNS provider like 9.9.9.9. DoH is mostly something that is done by the individual clients, e.g. the web browser itself, but it does not make much sense to use it for upstream resolving from unbound since all its benefits don't apply (mix DNS requests with existing HTTPS connections and traffic) and only the overhead (HTTP) stays.

So the only missing DNS feature, after unbound implementation, is DNSCrypt-Proxy, either as an alternative to unbound, or even in combination with unbound where DNSCrypt is used to encrypt upstream DNS requests and unbound mostly for DNS caching.

Ernstian commented 3 years ago

I recently switched to the dev branch and tried Unbound, but if I install it, I get an error while it tries to restart the service

[  OK  ] DietPi-Software | Setting in /etc/pihole/setupVars.conf adjusted: PIHOLE_DNS_1=127.0.0.1#5353
[  OK  ] DietPi-Software | Setting in /etc/pihole/setupVars.conf adjusted: PIHOLE_DNS_2=
[FAILED] DietPi-Software | systemctl restart unbound
[  OK  ] DietPi-Software | systemctl restart unbound

Retrying the command installs it just fine. This happened on 3/3 installations.

After installation, Unbound doesn't work. I'm running Pi-Hole, and while my local IP and port is written under custom DNS servers, if I disable all other DNS servers, nothing resolves. Unbound is running, the log has no errors, and Unbound is running in htop.

-- Logs begin at Thu 2019-02-14 10:11:58 GMT, end at Fri 2020-12-04 23:58:55 GMT. --
Dec 04 23:56:45 DietPi systemd[1]: Starting Unbound DNS server...
Dec 04 23:56:51 DietPi package-helper[431]: /var/lib/unbound/root.key has content
Dec 04 23:56:51 DietPi package-helper[431]: fail: the anchor is NOT ok and could not be fixed
Dec 04 23:57:06 DietPi unbound[470]: [1607126226] unbound[470:0] info: start of service (unbound 1.9.0).
Dec 04 23:57:06 DietPi systemd[1]: Started Unbound DNS server.

If I keep the IPv6-DNS active (as it is default after installing Unbound), I'll get the following result (commands taken from here, I just changed the port to 5353 ). It should not result in a SERVFAIL.

root@DietPi:~# dig pi-hole.net @127.0.0.1 -p 5353

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> pi-hole.net @127.0.0.1 -p 5353
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 35995
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;pi-hole.net.                   IN      A

;; Query time: 30 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Sat Dec 05 01:10:39 CET 2020
;; MSG SIZE  rcvd: 40

root@DietPi:~# dig sigfail.verteiltesysteme.net @127.0.0.1 -p 5353

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> sigfail.verteiltesysteme.net @127.0.0.1 -p 5353
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 7850
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;sigfail.verteiltesysteme.net.  IN      A

;; Query time: 36 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Sat Dec 05 01:11:50 CET 2020
;; MSG SIZE  rcvd: 57

root@DietPi:~# dig sigok.verteiltesysteme.net @127.0.0.1 -p 5353

; <<>> DiG 9.11.5-P4-5.1+deb10u2-Debian <<>> sigok.verteiltesysteme.net @127.0.0.1 -p 5353
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52011
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;sigok.verteiltesysteme.net.    IN      A

;; Query time: 29 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Sat Dec 05 01:11:56 CET 2020
;; MSG SIZE  rcvd: 55

Minor nitpick here: the custom DNS is not removed from Pi-Hole if Unbound is uninstalled.

Other things I'm running: Pi-Hole, OwnCloud, fail2ban

ravenclaw900 commented 3 years ago

Strange, since everything works for me (even automatically removing the Unbound DNS from Pi-hole on uninstall). Reading a Reddit post, it seems that some ISPs/Routers can intercept DNS requests, even when they're not supposed to, stopping Unbound from opening a secure connection. What happens if you remove Unbound from Pi-hole and run dig @127.0.0.1 pi-hole.net +norecurse?

Ernstian commented 3 years ago

Well, I'm using the Easybox 804 too. I guess that's the problem, then - at least it's not a problem with the implementation!

MichaIng commented 3 years ago

I'll also run another test, just to be sure.

faxesystem commented 3 years ago

I also ran into the problem when installing unbound and pihole on a fresh dietpi.

[ OK ] DietPi-Software | Setting in /etc/pihole/setupVars.conf adjusted: PIHOLE_DNS_1=127.0.0.1#5353 [ OK ] DietPi-Software | Setting in /etc/pihole/setupVars.conf adjusted: PIHOLE_DNS_2= [FAILED] DietPi-Software | systemctl restart unbound [ OK ] DietPi-Software | systemctl restart unbound

But for me retrying the command did not fix it and I had to cancel the installation.

MichaIng commented 3 years ago

Strange, I faced it as well now, however a retry fixed it here as well. Here the reason:

Dec 20 22:05:49 VM-Buster unbound[17816]: [1608498349] unbound[17816:0] error: can't bind socket: Address already in use for ::1 port 53
Dec 20 22:05:49 VM-Buster unbound[17816]: [1608498349] unbound[17816:0] fatal error: could not open ports

The override config is created before the restart is done: https://github.com/MichaIng/DietPi/blob/master/dietpi/dietpi-software#L8795-L8806

However, ss -tulp shows that actually dietpi-pihole.conf does not really override port and listening IP but adds it... More strangely after it worked once, unbound binds to 0.0.0.0:5353 and 127.0.0.1:5353, hence the port is overridden but the IP added 🤔.

I also tried to add it like:

server:
    port: 5353
    interface: 127.0.0.1

(the server: block definition) but that does not have any effect.

Bad that v6.34 has just been released. It seems like overriding settings does not work as expected, hence we need to change port and listening IP in our main config when installing Pi-hole.

MichaIng commented 3 years ago

@faxesystem That means you can change port and interface settings in /etc/unbound/unbound.conf.d/dietpi.conf to 5353 and 127.0.0.1 as above and redo the install. The settings file will be preserved and unbound start should then succeed directly.

faxesystem commented 3 years ago

Will try this tonight. So this is something that will have to be fixed in a future DietPi release?

MichaIng commented 3 years ago

Yes, at least a beautiful long-term solution, as it requires likely some research and tests. But I'm sure we can hack an ugly but reliable solution meanwhile via MOTD script 😉.

faxesystem commented 3 years ago

Hmmmm does not work for me. I choose the open a sub shell when the error occurred and changed port and interface in dietpi.conf from 0.0.0.0 and 53 to 127.0.0.1 and 5353. After that I re-ran the command which still resulted in the same error.

systemctl status unbound.service
● unbound.service - Unbound DNS server
   Loaded: loaded (/lib/systemd/system/unbound.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2020-12-21 20:48:09 CET; 24s ago
     Docs: man:unbound(8)
  Process: 19566 ExecStartPre=/usr/lib/unbound/package-helper chroot_setup (code=exited, status=0/SUCCESS)
  Process: 19569 ExecStartPre=/usr/lib/unbound/package-helper root_trust_anchor_update (code=exited, status=0/SUCCESS)
  Process: 19573 ExecStart=/usr/sbin/unbound -d $DAEMON_OPTS (code=exited, status=1/FAILURE)
 Main PID: 19573 (code=exited, status=1/FAILURE)

Dec 21 20:48:09 thesquirrelmaster systemd[1]: Stopped Unbound DNS server.
Dec 21 20:48:09 thesquirrelmaster systemd[1]: unbound.service: Start request repeated too quickly.
Dec 21 20:48:09 thesquirrelmaster systemd[1]: unbound.service: Failed with result 'exit-code'.
Dec 21 20:48:09 thesquirrelmaster systemd[1]: Failed to start Unbound DNS server.
Dec 21 20:48:09 thesquirrelmaster systemd[1]: unbound.service: Start request repeated too quickly.
Dec 21 20:48:09 thesquirrelmaster systemd[1]: unbound.service: Failed with result 'exit-code'.
Dec 21 20:48:09 thesquirrelmaster systemd[1]: Failed to start Unbound DNS server.
Dec 21 20:48:13 thesquirrelmaster systemd[1]: unbound.service: Start request repeated too quickly.
Dec 21 20:48:13 thesquirrelmaster systemd[1]: unbound.service: Failed with result 'exit-code'.
Dec 21 20:48:13 thesquirrelmaster systemd[1]: Failed to start Unbound DNS server.

MichaIng commented 3 years ago

Strange, in your case I also don't see a reason for the failure.

Please try to remove /etc/unbound/unbound.conf.d/dietpi-pihole.conf additionally and paste the output of:

journalctl -u unbound
ss -tulp

Else. probably we see more when you manually run the binary: /usr/sbin/unbound

faxesystem commented 3 years ago

journalctl -u unbound

Dec 21 20:21:17 thesquirrelmaster systemd[1]: Starting Unbound DNS server...
Dec 21 20:21:17 thesquirrelmaster package-helper[1275]: /var/lib/unbound/root.key does not exist, copying from /usr/share/dns/root.key
Dec 21 20:21:17 thesquirrelmaster package-helper[1275]: /var/lib/unbound/root.key has content
Dec 21 20:21:17 thesquirrelmaster package-helper[1275]: success: the anchor is ok
Dec 21 20:21:17 thesquirrelmaster unbound[1280]: [1280:0] notice: init module 0: subnet
Dec 21 20:21:17 thesquirrelmaster unbound[1280]: [1280:0] notice: init module 1: validator
Dec 21 20:21:17 thesquirrelmaster unbound[1280]: [1280:0] notice: init module 2: iterator
Dec 21 20:21:17 thesquirrelmaster unbound[1280]: [1280:0] info: start of service (unbound 1.9.0).
Dec 21 20:21:17 thesquirrelmaster systemd[1]: Started Unbound DNS server.
Dec 21 20:32:01 thesquirrelmaster unbound[1280]: [1280:0] info: service stopped (unbound 1.9.0).
Dec 21 20:32:01 thesquirrelmaster unbound[1280]: [1280:0] info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Dec 21 20:32:01 thesquirrelmaster systemd[1]: Stopping Unbound DNS server...
Dec 21 20:32:01 thesquirrelmaster unbound[1280]: [1280:0] info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0
Dec 21 20:32:01 thesquirrelmaster systemd[1]: unbound.service: Succeeded.
Dec 21 20:32:01 thesquirrelmaster systemd[1]: Stopped Unbound DNS server.
Dec 21 20:32:01 thesquirrelmaster systemd[1]: Starting Unbound DNS server...
Dec 21 20:32:01 thesquirrelmaster package-helper[15501]: /var/lib/unbound/root.key has content
Dec 21 20:32:01 thesquirrelmaster package-helper[15501]: success: the anchor is ok
Dec 21 20:32:02 thesquirrelmaster unbound[15506]: [1608579122] unbound[15506:0] error: cannot parse ip address: '.0/24'
Dec 21 20:32:02 thesquirrelmaster unbound[15506]: [1608579122] unbound[15506:0] error: cannot parse access control: .0/24 allow
Dec 21 20:32:02 thesquirrelmaster unbound[15506]: [1608579122] unbound[15506:0] fatal error: Could not setup access control list
Dec 21 20:32:02 thesquirrelmaster systemd[1]: unbound.service: Main process exited, code=exited, status=1/FAILURE

ss -tulp

Netid   State    Recv-Q   Send-Q     Local Address:Port     Peer Address:Port
udp     UNCONN   0        0                0.0.0.0:53            0.0.0.0:*      users:(("pihole-FTL",pid=17135,fd=4))
udp     UNCONN   0        0                      *:53                  *:*      users:(("pihole-FTL",pid=17135,fd=6))
tcp     LISTEN   0        1024             0.0.0.0:80            0.0.0.0:*      users:(("lighttpd",pid=21827,fd=4))
tcp     LISTEN   0        32               0.0.0.0:53            0.0.0.0:*      users:(("pihole-FTL",pid=17135,fd=5))
tcp     LISTEN   0        1000             0.0.0.0:22            0.0.0.0:*      users:(("dropbear",pid=376,fd=3))
tcp     LISTEN   0        5              127.0.0.1:4711          0.0.0.0:*      users:(("pihole-FTL",pid=17135,fd=10))
tcp     LISTEN   0        1024                [::]:80               [::]:*      users:(("lighttpd",pid=21827,fd=5))
tcp     LISTEN   0        32                  [::]:53               [::]:*      users:(("pihole-FTL",pid=17135,fd=7))
tcp     LISTEN   0        1000                [::]:22               [::]:*      users:(("dropbear",pid=376,fd=4))
tcp     LISTEN   0        5                  [::1]:4711             [::]:*      users:(("pihole-FTL",pid=17135,fd=11))

Will try to setup unbound only on a fresh dietpi.

MichaIng commented 3 years ago

Okay, the ports are not the issue in your case, but:

Dec 21 20:32:02 thesquirrelmaster unbound[15506]: [1608579122] unbound[15506:0] error: cannot parse ip address: '.0/24'
Dec 21 20:32:02 thesquirrelmaster unbound[15506]: [1608579122] unbound[15506:0] error: cannot parse access control: .0/2
4 allow

It's the third access-control: line in /etc/unbound/unbound.conf.d/dietpi.conf which we derive via echo "$(hostname -I | grep -o '192.168.[[:digit:]]').0/24 allow". That is not 100% reliably since the system then needs to have an IP address assigned that starts with 192.168. which is not always the case, as there are other local IP ranges. We should 1. allow all private IP ranges and 2. check whether a valid IP has been found, else ask the user to enter the IP range of its local network (the ones that shall have access to Unbound).

Can you please paste:

hostname -I
sed -n 4p /run/dietpi/.network

faxesystem commented 3 years ago

Yes this will be the problem. I'm using 10.19.82.0/24 in my home network. The Pi has the static IP 10.19.82.253.

hostname -I
10.19.82.253 fd00::ba27:ebff:fe5e:aaf3 2a02:8109:98c0:a67:ba27:ebff:fe5e:aaf3

sed -n 4p /run/dietpi/.network
10.19.82.253

MichaIng commented 3 years ago

Okay, we'll make this better with next release. Until then the following should do (if you didn't fix it yet):

sed -i 's/access-control: .0/24 allow/access-control: 10.19.82.0/24 allow/' /etc/unbound/unbound.conf.d/dietpi.conf
systemctl restart unbound

faxesystem commented 3 years ago

Yes already did that and now everything is working. :) Thanks for your help and sticking with me. #staysafe

faxesystem commented 3 years ago

Another "bug" that a just recognized is that in Pi-hole - Custom 1 (IPv4) is correctly set to 127.0.0.1#5353 but Custom 3 (IPv6) is empty whereas it should be ::1#5353 in order to resolve IPv6 through Unbound if I'm not mistaken.

ravenclaw900 commented 3 years ago

If I add that in, it resolves back to localhost#5353 in the chart, just like the IPv4 address. Also, all of my IPv6 queries seemed to be working before anyway.

MichaIng commented 3 years ago

Yes I recognised this as well, not sure if forgotten or intended @ravenclaw900? However, it is not required to connect via IPv6 in order to resolve hostnames to IPv6 addresses, AFAIK. And since every local network will have IPv4 addresses (at least additionally to IPv6), it should work fine. Otherwise, consequently we'd need to derive the local IPv6 network range as well for access-control.

Joulinar commented 3 years ago

I recall an old post from PiHole GitHub that it was not possible to have both IPv4 as well as IPv6 on custom entries. https://github.com/pi-hole/AdminLTE/issues/1499#issuecomment-659587074

Joulinar commented 3 years ago

guys, I know a total n00b question. But which upstream DNS is used on unbound?

ravenclaw900 commented 3 years ago

It uses the DNS root servers in https://www.internic.net/domain/named.root.

Joulinar commented 3 years ago

inside PiHole, I see queries answered by Cloudflare.

ravenclaw900 commented 3 years ago

Strange, the queries should just be resolved by localhost#5353. It sounds like Cloudflare is selected instead of Unbound.

Joulinar commented 3 years ago

that's why I'm asking. Custom DNS is set to 127.0.0.1#5353 as expected but still queries are resolved by Cloudflare. Even after a reboot. 🤔

ok I got it working now. It changed the moment I hit save button again on PiHole > Settings > DNS. Some more testing would be needed to see if I can replicate the behaviour.

ravenclaw900 commented 3 years ago

Mine had 1 request answered by Cloudflare, before the configuration changed it.

MichaIng commented 3 years ago

Btw, we apply Unbound as (only) DNS via /etc/pihole/setupVars.conf, but while this is important for Pi-hole repairs/reinstalls and updates, it does not affect the currently used DNS. For that we'd need to edit /etc/dnsmasq.d/01-pihole.conf and adjust the two server entries (https://github.com/pi-hole/pi-hole/blob/master/advanced/01-pihole.conf#L32-L33), respectively remove both and add an own server=127.0.0.1#5353.

ravenclaw900 commented 3 years ago

We'd probably need to do the same when uninstalling Unbound and replacing it with Quad9, then. Possibly the reason for the 'minor nitpick' in https://github.com/MichaIng/DietPi/issues/2409#issuecomment-739154892.

Joulinar commented 3 years ago

good find @MichaIng That would explain why on an existing installation still old DNS sever are using instead of unbound

MichaIng commented 3 years ago

Issues are addressed with this PR: #4022 Additionally I made the default configuration a dedicated download: https://github.com/MichaIng/DietPi/commit/a23d799a9c61c4b9c8df1a09a9b6a51a0e31e7c4 To be more compatible, all local IPv4 ranges are allowed to access since the previous 192.168.* only estimation does not apply in all cases and it is generally hard to reliably obtain the actual local target subnet which does not always match the one from the www interface, e.g. with a dedicated local network or even a VPN which shall have access.

One question: identity: "Server" Shall we give it a little more meaningful name, like "DietPi Unbound server" or has it a privacy reason to be unspecific? I have no idea where this would show up...

ravenclaw900 commented 3 years ago

To be honest, I'm not even sure if it's needed. Right before it is hide-identity: yes, which refuses all the queries (id.server and hostname.bind) that would ask for the server identity anyway.

MichaIng commented 3 years ago

Okay, I think it's fine to keep both to assure max privacy. Probably we can sort and explain (via comments) the config file a bid to make it clearer why we set what.

Joulinar commented 3 years ago

Guys, there is a user on our forum looking into config file creating as well. Next to this he is looking for some logging functionality and a graph that might be provided by unbound

https://dietpi.com/phpbb/viewtopic.php?f=9&t=8469

MichaIng / DietPi

DietPi-Software | Unbound: Fixes and enhancements #2409