mjl- / mox

modern full-featured open source secure mail server for low-maintenance self-hosted email
https://www.xmox.nl
MIT License
3.36k stars 89 forks source link

DNS resolvers configured on your system do not verify DNSSEC... #158

Closed vipas84 closed 2 months ago

vipas84 commented 2 months ago

is there any step-by-step guide how to fix it with unbound? its show-stopper, since emails are regected by gmail and outlook

mjl- commented 2 months ago

Hi @vipas84, I typically run "apt install unbound unbound-anchor" on a debian machine. Then verify unbound has an "auto-trust-anchor-file" configured, check the localhost nameserver is being used (in /etc/resolv.conf), and run dig NS com. and look for "ad" (authentic data) in the line with "flags".

Some example config snippets below.

The root trust anchors are added by installing unbound/unbound-anchor (on some machines (debian on arm maybe?) I needed to run unbound-anchor -a path/to/root/key):

$ cat /etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf
server:
    # The following line will configure unbound to perform cryptographic
    # DNSSEC validation using the root trust anchor.
    auto-trust-anchor-file: "/var/lib/unbound/root.key"

Checking that local resolver is used (you may have to be careful that a systemd isn't overwriting /etc/resolv.conf, on debian I would install package openresolv):

~$ cat /etc/resolv.conf
nameserver 127.0.0.1

Testing local resolver, the "ad" in the "flags:" indicates dnssec-verification, the "server" line at the bottom shows the local resolver is used:

$ dig NS com.

; <<>> DiG 9.19.21-1+b1-Debian <<>> NS com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35217
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;com.                           IN      NS

;; ANSWER SECTION:
com.                    86053   IN      NS      b.gtld-servers.net.
[...]

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Sat Apr 27 08:57:59 CEST 2024
;; MSG SIZE  rcvd: 256

You can enable logging in unbound to debug:

$ cat /etc/unbound/unbound.conf.d/log.conf 
server:
    verbosity: 2
    log-queries: yes

its show-stopper, since emails are regected by gmail and outlook

I'm not so sure your resolver not verifying DNSSEC would be a reason gmail or outlook rejects your email. Why do you think this is causing the issue? Common issues for gmail/outlook to reject are:

In my experience especially the freemail outlook (not enterprise outlook) tends to block messages and blocklist IPs without reason.

RobSlgm commented 2 months ago

It depends on your distro, with a debian 12.5 it's easy (as described above)

The functionality seems to depend on a recent version of unbound (possibly >=1.16.0). With Ubuntu 22.04 LTS - which has an older version of unbound (1.13) - i didn't manage to get it working.

With debian 12.5 all the steps I performed on a cloud instance are as follows:

apt install dns-root-data unbound-anchor unbound

Add following file /etc/unbound/unbound.conf.d/ede.conf with content:

server:
     ede: yes
     val-log-level: 2

Restart unbound systemctl restart unbound

Testing: The dig NS com. is correct but not enough. You will get a correct response (+ad) with an Ubuntu 22.04 setup, but mox will not work.

Using ./mox -loglevel debug dns lookup a xmox.nl the output must show 'authentic=true'.

mjl- commented 2 months ago

Using ./mox -loglevel debug dns lookup a xmox.nl the output must show 'authentic=true'.

Indeed, this is the better test, closest to what mox will see/interpret.

You will get a correct response (+ad) with an Ubuntu 22.04 setup, but mox will not work.

This is unexpected. Do you have any idea why mox would see different results? I'm not aware of any differences in dnssec/"ad" interpretation between dig and mox. Possibly related: the mox quickstart in v0.0.9 and before would check DNSSEC support by resolving NS for ., and some resolvers return the root servers as unauthenticated (but not unbound in my testing). Since v0.0.10, we look up NS on com., which would see the expected DNSSEC status. However, testing with dig should show the same results (when resolving the same name).

RobSlgm commented 2 months ago

No, unfortunately not.

I'm using v0.0.10 for my POC on Ubuntu and now Debian. The only difference I see, is that Ubuntu uses a older version of unbound, which doesn't support extended DNS errors. If they are really required, I don't now.

vipas84 commented 2 months ago

I run Mox on Ubuntu 22.04 LTS (Hetzner VPS), so I do have an issue with outdated unbound version (Version 1.13.1) from Ubuntu repositories. I will experiment a bit and if find no solution will have to migrate to Debian 12,5...

mjl- commented 2 months ago

which doesn't support extended DNS errors. If they are really required, I don't now.

They aren't required for DNSSEC. But they are helpful in case DNSSEC verification fails, because it allows giving details about the failure, instead of getting a generic dns "SERVFAIL" response. For sending TLS reports, it is helpful to give these details about failures.

@vipas84 What is in your /etc/resolv.conf? Does it have a line like options edns0 or options edns0 trust-ad? Could be worth a try with that. Could you run the dig and mox lookup commands and paste the output? Maybe there's a hint in there.

vipas84 commented 2 months ago

/etc/resolv.conf nameserver 127.0.0.53

dig ; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> NS com. ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43479 ;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;com. IN NS

;; ANSWER SECTION: com. 5713 IN NS d.gtld-servers.net. com. 5713 IN NS b.gtld-servers.net. com. 5713 IN NS c.gtld-servers.net. com. 5713 IN NS k.gtld-servers.net. com. 5713 IN NS m.gtld-servers.net. com. 5713 IN NS f.gtld-servers.net. com. 5713 IN NS a.gtld-servers.net. com. 5713 IN NS i.gtld-servers.net. com. 5713 IN NS l.gtld-servers.net. com. 5713 IN NS h.gtld-servers.net. com. 5713 IN NS g.gtld-servers.net. com. 5713 IN NS e.gtld-servers.net. com. 5713 IN NS j.gtld-servers.net.

;; Query time: 0 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Mon Apr 29 07:37:13 UTC 2024 ;; MSG SIZE rcvd: 256

mox

debug: dns lookup result; pkg=dns; type=ip; network=ip4; host=xmox.nl.; resp=[84.22.96.237]; authentic=false; duration=10.965378ms records (1, without dnssec):

RobSlgm commented 2 months ago

From a ubuntu 22.04 LTS VPS hetzner I get this responses:

dig NS com.

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> NS com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44131
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;com.                           IN      NS

;; ANSWER SECTION:
com.                    86220   IN      NS      k.gtld-servers.net.
com.                    86220   IN      NS      d.gtld-servers.net.
com.                    86220   IN      NS      e.gtld-servers.net.
com.                    86220   IN      NS      m.gtld-servers.net.
com.                    86220   IN      NS      j.gtld-servers.net.
com.                    86220   IN      NS      c.gtld-servers.net.
com.                    86220   IN      NS      f.gtld-servers.net.
com.                    86220   IN      NS      h.gtld-servers.net.
com.                    86220   IN      NS      l.gtld-servers.net.
com.                    86220   IN      NS      g.gtld-servers.net.
com.                    86220   IN      NS      a.gtld-servers.net.
com.                    86220   IN      NS      b.gtld-servers.net.
com.                    86220   IN      NS      i.gtld-servers.net.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP)
;; WHEN: Mon Apr 29 07:58:42 UTC 2024
;; MSG SIZE  rcvd: 256
./mox -loglevel debug dns lookup a xmox.nl
debug: dns lookup result; pkg=dns; type=ip; network=ip4; host=xmox.nl.; resp=[84.22.96.237]; authentic=false; duration=3.34154ms
records (1, without dnssec):
- 84.22.96.237
delv xmox.nl
; fully validated
xmox.nl.                3600    IN      A       84.22.96.237
xmox.nl.                3600    IN      RRSIG   A 13 2 3600 20240510043612 20240426035243 58294 xmox.nl. gLlmZVBXgz/Cdyv161kQXmX9XAxf6LE/gjvq1rVygpCNmrPaynn6GIVe cndVXPL0ER9LVQ/y0JcwqZrbtUbzlQ==

Setup of instance

Pure instance with unbound (apt add unbound) and following changes:

In /etc/systemd/resolved.conf change

DNS=127.0.0.1
DNSSEC=yes
DNSStubListener=no

and restart systemctl restart systemd-resolved.service and systemctl restart unbound.service

mjl- commented 2 months ago

@vipas84 The nameserver 127.0.0.53 is an IP of systemd-resolved. So you're not actually talking to the unbound when resolving. Dig shows the same IP. I have systemd-resolved removed on my machines, and debian package openresolv installed to keep my own /etc/resolv.conf.

@RobSlgm That dig command indeed looks like DNSSEC verification works as intended, but mox doesn't see the verification. What is in your /etc/resolv.conf? And is ./mox -loglevel debug dns lookup ns com. also not "authentic"? I suppose the /etc/systemd/resolved.conf change is to prevent systemd from starting its own resolver and putting it in /etc/resolv.conf. Does the DNSSEC=yes do anything in this case? (I have a vague recollection of it being trouble when used with systemds own resolverd). The delv command isn't a great way to check: It doesn't check if the recursive resolver did DNSSEC, but fetches all DNSSEC-related records itself and does the verification itself. So it's a good check if a domain has DNSSEC configured, but not if a (local) recursive resolver is doing DNSSEC verification.

RobSlgm commented 2 months ago

Thanks for the hints!

Now it looks like

./mox -loglevel debug dns lookup a xmox.nl
debug: dns lookup result; pkg=dns; type=ip; network=ip4; host=xmox.nl.; resp=[84.22.96.237]; authentic=true; duration=38.467462ms
records (1, with dnssec):
- 84.22.96.237
./mox -loglevel debug dns lookup ns com.
debug: dns lookup result; pkg=dns; type=ns; name=com.; resp="[host=c.gtld-servers.net.;host=g.gtld-servers.net.;host=h.gtld-servers.net.;host=j.gtld-servers.net.;host=l.gtld-servers.net.;host=k.gtld-servers.net.;host=a.gtld-servers.net.;host=d.gtld-servers.net.;host=e.gtld-servers.net.;host=f.gtld-servers.net.;host=b.gtld-servers.net.;host=i.gtld-servers.net.;host=m.gtld-servers.net.]"; authentic=true; duration="210.756µs"
ns records (13, with dnssec):
- &{c.gtld-servers.net.}
- &{g.gtld-servers.net.}
- &{h.gtld-servers.net.}
- &{j.gtld-servers.net.}
- &{l.gtld-servers.net.}
- &{k.gtld-servers.net.}
- &{a.gtld-servers.net.}
- &{d.gtld-servers.net.}
- &{e.gtld-servers.net.}
- &{f.gtld-servers.net.}
- &{b.gtld-servers.net.}
- &{i.gtld-servers.net.}
- &{m.gtld-servers.net.}

Setup of instance

Stop and disable systemd-resolved systemctl disable systemd-resolved.service

Reboot instance to make sure nothing get's overwritten.

After a reboot no /etc/resolv.conf exists anymore. The changes in /etc/systemd/resolved.conf are no longer relevant. But you are right, DNSSEC=yes or no doesn't matter at all.

But be aware, I'm no expert on this matter (e.g. side effects of disabling the 'resolved'-service). I'm just happy that mox dns lookup gives now consistently authentic=true responses.

vipas84 commented 2 months ago

Thank you for your answer! I'm afraid I have not enough knowledge to understand this general answer. Would you be so kind as to explain step by step what needs to be done to fix the issue. thanks in advance

RobSlgm commented 2 months ago

Manual setup

Create a new instance on Hetzner - either with the web ui or with hcloud (hcloud server create). Don't use a pre-existing server.

Connect to your new server and as root perform:

To ensure you have fresh packages (but not related to the question of DNSSEC with mox):

apt update; apt upgrade; apt autoremove

Install unbound and no further packages at the moment:

apt install unbound
systemctl disable systemd-resolved.service
reboot

Please consider these are minimal instructions to start mox with DNSSEC; hardening of the server, firewall rules etc. are not really optional but depend on your requirements.

Cloud init to re-produce setup

As alternative approach to the manual setup is using cloud-init files. This is well supported by Hetzner with hcloud cli.

The cloud init (stripped to the minimum) would be:

Store as my.cloud-init.yaml

#cloud-config
hostname: mail01
users:
  - name: mox
    groups: users
    shell: /bin/bash
packages:
  - unbound
package_update: true
package_upgrade: true
runcmd:
    - systemctl disable systemd-resolved.service
    - reboot

and you would use on your LOCAL machine hcloud server create --type cpx11 --name mail01 --image ubuntu-22.04 --ssh-key your_ssh_key_name_in_hetzner_console --user-data-from-file my.cloud-init.yaml Wait some few minutes and continue with the verification

Verification

Perform after you connected again following tests:

dig NS com.
dig NS xmox.nl

Check that the output contains in the line with flags: 'ad', see above flags: qr rd ra ad; QUERY; ....

If this is not the case, check your setup again.

For your own mail domain the same must be true

dig NS example.com

If the tests with com. and xmox.nl are working and yours not, check if your domain has DNSSEC entries at all

devl example.com

must output something similar what you see with devl xmox.nl.

Possible causes: DNSSEC is not enabled at your domain registrar.

If all is fine, then follow the instruction to install mox (see doc quickstart), to highlight some points:

create user mox useradd -m -d /home/mox mox. This is not needed if you used the cloud-init approach.

still as user root

cd /home/mox
wget https://beta.gobuilds.org/github.com/mjl-/mox@latest/linux-amd64-latest/dl -O mox
chmod +x ./mox

Now you are able to verify the DNSSEC setup with mox directly. That's

./mox -loglevel debug dns lookup a xmox.nl
./mox -loglevel debug dns lookup a example.com

Now you may continue according to the mox documentation (mox quickstart...)

RobSlgm commented 2 months ago

Not to muddy the water even more, but as already mentioned @mjl-: You receiving or sending mail to gmail or outlook does not depend on DNSSEC and/or DANE.

With Hetzner make sure you set the reverse DNS:

hcloud server set-rdns --hostname mail.example.com --ip your_ip4 your-server-name and the same for IPVv6

You may need to request through Hetzner's support to open your port 25. By default it's blocked.

vipas84 commented 2 months ago

thanks, it looks like i missed 2 points: 1) systemctl disable systemd-resolved.service 2) I use Hetzner DNS to manage DNS records and turned out that it does not support DNSSEC

Will migrate to cloudflare and let you know the final result

vipas84 commented 2 months ago

I still experience issues:

I made step-by step guideline for Debian 12 , tell me what I did wrong


💡 Debian12 is preferable, Ubuntu has outdated unbound version

apt install unbound unbound-anchor unbound version nano /etc/unbound/unbound.conf.d/root-auto-trust-anchor-file.conf

server:
    # The following line will configure unbound to perform cryptographic
    # DNSSEC validation using the root trust anchor.
    auto-trust-anchor-file: "/var/lib/unbound/root.key"

nano /etc/resolv.conf

nameserver 127.0.0.1`

apt install openresolv --fix-missing reboot

nano /etc/unbound/unbound.conf.d/log.conf

server:
    verbosity: 2
    log-queries: yes

systemctl disable systemd-resolved.service reboot

wait few minutes & continue with verification dig NS com. dig NS xmox.nl

in results there should be flags: qr rd ra ad

RobSlgm commented 2 months ago

For Debian 12.5 - which is the setup I'm currently testing (in the final stages) - the setup with a Hetzner VPS is as follows:

Install

apt install dns-root-data unbound-anchor unbound

add a file /etc/unbound/unbound.conf.d/ede.conf with this content:

server:
    ede: yes
    val-log-level: 2

(which is most probably not even necessary)

And reboot to be on the safe side.

That's really all.

vipas84 commented 2 months ago

so you mean apt install unbound-anchor unbound is enough?

RobSlgm commented 2 months ago

No, it's apt install dns-root-data unbound-anchor unbound

Add dns-root-data

vipas84 commented 2 months ago

I rebuild server again and ...

Checking if DNS resolvers are DNSSEC-verifying...

It looks like the DNS resolvers configured on your system do not
verify DNSSEC, or aren't trusted (by having loopback IPs or through "options
trust-ad" in /etc/resolv.conf).  Without DNSSEC, outbound delivery with SMTP
used unprotected MX records, and SMTP STARTTLS connections cannot verify the TLS
certificate with DANE (based on a public key in DNS), and will fall back to
either MTA-STS for verification, or use "opportunistic TLS" with no certificate
verification.

Recommended action: Install unbound, a DNSSEC-verifying recursive DNS resolver,
ensure it has DNSSEC root keys (see unbound-anchor), and enable support for
"extended dns errors" (EDE, available since unbound v1.16.0). Test with
"dig com. ns" and look for "ad" (authentic data) in response "flags".
RobSlgm commented 2 months ago

Then maybe the /etc/unbound/unbound.conf.d/ede.conf is needed and not optional. Just add it and restart unbound service.

RobSlgm commented 2 months ago

This is the cloud-init yaml file, which I use. I stripped ssh and wireguard related stuff

#cloud-config
hostname: mail01
users:
  - name: mox
    groups: users
    shell: /bin/bash
packages:
  - dns-root-data
  - unbound-anchor
  - unbound
package_update: true
package_upgrade: true
runcmd:
  - reboot
write_files:
  - path: /etc/unbound/unbound.conf.d/ede.conf
    permission: "0755"
    content: |
      server:
          ede: yes
          val-log-level: 2
vipas84 commented 2 months ago

yes, it turned out to be mandatory...

So, for those who will encounter similar preconfiguration issue: 1) use Debian 12 2) and do the following minimal pre configuration

apt update && apt install dns-root-data unbound-anchor unbound nano /etc/unbound/unbound.conf.d/ede.conf

server:
    ede: yes
    val-log-level: 2

@mjl- I believe you should consider to put this example guide for console output or somewhere in official step-by-step guide to increase number of people who are capable to run mox

vipas84 commented 2 months ago

use Debian 12 and do the following minimal pre configuration apt update && apt install dns-root-data unbound-anchor unbound nano /etc/unbound/unbound.conf.d/ede.conf

server:
    ede: yes
    val-log-level: 2