search

systemd / systemd

The systemd System and Service Manager
https://systemd.io
GNU General Public License v2.0
13.24k stars 3.79k forks source link

resolved fails to resolve some names when DNSSEC is enabled #24747

Closed Tachi107 closed 3 months ago

Tachi107 commented 2 years ago

systemd version the issue has been seen with

251.3-1~bpo11+1

Used distribution

Debian 11

Linux kernel version used

5.15.32-v8+

CPU architectures issue was seen on

aarch64

Component

systemd-resolved

Expected behaviour you didn't see

$ resolvectl query uninsubria-it.mail.protection.outlook.com
uninsubria-it.mail.protection.outlook.com: 104.47.0.36 -- link: wlan0
                                           104.47.2.36 -- link: wlan0

-- Information acquired via protocol DNS in 1.9709s.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

Unexpected behaviour you saw

$ resolvectl query uninsubria-it.mail.protection.outlook.com                                       
uninsubria-it.mail.protection.outlook.com: resolve call failed: All attempts to contact name servers or networks failed

$ journalctl --pager-end --output=cat --unit=systemd-resolved
DNSSEC validation failed for question uninsubria-it.mail.protection.outlook.com IN SOA: failed-auxiliary
DNSSEC validation failed for question uninsubria-it.mail.protection.outlook.com IN A: failed-auxiliary
DNSSEC validation failed for question uninsubria-it.mail.protection.outlook.com IN AAAA: failed-auxiliary

Steps to reproduce the problem

$ cat /etc/systemd/resolved.conf.d/*.conf
[Resolve]
DNS=9.9.9.9#dns.quad9.net
DNSOverTLS=yes
DNSSEC=yes

$ resolvectl status      
Global
       Protocols: +LLMNR +mDNS +DNSOverTLS DNSSEC=yes/supported
resolv.conf mode: stub
     DNS Servers: 9.9.9.9#dns.quad9.net

Link 2 (eth0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS +DNSOverTLS DNSSEC=yes/supported
Current DNS Server: 192.168.178.1
       DNS Servers: 192.168.178.1

Link 3 (wlan0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported

$ resolvectl flush-caches

$ resolvectl query uninsubria-it.mail.protection.outlook.com 
uninsubria-it.mail.protection.outlook.com: resolve call failed: All attempts to contact name servers or networks failed

Additional program output to the terminal or log subsystem illustrating the issue

No response

kiwixz commented 2 years ago

I think the problem is on domain's side.

https://dnsviz.net/d/uninsubria-it.mail.protection.outlook.com/dnssec/

Tachi107 commented 2 years ago

I think the problem is on domain's side.

Thanks for the reply. Even if they don't deploy DNSSEC, shouldn't the record be resolvable anyway?

pemensik commented 2 years ago

I think there is something rotten in systemd when doing this query. Mine Fedora 37 beta cannot resolve even outlook.com.

# going around resolved
$ delv +vtrace @192.168.122.1 uninsubria-it.mail.protection.outlook.com
;; fetch: uninsubria-it.mail.protection.outlook.com/A
;; validating uninsubria-it.mail.protection.outlook.com/A: starting
;; validating uninsubria-it.mail.protection.outlook.com/A: attempting insecurity proof
;; validating uninsubria-it.mail.protection.outlook.com/A: checking existence of DS at 'com'
;; fetch: com/DS
;; validating com/DS: starting
;; validating com/DS: attempting positive response validation
;; fetch: ./DNSKEY
;; validating ./DNSKEY: starting
;; validating ./DNSKEY: attempting positive response validation
;; validating ./DNSKEY: verify rdataset (keyid=20326): success
;; validating ./DNSKEY: marking as secure (DS)
;; validating com/DS: in fetch_callback_dnskey
;; validating com/DS: keyset with trust secure
;; validating com/DS: resuming validate
;; validating com/DS: verify rdataset (keyid=20826): success
;; validating com/DS: marking as secure, noqname proof not needed
;; validating uninsubria-it.mail.protection.outlook.com/A: in fetch_callback_ds
;; validating uninsubria-it.mail.protection.outlook.com/A: resuming proveunsecure
;; validating uninsubria-it.mail.protection.outlook.com/A: checking existence of DS at 'outlook.com'
;; fetch: outlook.com/DS
;; validating outlook.com/DS: starting
;; validating outlook.com/DS: attempting negative response validation from message
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: starting
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: attempting positive response validation
;; fetch: com/DNSKEY
;; validating com/DNSKEY: starting
;; validating com/DNSKEY: attempting positive response validation
;; validating com/DNSKEY: verify rdataset (keyid=30909): success
;; validating com/DNSKEY: marking as secure (DS)
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: in fetch_callback_dnskey
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: keyset with trust secure
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: resuming validate
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: verify rdataset (keyid=32298): success
;;   validating CK0POJMG874LJREF7EFN8430QVIT8BSM.com/NSEC3: marking as secure, noqname proof not needed
;; validating outlook.com/DS: in validator_callback_nsec
;; validating outlook.com/DS: resuming validate_nx
;;   validating com/SOA: starting
;;   validating com/SOA: attempting positive response validation
;;   validating com/SOA: keyset with trust secure
;;   validating com/SOA: verify rdataset (keyid=32298): success
;;   validating com/SOA: marking as secure, noqname proof not needed
;; validating outlook.com/DS: in validator_callback_nsec
;; validating outlook.com/DS: resuming validate_nx
;;   validating 4n822aim85h3a7lsd1mkie0p53le55kd.com/NSEC3: starting
;;   validating 4n822aim85h3a7lsd1mkie0p53le55kd.com/NSEC3: attempting positive response validation
;;   validating 4n822aim85h3a7lsd1mkie0p53le55kd.com/NSEC3: keyset with trust secure
;;   validating 4n822aim85h3a7lsd1mkie0p53le55kd.com/NSEC3: verify rdataset (keyid=32298): success
;;   validating 4n822aim85h3a7lsd1mkie0p53le55kd.com/NSEC3: marking as secure, noqname proof not needed
;; validating outlook.com/DS: in validator_callback_nsec
;; validating outlook.com/DS: resuming validate_nx
;; validating outlook.com/DS: looking for relevant NSEC3
;; validating outlook.com/DS: looking for relevant NSEC3
;; validating outlook.com/DS: looking for relevant NSEC3
;; validating outlook.com/DS: NSEC3 indicates potential closest encloser: 'com'
;; validating outlook.com/DS: NSEC3 at super-domain com
;; validating outlook.com/DS: looking for relevant NSEC3
;; validating outlook.com/DS: NSEC3 proves name does not exist: 'outlook.com'
;; validating outlook.com/DS: NSEC3 indicates optout
;; validating outlook.com/DS: in checkwildcard: *.com
;; validating outlook.com/DS: looking for relevant NSEC3
;; validating outlook.com/DS: NSEC3 at super-domain com
;; validating outlook.com/DS: looking for relevant NSEC3
;; validating outlook.com/DS: in checkwildcard: *.com
;; validating outlook.com/DS: nonexistence proof(s) found
;; validating uninsubria-it.mail.protection.outlook.com/A: in fetch_callback_ds
;; validating uninsubria-it.mail.protection.outlook.com/A: marking as answer (fetch_callback_ds)
; unsigned answer
uninsubria-it.mail.protection.outlook.com. 30 IN A 104.47.0.36
uninsubria-it.mail.protection.outlook.com. 30 IN A 104.47.2.36

But when I ask 127.0.0.53, it just fails. Even direct query for DS record fails to me:

$ dig +dnssec com ds @127.0.0.53

; <<>> DiG 9.18.5 <<>> +dnssec com ds @127.0.0.53
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 42597
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;com.               IN  DS

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Mon Sep 26 17:30:27 EDT 2022
;; MSG SIZE  rcvd: 32

Link 2 (enp1s0) Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=yes/supported Current DNS Server: 192.168.122.1 DNS Servers: 192.168.122.1

jonnybarnes commented 1 year ago

Just throwing my 2 cents in here, I get random query failures with DNSSEC=true enabled.

svillemot commented 11 months ago

Same issue here with *.mail.protection.outlook.com hosts whose resolution fails with DNSSEC=true enabled (systemd 252.17 from Debian 12)

rpigott commented 3 months ago

The issue is that the authoritative nameservers reject DNSSEC queries with NOTIMP and the recursive resolver replies with SERVFAIL in kind:

$ dig @8.8.8.8 +do +nocrypto DS uninsubria-it.mail.protection.outlook.com

; <<>> DiG 9.18.28 <<>> @8.8.8.8 +do +nocrypto DS uninsubria-it.mail.protection.outlook.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 42956
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
; EDE: 23 (Network Error): ([104.47.68.17] rcode=NOTIMP for uninsubria-it.mail.protection.outlook.com/ds)
; EDE: 23 (Network Error): ([104.47.118.145] rcode=NOTIMP for uninsubria-it.mail.protection.outlook.com/ds)
; EDE: 22 (No Reachable Authority): (At delegation mail.protection.outlook.com for uninsubria-it.mail.protection.outlook.com/ds)
; EDE: 23 (Network Error): ([104.47.118.177] rcode=NOTIMP for uninsubria-it.mail.protection.outlook.com/ds)
;; QUESTION SECTION:
;uninsubria-it.mail.protection.outlook.com. IN DS

;; Query time: 170 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Thu Jul 25 12:17:27 MST 2024
;; MSG SIZE  rcvd: 416

sd-resolved should probably treat this like an empty response for the purpose of validation — this zone doesn't implement DNSSEC but we can still possibly prove it is unsigned if we find an insecure delegation in the parent zone. Currently the SERVFAIL response results in failure.

rpigott commented 3 months ago

If I'm reading it correctly, it seems like this was fixed once before in #3408 and then regressed?

pemensik commented 3 months ago 
$ dig @ns1-proddns.glbdns.o365filtering.com. +noedns +nocrypto DS uninsubria-it.mail.protection.outlook.com

; <<>> DiG 9.18.28 <<>> @ns1-proddns.glbdns.o365filtering.com. +noedns +nocrypto DS uninsubria-it.mail.protection.outlook.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOTIMP, id: 15814
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;uninsubria-it.mail.protection.outlook.com. IN DS

;; Query time: 39 msec
;; SERVER: 104.47.34.49#53(ns1-proddns.glbdns.o365filtering.com.) (UDP)
;; WHEN: Fri Jul 26 11:44:54 CEST 2024
;; MSG SIZE  rcvd: 59

# Above seems invalid answer for authoritative server. It should respond 

$ dig +dnssec +norec +nocrypto DS protection.outlook.com @ns1-gtm.glbdns.o365filtering.com. 

; <<>> DiG 9.18.28 <<>> +dnssec +norec +nocrypto DS protection.outlook.com @ns1-gtm.glbdns.o365filtering.com.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45220
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;protection.outlook.com.        IN  DS

;; AUTHORITY SECTION:
outlook.com.        60  IN  SOA ch0mgt0101dc001.prdmgt01.prod.exchangelabs.com. msnhst.microsoft.com. 2015912661 300 900 2419200 60

;; Query time: 103 msec
;; SERVER: 104.47.38.8#53(ns1-gtm.glbdns.o365filtering.com.) (UDP)
;; WHEN: Fri Jul 26 11:49:35 CEST 2024
;; MSG SIZE  rcvd: 147

$ dig +dnssec +norec +nocrypto DS outlook.com @f.gtld-servers.net. 

; <<>> DiG 9.18.28 <<>> +dnssec +norec +nocrypto DS outlook.com @f.gtld-servers.net.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48841
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;outlook.com.           IN  DS

;; AUTHORITY SECTION:
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q3UDG8CEKKAE7RUKPGCT1DVSSH8LL NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 13 2 86400 20240731002458 20240723231458 59354 com. [omitted]
com.            900 IN  SOA a.gtld-servers.net. nstld.verisign-grs.com. 1721987750 1800 900 604800 86400
com.            900 IN  RRSIG   SOA 13 1 900 20240802095550 20240726084550 59354 com. [omitted]
4N822AIM85H3A7LSD1MKIE0P53LE55KD.com. 86400 IN NSEC3 1 1 0 - 4N82FO6C8UT3J70HTOE73J0TESV6FDPA NS DS RRSIG
4N822AIM85H3A7LSD1MKIE0P53LE55KD.com. 86400 IN RRSIG NSEC3 13 2 86400 20240731014110 20240724003110 59354 com. [omitted]

;; Query time: 39 msec
;; SERVER: 2001:503:d414::30#53(f.gtld-servers.net.) (UDP)
;; WHEN: Fri Jul 26 11:56:05 CEST 2024
;; MSG SIZE  rcvd: 569

I would say that nameservers have broken implementation. It may return FORMERR if they do not understand edns0 header. Therefore no possible DNSSEC. But the issue is caused by wrong algorithm of queries, DNSSEC queries should not have reason to reach those servers, because outlook.com does have cryptographic proof DS record does not exist, and therefore anything under it will be unsigned (insecure). I think that should be fixed since systemd-resolved 256 at least. Authoritative azure servers should be fixed anyway to return just non-existent record for DS and DNSKEY.

pemensik commented 3 months ago

Sadly this authoritative server is known to be deficient, but still not fixed. It has DNS violation entry from year 2020: https://github.com/dns-violations/dns-violations/blob/master/2020/DVE-2020-0003.md

pemensik commented 3 months ago

Cloudflare reports just unreachable authority. If that is received for name, which already has positive response being validated, it probably means some broken implementation is somewhere in the path.

It should be possible to fix this by manually setting NTA for outlook.com domain on versions, which do not yet have correct validation direction from root to child zones. Those where issue #26970 is not yet fixed. I think this were fixed by https://github.com/systemd/systemd/pull/31827 instead.

Any other fixes should be done by authoritative server owners.

$ dig @1.1.1.1 uninsubria-it.mail.protection.outlook.com DS

; <<>> DiG 9.18.28 <<>> @1.1.1.1 uninsubria-it.mail.protection.outlook.com DS
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17290
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 22 (No Reachable Authority): (at delegation mail.protection.outlook.com.)
;; QUESTION SECTION:
;uninsubria-it.mail.protection.outlook.com. IN DS

;; Query time: 267 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Fri Jul 26 12:50:52 CEST 2024
;; MSG SIZE  rcvd: 118

$ dig @1.1.1.1 +dnssec +nocrypto uninsubria-it.mail.protection.outlook.com A

; <<>> DiG 9.18.28 <<>> @1.1.1.1 +dnssec +nocrypto uninsubria-it.mail.protection.outlook.com A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54350
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 1232
;; QUESTION SECTION:
;uninsubria-it.mail.protection.outlook.com. IN A

;; ANSWER SECTION:
uninsubria-it.mail.protection.outlook.com. 10 IN A 52.101.73.11
uninsubria-it.mail.protection.outlook.com. 10 IN A 52.101.68.18
uninsubria-it.mail.protection.outlook.com. 10 IN A 52.101.73.28
uninsubria-it.mail.protection.outlook.com. 10 IN A 52.101.73.21
uninsubria-it.mail.protection.outlook.com. 10 IN A 52.101.68.3
uninsubria-it.mail.protection.outlook.com. 10 IN A 52.101.73.22
uninsubria-it.mail.protection.outlook.com. 10 IN A 52.101.68.5

;; Query time: 296 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Fri Jul 26 12:59:34 CEST 2024
;; MSG SIZE  rcvd: 182
arinc9 commented 2 months ago

I've noticed this happening on systemd 255 on the domains of digital payment services, jccpg.jccsecure.com from Cyprus and a domain or domains from the Masterpass service.

When DNSSEC=allow-downgrade:

$ resolvectl query jccpg.jccsecure.com
jccpg.jccsecure.com: resolve call failed: DNSSEC validation failed: failed-auxiliary

When DNSSEC=no:

$ resolvectl query jccpg.jccsecure.com
jccpg.jccsecure.com: 81.4.175.72               -- link: ens18

-- Information acquired via protocol DNS in 98.2ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: yes
-- Data from: network
rpigott commented 2 months ago

This issue is resolved. Open a new issue if you think there is a problem.

arinc9 commented 2 months ago

Let me post this under #33620 then.

agowa commented 2 months ago

@rpigott the other issue was closed as a duplicate. So are we supposed to reopen this one then???

rpigott commented 2 months ago

You're supposed to open a new issue and provide details if you think there is still a problem.

agowa commented 2 months ago

@rpigott well that's what I did with https://github.com/systemd/systemd/issues/33620 but it got closed as a duplicate of this one...

rpigott commented 2 months ago

It was closed because the issue was resolved.

agowa commented 2 months ago

Good to know, just seeing this as last entry in the ticket sounded like closing as duplicate was the solution...

YHNdnzj added the duplicate label last week
YHNdnzj closed this as completed last week