search

getdnsapi / stubby

Stubby is the name given to a mode of using getdns which enables it to act as a local DNS Privacy stub resolver (using DNS-over-TLS).
https://dnsprivacy.org/dns_privacy_daemon_-_stubby/
BSD 3-Clause "New" or "Revised" License
1.2k stars 99 forks source link

Cloudflare DoT w/multiple requests = DNSSEC fail until connection closed #331

Open uzlonewolf opened 1 year ago

uzlonewolf commented 1 year ago

So, I don't know if this is a Cloudflare, Stubby, or GetDNS bug, but when Stubby is configured to use only Cloudflare and multiple requests for the same name are sent, both of those requests plus every request sent after that point will fail. Things only start working again once the connection has been idle long enough to be closed.

Using this configuration:

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
tls_query_padding_blocksize: 128
edns_client_subnet_private : 1
round_robin_upstreams: 0
idle_timeout: 10000
tls_min_version: GETDNS_TLS1_2
listen_addresses: 
  - 127.0.0.1@9953
  - 0::1@9953
dnssec: GETDNS_EXTENSION_TRUE
appdata_dir: "/var/cache/stubby"
dnssec_trust_anchors: "/var/lib/unbound/root.key"
upstream_recursive_servers:
## Cloudflare 1.1.1.1 and 1.0.0.1
  - address_data: 1.1.1.1
    tls_auth_name: "cloudflare-dns.com"
  - address_data: 1.0.0.1
    tls_auth_name: "cloudflare-dns.com"

I can reliably get it to fall over with:

dig @0 -p 9953 google.com; dig @0 -p 9953 google.com & dig @0 -p 9953 google.com

Changing round_robin_upstreams: 0 to round_robin_upstreams: 1 makes it take slightly longer, but it still happens if you run the above 2-3 times.

Recompiling GetDNS with full debugging enabled shows that once the bug is triggered, every subsequent request over that TLS/TCP connection will result in a DNSSEC failure. Once that happens, Stubby/GetDNS just sits there and times out instead of retrying with the next server.

I have not tried DoH, but this does not happen with Google's DoT servers.

hanvinke commented 1 year ago

Strange behaviour indeed. How exactly does it fall over? Just AD-bit not showing in the response? Does the same thing also happen when you use 'dnssectest.sidn.nl' instead of 'google.com' ?

I am just asking because 'google.com' does not give me the AD-bit in return (I have a similar configuration). But 'dnssectest.sidn.nl' does and answers more quickly.

uzlonewolf commented 1 year ago

I'm not sure exactly, getdns just says DNSSEC validation failed and drops the response. google.com is just an example, it does it for every name. Here is a portion of the rather verbose debug logs when using dnssectest.sidn.nl:

[11:20:05.840805] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:20:05.840829]   with trust anchor . IN DNSKEY, rrs: 1, rrsigs:
[11:20:05.840861] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.840902] Commencing NX proof for: sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.840945] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.841026] sizes: 2 rrs, 569 bytes for validation buffer
[11:20:05.841056] written to valbuf: 569 bytes
[11:20:05.841476] key . 86400   IN  DNSKEY  257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
[11:20:05.841522] signed . IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.841547] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.841596] sizes: 1 rrs, 69 bytes for validation buffer
[11:20:05.841626] written to valbuf: 69 bytes
[11:20:05.841953] key . 172474  IN  DNSKEY  256 3 8 AwEAAeB54o2xvW6vY4qQZ0krDsEZCe6MsRWCqsXd4+cNJZMePnlV/xwDrIbbeH1SJzv742rOHzgAKM1/3SQHHSkoEIPx8XQdHAZBxfhaXl3e8c5WrE3aGXS5AeTWAkt85ccqWgKyitxjFmJEOol0BqS2xueltaDwgWcC10nPUY+y5l/kTOYyptYQS4gg1uJNXIob/R1XIEJ10ZCurkYqZxgqyHc7tZv09N23o9rnGdjnYiArH7FjlXD8Rvjde8YWkmfdbCEWnchrnxDK8KV2/ZvBpG/WYnRKXYPUceGCw59OJdJ5M7utkm547RB3eEd8CVVhbXopZlsKq3GCrBwaIVe9ci0= ;{id = 18733 (zsk), size = 2048b}
[11:20:05.841994] signed nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.842027] ds_authenticates_keys DS: nl. IN DS, rrs: 1, rrsigs: 1
[11:20:05.842055] ds_authenticates_keys DNSKEY: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842110] sizes: 2 rrs, 450 bytes for validation buffer
[11:20:05.842131] written to valbuf: 450 bytes
[11:20:05.842453] key nl.   1442    IN  DNSKEY  257 3 8 AwEAAcb+4kIsKoZM+3ZZpU9kzxrzw30e3b+L0KZeX+aAS3eM+Q+q27Jw0NZ3dqsPSif61GjRW6apjDZ9Ciab3oyEu7IpihVrw94DTjWZTVViZAijAIHwKUzY0YjkT3RvN+xgpw4uZs1SnqCZxYko+15esteKXW/nJpde0d9OeFFBaS2WTCycK+A6gd9DsOw91Y7Z2vrR/2g9N9dMIVq9neB1/KXXm4MttLqJyxRWZNAFTyLGQKzPpQDp9s3qowV2+pcHOh6lUTEeOWiAtotJ/5WyO91viZ5tBfClsyGpggBTaeUQ7T5adhAtX6nRkhePyAtQgCCf63ZpHyoyxvbkDM7yuA0= ;{id = 34112 (ksk), size = 2048b}
[11:20:05.842496] signed nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842531] keyset authenticated: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842547] valid_dsses: 1, supported_dsses: 1
[11:20:05.842570] Commencing NX proof for: sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.842609] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:20:05.842650] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:20:05.842677]   with keys nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:20:05.842727] BOGUS: dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:20:05.842748]   with trust anchor: . IN DNSKEY, rrs: 1, rrsigs:

Edit: and here's a successful single request for that domain:

[11:28:54.249983] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:28:54.250007]   with trust anchor . IN DNSKEY, rrs: 1, rrsigs:
[11:28:54.250047] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:28:54.250099] Commencing NX proof for: sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.250157] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.250212] sizes: 2 rrs, 569 bytes for validation buffer
[11:28:54.250245] written to valbuf: 569 bytes
[11:28:54.250650] key . 86400   IN  DNSKEY  257 3 8 AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTOiW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLYA4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU= ;{id = 20326 (ksk), size = 2048b}
[11:28:54.250698] signed . IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.250722] Commencing NX proof for: nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.250770] sizes: 1 rrs, 69 bytes for validation buffer
[11:28:54.250788] written to valbuf: 69 bytes
[11:28:54.251109] key . 165104  IN  DNSKEY  256 3 8 AwEAAeB54o2xvW6vY4qQZ0krDsEZCe6MsRWCqsXd4+cNJZMePnlV/xwDrIbbeH1SJzv742rOHzgAKM1/3SQHHSkoEIPx8XQdHAZBxfhaXl3e8c5WrE3aGXS5AeTWAkt85ccqWgKyitxjFmJEOol0BqS2xueltaDwgWcC10nPUY+y5l/kTOYyptYQS4gg1uJNXIob/R1XIEJ10ZCurkYqZxgqyHc7tZv09N23o9rnGdjnYiArH7FjlXD8Rvjde8YWkmfdbCEWnchrnxDK8KV2/ZvBpG/WYnRKXYPUceGCw59OJdJ5M7utkm547RB3eEd8CVVhbXopZlsKq3GCrBwaIVe9ci0= ;{id = 18733 (zsk), size = 2048b}
[11:28:54.251155] signed nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.251194] ds_authenticates_keys DS: nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.251221] ds_authenticates_keys DNSKEY: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.251275] sizes: 2 rrs, 450 bytes for validation buffer
[11:28:54.251301] written to valbuf: 450 bytes
[11:28:54.251623] key nl.   1237    IN  DNSKEY  257 3 8 AwEAAcb+4kIsKoZM+3ZZpU9kzxrzw30e3b+L0KZeX+aAS3eM+Q+q27Jw0NZ3dqsPSif61GjRW6apjDZ9Ciab3oyEu7IpihVrw94DTjWZTVViZAijAIHwKUzY0YjkT3RvN+xgpw4uZs1SnqCZxYko+15esteKXW/nJpde0d9OeFFBaS2WTCycK+A6gd9DsOw91Y7Z2vrR/2g9N9dMIVq9neB1/KXXm4MttLqJyxRWZNAFTyLGQKzPpQDp9s3qowV2+pcHOh6lUTEeOWiAtotJ/5WyO91viZ5tBfClsyGpggBTaeUQ7T5adhAtX6nRkhePyAtQgCCf63ZpHyoyxvbkDM7yuA0= ;{id = 34112 (ksk), size = 2048b}
[11:28:54.251665] signed nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.251699] keyset authenticated: nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.251723] valid_dsses: 1, supported_dsses: 1
[11:28:54.251756] Commencing NX proof for: sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.251807] sizes: 1 rrs, 77 bytes for validation buffer
[11:28:54.251826] written to valbuf: 77 bytes
[11:28:54.251988] key nl.   1237    IN  DNSKEY  256 3 8 AwEAAdBMHLVXYPJGb+o8QXIj9v1CtqTu+EybrWyG/ShJp8/tc37GBt5NR7DzugoE9nfr2M0eKjnc4E/qO6GauZouq6tLQuC8k4RD49u0/ecYZV/+EZ6Qv3I/iNOyatR9qhluj1V16hLRpMsHApVFE8rWoSlcxqM9Sbe4BfAHBmx582cp ;{id = 52584 (zsk), size = 1024b}
[11:28:54.252026] signed sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.252059] ds_authenticates_keys DS: sidn.nl. IN DS, rrs: 1, rrsigs: 1
[11:28:54.252087] ds_authenticates_keys DNSKEY: sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.252129] sizes: 2 rrs, 201 bytes for validation buffer
[11:28:54.252156] written to valbuf: 201 bytes
[11:28:54.252967] key sidn.nl.  2946    IN  DNSKEY  257 3 13 jh1pVS3yzgJk4lM9fEtwS5V+nbISmn8AmGZAdY0T1o4351IQEBd/v518tl00YmQL5Mq26YLE8RGKIOCXAYzWiQ== ;{id = 62949 (ksk), size = 256b}
[11:28:54.253044] signed sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.253079] keyset authenticated: sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.253098] valid_dsses: 1, supported_dsses: 1
[11:28:54.253128] Commencing NX proof for: dnssectest.sidn.nl. IN DS, rrs:, rrsigs:
[11:28:54.253172] Validating dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:28:54.253202]   with keys sidn.nl. IN DNSKEY, rrs: 1 2, rrsigs: 1
[11:28:54.253240] sizes: 1 rrs, 61 bytes for validation buffer
[11:28:54.253266] written to valbuf: 61 bytes
[11:28:54.254002] key sidn.nl.  2946    IN  DNSKEY  256 3 13 Mel453Ed98AIkFkU4/SW8hvlCB6SCqP8zNjhsVJWzXIj7TAxC+8CWY2FbgGK5pVe9hc4jrL+BET4UWJ+gJ4/WA== ;{id = 39816 (zsk), size = 256b}
[11:28:54.254047] signed dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
[11:28:54.254075] Key 105352 proved
[11:28:54.254105]   SECURE: dnssectest.sidn.nl. IN A, rrs: 1, rrsigs: 1
hanvinke commented 1 year ago

I have also tried this on my openWRT device. It seems that there is always an answer either very slow or with a time-out. But I do have the AD-flag in the response.

dnssectest.txt

hanvinke commented 1 year ago

@uzlonewolf could you please try 'Zero configuration DNSSEC' with Stubby by disabling 'dnssec_trust_anchors', the 3 files (root-anchors.p7s ,root-anchors.xml and root.key) should appear in your appdata_dir.

Recently I had some troubles with 'dig' after an update and had to force-remove the depending ssl libraries and reinstall them. ldd /usr/bin/dig gives me currently:

    /lib/ld-musl-aarch64.so.1 (0xffffb377a000)
    libisc-9.18.7.so => /usr/lib/libisc-9.18.7.so (0xffffb31e0000)
    libdns-9.18.7.so => /usr/lib/libdns-9.18.7.so (0xffffb2ffe000)
    libisccfg-9.18.7.so => /usr/lib/libisccfg-9.18.7.so (0xffffb2fbb000)
    libirs-9.18.7.so => /usr/lib/libirs-9.18.7.so (0xffffb2fa7000)
    libbind9-9.18.7.so => /usr/lib/libbind9-9.18.7.so (0xffffb2f82000)
    libc.so => /lib/ld-musl-aarch64.so.1 (0xffffb377a000)
    libuv.so.1 => /usr/lib/libuv.so.1 (0xffffb2f46000)
    libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0xffffb2eb3000)
    libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0xffffb2c49000)
    libz.so.1 => /usr/lib/libz.so.1 (0xffffb2c22000)
    libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xffffb2be6000)
    libns-9.18.7.so => /usr/lib/libns-9.18.7.so (0xffffb2b95000)
    libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xffffb2b70000)
wtoorop commented 1 year ago

I have tried this, but I have had mostly slow responses like @hanvinke had, But I did see a servfail at some point. With Quad8 I have slower responses, but not as slow as with Quad1. Quad9 never gives me a slow response in this configuration.

I'm not 100% sure, but I suspect there some kind of stricter rate-limiting with CF hampering the lookup of some of the records to do validation successfully. If that would be the case, some kind of caching mechanism (at least for the DNSSEC answers) would be the definite solution.

hanvinke commented 1 year ago

With DiG 9.18.10 there seem to be improvements in speed and handling. I get a much quicker response from dnssectest.sidn.nl and no time-outs so far.

dnssectest1.txt

// edit: still have slow responses or time-outs occasionally. A query to a non existing website gives me no SERVFAIL btw.

uzlonewolf commented 1 year ago

@hanvinke

With DiG

I'm just using DiG to demonstrate how to reproduce the problem. In normal everyday use it is my web browser which is sending the multiple requests thereby causing stubby/Cloudflare to fall over.

@wtoorop I'm actually using unbound to cache responses and redirect certain names, however if there is no cached entry then it sends everything (including multiple requests for the same name) straight through.