DNS lookups totally break when "browsing security web service" is down

Daniel15 commented 3 months ago

Prerequisites

[X] I have checked the Wiki and Discussions and found no answer
[X] I have searched other issues and found no duplicates
[X] I want to report a bug and not ask a question or ask for help
[X] I have set up AdGuard Home correctly and configured clients to use it. (Use the Discussions for help with installing and configuring clients.)

Platform (OS and CPU architecture)

Linux, AMD64 (aka x86_64)

Installation

Docker

Setup

On one machine

AdGuard Home version

v0.107.52

Action

I noticed that neither of my AdGuard Home servers were responsive. Looking at the logs, there were a large number of failed queries to family.adguard-dns.com:

2024/08/07 18:06:37.709298 [error] dnsproxy: https://family.adguard-dns.com:443/dns-query: response received over tcp: "requesting https://family.adguard-dns.com:443/dns-query: Get \"https://family.adguard-dns.com:443/dns-query?dns=AAABAAABAAAAAAAABGQyZjQENTE5ZAQ3MWI0AnNiA2RucwdhZGd1YXJkA2NvbQAAEAAB\": context deadline exceeded"
2024/08/07 18:06:37.709324 [error] dnsproxy: https://family.adguard-dns.com:443/dns-query: response received over tcp: "requesting https://family.adguard-dns.com:443/dns-query: Get \"https://family.adguard-dns.com:443/dns-query?dns=AAABAAABAAAAAAAABGNmNWYENGRjMwJzYgNkbnMHYWRndWFyZANjb20AABAAAQ\": context deadline exceeded"
2024/08/07 18:06:37.709333 [error] dnsproxy: https://family.adguard-dns.com:443/dns-query: response received over tcp: "requesting https://family.adguard-dns.com:443/dns-query: Get \"https://family.adguard-dns.com:443/dns-query?dns=AAABAAABAAAAAAAABDhmMTYEZDRjOQJzYgNkbnMHYWRndWFyZANjb20AABAAAQ\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
2024/08/07 18:06:37.709353 [error] dnsproxy: https://family.adguard-dns.com:443/dns-query: response received over tcp: "requesting https://family.adguard-dns.com:443/dns-query: Get \"https://family.adguard-dns.com:443/dns-query?dns=AAABAAABAAAAAAAABGU1ZWYEM2U1NwJzYgNkbnMHYWRndWFyZANjb20AABAAAQ\": dial tcp 94.140.14.15:443: i/o timeout\ndial tcp 94.140.14.16:443: i/o timeout\ndial tcp [2a10:50c0::bad1:ff]:443: i/o timeout\ndial tcp [2a10:50c0::bad2:ff]:443: i/o timeout"
2024/08/07 18:06:37.709359 [error] dnsproxy: https://family.adguard-dns.com:443/dns-query: response received over tcp: "requesting https://family.adguard-dns.com:443/dns-query: Get \"https://family.adguard-dns.com:443/dns-query?dns=AAABAAABAAAAAAAABDQxZjUCc2IDZG5zB2FkZ3VhcmQDY29tAAAQAAE\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
2024/08/07 18:06:37.709366 [error] dnsproxy: https://family.adguard-dns.com:443/dns-query: response received over tcp: "requesting https://family.adguard-dns.com:443/dns-query: Get \"https://family.adguard-dns.com:443/dns-query?dns=AAABAAABAAAAAAAABGJkNzYENDFlZARiY2MxAnNiA2RucwdhZGd1YXJkA2NvbQAAEAAB\": context deadline exceeded"
2024/08/07 18:06:37.709367 [error] dnsproxy: https://family.adguard-dns.com:443/dns-query: response received over tcp: "requesting https://family.adguard-dns.com:443/dns-query: Get \"https://family.adguard-dns.com:443/dns-query?dns=AAABAAABAAAAAAAABDVhOWUCc2IDZG5zB2FkZ3VhcmQDY29tAAAQAAE\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

I determined that these failing requests were coming from the "Use AdGuard browsing security web service" option. Once I disabled this option, DNS requests started working properly again.

The current failure mode is not ideal because:

Even if my upstream DNS servers are OK, the DNS requests are timing out because of this failing service.
The failing requests are not visible in the query log, so it was hard for me to even work out what was failing
The latency of these lookups is not reported in the query log - it turns out this was also the cause of some DNS lookup slowness I've been trying to investigate.

I didn't enable this option myself - it was enabled by default. Users of AdGuard Home may not expect that it's hitting a server other than the servers that are explicitly configured as DNS upstreams...

Proposed changes:

If calling this service fails, it should fail open (i.e. return the DNS response without checking it against the security web service) rather than failing closed.
Failures to call this service should be shown in the query log
Latency from calling this service should be shown in the query log

Expected result

Working DNS requests

Actual result

Failing DNS requests when family.adguard-dns.com is down and "browsing security web service" option is enabled

Additional information and/or screenshots

No response

rursache commented 3 months ago

This happens from time to time even with the "browsing security web service" disabled in settings Lots of users reporting this as well: https://github.com/AdguardTeam/AdGuardHome/issues/6817

ainar-g commented 3 months ago

Duplicate of #6817.

@rursache, make sure it's disabled is both the global settings and in the custom settings of the client experiencing the issue. If you're sure that it's disabled in these instances, please send the verbose logs as well as the anonymized configuration of your AGH to devteam@adguard.com with a subject line mentioning AdGuard Home.

Daniel15 commented 3 months ago

Duplicate of https://github.com/AdguardTeam/AdGuardHome/issues/6817.

That issue is closed, and doesn't really have much actionable in it. Should I split these into separate issues:

If calling this service fails, it should fail open (i.e. return the DNS response without checking it against the security web service) rather than failing closed.
Failures to call this service should be shown in the query log
Latency from calling this service should be shown in the query log

I saw in #6817 that you disagree with the first one. That's fine, but IMO it should be an option in the UI (e.g. "Fail lookup if Browsing Security and Parental Control system is unavailable"). Enable it by default if you like. The second two points are still valid. There's no metrics around this service and it can have some large latency spikes (while testing it, I saw latency spikes of 2-3 seconds).

AdguardTeam / AdGuardHome