search

AdguardTeam / AdGuardHome

Network-wide ads & trackers blocking DNS server
https://adguard.com/adguard-home.html
GNU General Public License v3.0
25.42k stars 1.83k forks source link

Despite enabling the 'Ignore this client in query log' checkbox in the Client settings, DNS queries from clients with a textual client_id value still appear in the query log #5812

Closed Dementor316 closed 6 months ago

Dementor316 commented 1 year ago

Prerequisites

Operating system type

Linux, Other (please mention the version in the description)

CPU architecture

AMD64

Installation

GitHub releases or script from README

Setup

On one machine

AdGuard Home version

v0.107.29

Description

Operating system version

Ubuntu Server 20.04.6 LTS

What did you do?

  1. I added a new client under the name "Android" in the "Client settings" section, assigned the client_id "android", and checked the box to ignore all DNS queries from this client in the query log.
  2. In Android, I set the private DNS server to "android.domain.com" in order to use my DNS server via the DNS-over-TLS protocol.
  3. I started using Android, resulting in many requests being sent to my DNS server.

    Expected result

    After active usage of Android, the number of DNS queries on the dashboard of the statistics should increase. However, there shouldn't be any queries from the client with client_id "android" in the queries log.

    Actual result

    DNS Queries from the client with the "android" client_id continue to appear in the log of queries despite the checkbox being selected to ignore queries from this client.

    Additional information

    • It has been discovered that the checkbox for ignoring queries from this client in the statistics works reliably, unlike the checkbox for ignoring queries in the log of queries
    • If an IP address is used as the client_id, or if the second client_id for this client is an IP address rather than a name, then the ignoring of their queries in the log of queries starts working reliably. It seems that the problem arises when a textual value, rather than an IP address, is used as the client_id which indicates an error and incorrect behavior
mxbchr commented 1 year ago

Same Issue here on v0.107.40.

If an IP address is used as the client_id, or if the second client_id for this client is an IP address rather than a name, then the ignoring of their queries in the log of queries starts working reliably. It seems that the problem arises when a textual value, rather than an IP address, is used as the client_id which indicates an error and incorrect behavior

Just for clarification, since I tried my luck here, it has to be the actual IP of the Client that should be ignored, just adding "an IP" wont work. Seems the ignoring is only triggered by IP. Unfortunately adding an IP wont work for external/wan clients, since their IP changes regularly.

ghost commented 7 months ago

Hi @Dementor316, are you still experiencing this?

mxbchr commented 7 months ago

Hey @jslawler-gh Not the one you asked, but I can confirm this is still an issue in v0.107.46

Dementor316 commented 7 months ago

Hello, @jslawler-gh! Yes, the issue still persists in the latest stable version of AdGuard Home.

ghost commented 7 months ago

Hi @mxbchr and @Dementor316, thanks for confirming.

I am unable to reproduce what you're both experiencing on my test server. Could you please provide your yaml, redacting any private information?

mxbchr commented 7 months ago

Hey @jslawler-gh , there you go: I simplified it a bit for readability in terms of filters and custom rules. For context: Client1 and Client2 access AdGuard via DoT, using their client id ('abcdefg' and 'hijklmno') as a subdomain i.e. abcdefg.dns.domain.org. Using this method their requests get identified correctly in statistics and in query log. But the "ignore in query log" checkbox has no effect for them. When Client1 accesses from the home network with the known and specified ip (10.0.2.10), the matching works correctly and the requests do not show up in query log.

http:
  pprof:
    port: 6060
    enabled: false
  address: 127.0.0.1:45158
  session_ttl: 720h
users: []
auth_attempts: 5
block_auth_min: 15
http_proxy: ""
language: en
theme: auto
dns:
  bind_hosts:
      - 10.0.2.8
      - 172.30.32.1
      - 127.0.0.1
      - ::1
    port: 53
    anonymize_client_ip: false
    ratelimit: 20
    ratelimit_subnet_len_ipv4: 24
    ratelimit_subnet_len_ipv6: 56
    ratelimit_whitelist: []
    refuse_any: true
    upstream_dns:
      - https://cloudflare-dns.com/dns-query
      - tls://1.1.1.1
    upstream_dns_file: ""
    bootstrap_dns:
      - 1.1.1.1:53
    fallback_dns:
      - 1.1.1.1:53
    upstream_mode: parallel
    fastest_timeout: 1s
    allowed_clients:
      - abcdefg
      - hijklmno
      - 10.0.2.0/24
    disallowed_clients: []
    blocked_hosts:
      - version.bind
      - id.server
      - hostname.bind
    trusted_proxies:
      - 127.0.0.0/8
      - ::1/128
      - 172.30.33.0/24
      - 10.0.2.8/32
    cache_size: 4194304
    cache_ttl_min: 0
    cache_ttl_max: 0
    cache_optimistic: true
    bogus_nxdomain: []
    aaaa_disabled: false
    enable_dnssec: true
    edns_client_subnet:
      custom_ip: ""
      enabled: false
      use_custom: false
    max_goroutines: 300
    handle_ddr: true
    ipset: []
    ipset_file: ""
    bootstrap_prefer_ipv6: false
    upstream_timeout: 10s
    private_networks: []
    use_private_ptr_resolvers: true
    local_ptr_upstreams:
      - 10.0.2.1:53
    use_dns64: false
    dns64_prefixes: []
    serve_http3: false
    use_http3_upstreams: false
    serve_plain_dns: true
    hostsfile_enabled: true
  tls:
    enabled: true
    server_name: dns.domain.org
    force_https: false
    port_https: 9876
    port_dns_over_tls: 853
    port_dns_over_quic: 784
    port_dnscrypt: 0
    dnscrypt_config_file: ""
    allow_unencrypted_doh: false
    certificate_chain: ""
    private_key: ""
    certificate_path: /ssl/fullchain.pem
    private_key_path: /ssl/privkey.pem
    strict_sni_check: false
  querylog:
    dir_path: ""
    ignored: []
    interval: 24h
    size_memory: 1000
    enabled: true
    file_enabled: true
  statistics:
    dir_path: ""
    ignored: []
    interval: 24h
    enabled: true
  filters:
    - enabled: true
      url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_34.txt
      name: HaGeZi Personal Black & White
      id: 1685461722
  whitelist_filters: []
  user_rules:
    - '@@||firebasedynamiclinks.googleapis.com^$important'
  dhcp:
    enabled: false
    interface_name: ""
    local_domain_name: lan
    dhcpv4:
      gateway_ip: ""
      subnet_mask: ""
      range_start: ""
      range_end: ""
      lease_duration: 0
      icmp_timeout_msec: 1000
      options: []
    dhcpv6:
      range_start: ""
      lease_duration: 0
      ra_slaac_only: false
      ra_allow_slaac: false
  filtering:
    blocking_ipv4: ""
    blocking_ipv6: ""
    blocked_services:
      schedule:
        time_zone: Europe/Berlin
        sun:
          start: 20h
          end: 21h30m
        mon:
          start: 20h
          end: 21h30m
        tue:
          start: 20h
          end: 21h30m
        wed:
          start: 20h
          end: 21h30m
        thu:
          start: 20h
          end: 21h30m
        fri:
          start: 20h
          end: 21h30m
        sat:
          start: 20h
          end: 21h30m
      ids: []
    protection_disabled_until: null
    safe_search:
      enabled: false
      bing: true
      duckduckgo: true
      google: true
      pixabay: true
      yandex: true
      youtube: true
    blocking_mode: default
    parental_block_host: family-block.dns.adguard.com
    safebrowsing_block_host: standard-block.dns.adguard.com
    rewrites: []
    safebrowsing_cache_size: 1048576
    safesearch_cache_size: 1048576
    parental_cache_size: 1048576
    cache_time: 30
    filters_update_interval: 24
    blocked_response_ttl: 10
    filtering_enabled: true
    parental_enabled: false
    safebrowsing_enabled: false
    protection_enabled: true
  clients:
    runtime_sources:
      whois: true
      arp: true
      rdns: true
      dhcp: true
      hosts: true
    persistent:
      - safe_search:
          enabled: false
          bing: true
          duckduckgo: true
          google: true
          pixabay: true
          yandex: true
          youtube: true
        blocked_services:
          schedule:
            time_zone: Europe/Berlin
          ids: []
        name: Client2
        ids:
          - abcdefg
        tags:
          - device_phone
        upstreams: []
        uid: 018d8417-07e1-7ffe-91a4-c6a51d4d3cbf
        upstreams_cache_size: 0
        upstreams_cache_enabled: false
        use_global_settings: true
        filtering_enabled: true
        parental_enabled: false
        safebrowsing_enabled: false
        use_global_blocked_services: false
        ignore_querylog: true
        ignore_statistics: false
      - safe_search:
          enabled: false
          bing: true
          duckduckgo: true
          google: true
          pixabay: true
          yandex: true
          youtube: true
        blocked_services:
          schedule:
            time_zone: Europe/Berlin
          ids: []
        name: Client1
        ids:
          - hijklmno
          - 10.0.2.10
        tags:
          - os_android
        upstreams: []
        uid: 018d8417-07e1-7b37-9cb3-2d484e6c2a91
        upstreams_cache_size: 0
        upstreams_cache_enabled: false
        use_global_settings: true
        filtering_enabled: false
        parental_enabled: false
        safebrowsing_enabled: false
        use_global_blocked_services: true
        ignore_querylog: true
        ignore_statistics: false
      - safe_search:
          enabled: false
          bing: true
          duckduckgo: true
          google: true
          pixabay: true
          yandex: true
          youtube: true
        blocked_services:
          schedule:
            time_zone: Europe/Berlin
          ids: []
        name: VLAN 111 Clients
        ids:
          - 10.0.2.0/24
        tags:
          - os_other
        upstreams: []
        uid: 018d8417-07e1-70e0-ae89-3be7309c2cca
        upstreams_cache_size: 0
        upstreams_cache_enabled: false
        use_global_settings: true
        filtering_enabled: false
        parental_enabled: false
        safebrowsing_enabled: false
        use_global_blocked_services: true
        ignore_querylog: false
        ignore_statistics: false

  log:
    file: ""
    max_backups: 0
    max_size: 100
    max_age: 3
    compress: false
    local_time: false
    verbose: false
  os:
    group: ""
    user: ""
    rlimit_nofile: 0
  schema_version: 28
mxbchr commented 7 months ago

@jslawler-gh If the verbose logs are of any help to you, here they are. Client1 requesting 'www.netzwelt.de' via external access.

0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
;; QUESTION SECTION:
;www.netzwelt.de.   IN   A
2024/04/07 16:49:36.651226 301#3312 [debug] dnsforward: got client server name "hijklmno.dns.domain.org" from tls conn
2024/04/07 16:49:36.651278 301#3312 [debug] dnsforward: started processing recursion
2024/04/07 16:49:36.651327 301#3312 [debug] dnsforward: finished processing recursion
2024/04/07 16:49:36.651345 301#3312 [debug] dnsforward: started processing initial
2024/04/07 16:49:36.651523 301#3312 [debug] applying filters: looking for client with ip 176.6.XXX.XXX and clientid "hijklmno"
2024/04/07 16:49:36.651788 301#3312 [debug] applying filters: using settings for client "Client1" (176.6.XXX.XXX; "hijklmno")
2024/04/07 16:49:36.651689 301#14 [debug] clients: processing 176.6.XXX.XXX with rdns
2024/04/07 16:49:36.652005 301#14 [debug] clients: finished processing 176.6.XXX.XXX with rdns in 480.909µs
2024/04/07 16:49:36.652018 301#14 [debug] clients: processing 176.6.XXX.XXX with whois
2024/04/07 16:49:36.652033 301#14 [debug] clients: finished processing 176.6.XXX.XXX with whois in 14.367µs
2024/04/07 16:49:36.652184 301#3312 [debug] dnsforward: finished processing initial
2024/04/07 16:49:36.652240 301#3312 [debug] dnsforward: started processing ddr
2024/04/07 16:49:36.652259 301#3312 [debug] dnsforward: finished processing ddr
2024/04/07 16:49:36.652274 301#3312 [debug] dnsforward: started processing local detection
2024/04/07 16:49:36.652301 301#3312 [debug] dnsforward: finished processing local detection
2024/04/07 16:49:36.652316 301#3312 [debug] dnsforward: started processing dhcp hosts
2024/04/07 16:49:36.652333 301#3312 [debug] dnsforward: finished processing dhcp hosts
2024/04/07 16:49:36.652348 301#3312 [debug] dnsforward: started processing local restriction
2024/04/07 16:49:36.652364 301#3312 [debug] dnsforward: finished processing local restriction
2024/04/07 16:49:36.652379 301#3312 [debug] dnsforward: started processing dhcp addrs
2024/04/07 16:49:36.652393 301#3312 [debug] dnsforward: finished processing dhcp addrs
2024/04/07 16:49:36.652408 301#3312 [debug] dnsforward: started processing filtering before req
2024/04/07 16:49:36.652556 301#3312 [debug] dnsforward: finished processing filtering before req
2024/04/07 16:49:36.652573 301#3312 [debug] dnsforward: started processing local ptr
2024/04/07 16:49:36.652588 301#3312 [debug] dnsforward: finished processing local ptr
2024/04/07 16:49:36.652602 301#3312 [debug] dnsforward: started processing upstream
2024/04/07 16:49:36.652658 301#3485 [debug] parallel lookup: lookup for dns.quad9.net succeeded in 1.506µs: [9.9.9.9 149.112.112.112 2620:fe::9 2620:fe::fe]
2024/04/07 16:49:36.652687 301#3483 [debug] dot upstream: using existing conn 9.9.9.9:853
2024/04/07 16:49:36.652707 301#3483 [debug] dnsproxy: sending request to tls://dns.quad9.net:853 over tcp: A "www.netzwelt.de."
2024/04/07 16:49:36.652802 301#3480 [debug] dnsproxy: sending request to https://cloudflare-dns.com:443/dns-query over tcp: A "www.netzwelt.de."
2024/04/07 16:49:36.652948 301#3481 [debug] dot upstream: using existing conn 1.1.1.1:853
2024/04/07 16:49:36.652968 301#3481 [debug] dnsproxy: sending request to tls://1.1.1.1:853 over tcp: A "www.netzwelt.de."
2024/04/07 16:49:36.653008 301#3482 [debug] dnsproxy: sending request to https://dns.quad9.net:443/dns-query over tcp: A "www.netzwelt.de."
2024/04/07 16:49:36.653156 301#3484 [debug] parallel lookup: lookup for dns.quad9.net succeeded in 1.096µs: [9.9.9.9 149.112.112.112 2620:fe::9 2620:fe::fe]
2024/04/07 16:49:36.668432 301#3483 [debug] dnsproxy: tls://dns.quad9.net:853: response received over tcp: "ok"
2024/04/07 16:49:36.668556 301#3483 [debug] dnsproxy: upstream tls://dns.quad9.net:853 exchanged ;www.netzwelt.de.  IN   A successfully in 15.903138ms
2024/04/07 16:49:36.668644 301#3312 [debug] dnsproxy: replying from upstream: rtt is 16.001551ms
2024/04/07 16:49:36.668855 301#3312 [debug] dnsforward: finished processing upstream
2024/04/07 16:49:36.669010 301#3312 [debug] dnsforward: started processing filtering after resp
2024/04/07 16:49:36.668674 301#3482 [debug] dnsproxy: https://dns.quad9.net:443/dns-query: response received over tcp: "ok"
2024/04/07 16:49:36.669137 301#3482 [debug] dnsproxy: upstream https://dns.quad9.net:443/dns-query exchanged ;www.netzwelt.de.  IN   A successfully in 16.111618ms
2024/04/07 16:49:36.669350 301#3312 [debug] dnsforward: checked CNAME http2.netzwelt.map.fastly.net for www.netzwelt.de.
2024/04/07 16:49:36.669888 301#3312 [debug] dnsforward: checked A 151.101.1.63 for http2.netzwelt.map.fastly.net.
2024/04/07 16:49:36.669985 301#3312 [debug] dnsforward: checked A 151.101.65.63 for http2.netzwelt.map.fastly.net.
2024/04/07 16:49:36.670020 301#3312 [debug] dnsforward: checked A 151.101.193.63 for http2.netzwelt.map.fastly.net.
2024/04/07 16:49:36.670047 301#3312 [debug] dnsforward: checked A 151.101.129.63 for http2.netzwelt.map.fastly.net.
2024/04/07 16:49:36.670063 301#3312 [debug] dnsforward: finished processing filtering after resp
2024/04/07 16:49:36.670078 301#3312 [debug] dnsforward: ipset: started processing
2024/04/07 16:49:36.670096 301#3312 [debug] dnsforward: ipset: finished processing
2024/04/07 16:49:36.670111 301#3312 [debug] dnsforward: started processing querylog and stats
2024/04/07 16:49:36.670128 301#3312 [debug] dnsforward: client ip for stats and querylog: 176.6.XXX.XXX
2024/04/07 16:49:36.670173 301#3312 [debug] dnsforward: client 176.6.XXX.XXX (id "176.6.XXX.XXX") is not in access allowlist
2024/04/07 16:49:36.670243 301#3312 [debug] dnsforward: finished processing querylog and stats
2024/04/07 16:49:36.670301 301#3312 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).logDNSMessage(): OUT: ;; opcode: QUERY, status: NOERROR, id: 1
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 8192
;; QUESTION SECTION:
;www.netzwelt.de.   IN   A
;; ANSWER SECTION:
www.netzwelt.de.    6871    IN  CNAME   http2.netzwelt.map.fastly.net.
http2.netzwelt.map.fastly.net.  30  IN  A   151.101.1.63
http2.netzwelt.map.fastly.net.  30  IN  A   151.101.65.63
http2.netzwelt.map.fastly.net.  30  IN  A   151.101.193.63
http2.netzwelt.map.fastly.net.  30  IN  A   151.101.129.63
2024/04/07 16:49:36.693803 301#3481 [debug] dnsproxy: tls://1.1.1.1:853: response received over tcp: "ok"
2024/04/07 16:49:36.693835 301#3481 [debug] dnsproxy: upstream tls://1.1.1.1:853 exchanged ;www.netzwelt.de.    IN   A successfully in 40.906448ms
2024/04/07 16:49:36.710711 301#3312 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).logDNSMessage(): IN: ;; opcode: QUERY, status: NOERROR, id: 0
;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 8192
; PADDING: 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
Dementor316 commented 7 months ago

@jslawler-gh, here is my yaml file. I have edited it and removed all confidential data, including unnecessary filters. The issue continues to occur in the latest version. When the "Ignore DNS requests from this client in the query log" checkbox is checked and only a text value dnspr is added as an identifier, requests from this client continue to appear in the log. But if an IP is specified as the second value of the identifier or only an IP without a text client_id, then there are no issues and all requests from this client are absent in the log. AdGuardHome.yaml.txt

schzhn commented 6 months ago

Fixed in the edge release. Feel free to reopen if the problem persists.