AdguardTeam / AdGuardHome

Network-wide ads & trackers blocking DNS server
https://adguard.com/adguard-home/overview.html
GNU General Public License v3.0
25.78k stars 1.85k forks source link

Adguard Home has high cpu usage, high memory usage, constantly spikes the cpu and remains unresponsive during filterlist/allowlist or blocklist changes - insanely more so than pihole. #2041

Closed waffshappen closed 3 years ago

waffshappen commented 4 years ago

Prerequisites

Issue Details

Expected Behavior

When comparing adguard home to pihole i expected the following:

For the same amount of Filterlists it should remain as responsive, even if there are a lot Remain fast when i add or delete entries to the blocklists/allowlists Startup fast-ish To not constantly use cpu time

Actual Behavior

When booting the pi or restarting adguard home it takes ~2 Minutes to initialize and build the filterlists. The same goes for when i add or remove blocklist entires and adguard entirely stops responding for up to several minutes. This also shows in the "average time to process" since those queries are then listed as taking 3+ Minutes.

Adguard, after boot, constantly spikes on the cpu. It entirely goes 100% for 2 Minutes while rebuilding filterlists and keeps constantly using a surprisingly huge amount of cpu time afterwards. This also drives the pi into much higher thermals than with pihole.

Of course, entirely not responding via dns during filterlist rebuild - or boot - is less than an optimal experience.

And to top the above it uses several times the memory of pihole, making it only feasible to be run on a larger vm/raspi4 because the system already starts swapping on the pi3.

I expected it to be a smooth swap-over to get Dnscrypt/DoH/DoT but in this state, especially since debugging takes 2-3 minutes to rebuild with no dns response of any kind during the time when i add black/whitelists its not as easy to go entirely in on Adguard Home. Especially since everything here is setup the same way. Same dns servers (just with DoH/DoT/Dnscrypt), same filterlists as pihole before.

Screenshots

You can clearly see when i switched on adguard.

Screenshots: Cpu vs pihole The spike on the left is a pihole filterlist update, the spike on the right are adguards'. ![adguardcpu](https://user-images.githubusercontent.com/44290023/91024862-d8ca2d00-e5f8-11ea-8320-d6a8e4d930a1.png) Cpu spiking closeup ![adguardcpu-closeup](https://user-images.githubusercontent.com/44290023/91024871-dc5db400-e5f8-11ea-871b-b22f72387eac.png) Memory over time vs pihole ![adguardmemory](https://user-images.githubusercontent.com/44290023/91024880-dec00e00-e5f8-11ea-9add-2f42f5f03350.png)
Full Config: ```yml bind_host: 0.0.0.0 bind_port: 80 users: - name: [omitted] password: [omitted] http_proxy: "" language: "" rlimit_nofile: 0 debug_pprof: false web_session_ttl: 720 dns: bind_host: 0.0.0.0 port: 53 statistics_interval: 90 querylog_enabled: true querylog_file_enabled: true querylog_interval: 90 querylog_size_memory: 1000 anonymize_client_ip: false protection_enabled: true blocking_mode: default blocking_ipv4: "" blocking_ipv6: "" blocked_response_ttl: 10 parental_block_host: family-block.dns.adguard.com safebrowsing_block_host: standard-block.dns.adguard.com ratelimit: 20 ratelimit_whitelist: [] refuse_any: true upstream_dns: - https://dns.digitale-gesellschaft.ch/dns-query - tls://dns2.digitalcourage.de#853 - tls://dns.digitale-gesellschaft.ch#853 - sdns://AQcAAAAAAAAAEzYyLjIxMC4xNzcuMTg5OjEwNTMgW8vytBGk6u3kvCpl4q88XjqW-w6JJiJ7QBObcFV7gYAfMi5kbnNjcnlwdC1jZXJ0Lm5zMS5pcmlzZWRlbi5mcg bootstrap_dns: - 46.182.19.48 - 1.1.1.1 all_servers: true fastest_addr: false allowed_clients: [] disallowed_clients: [] blocked_hosts: [] cache_size: 4194304 cache_ttl_min: 0 cache_ttl_max: 0 bogus_nxdomain: [] aaaa_disabled: false enable_dnssec: true edns_client_subnet: false filtering_enabled: true filters_update_interval: 24 parental_enabled: false safesearch_enabled: false safebrowsing_enabled: false safebrowsing_cache_size: 1048576 safesearch_cache_size: 1048576 parental_cache_size: 1048576 cache_time: 30 rewrites: - domain: [omitted] answer: [omitted] - domain: [omitted] answer: [omitted] - domain: [omitted] answer: [omitted] blocked_services: - skype - origin - epic_games - vk - snapchat - facebook - whatsapp - mail_ru - instagram - discord - ok - tiktok tls: enabled: false server_name: "" force_https: false port_https: 443 port_dns_over_tls: 853 allow_unencrypted_doh: false strict_sni_check: false certificate_chain: "" private_key: "" certificate_path: "" private_key_path: "" filters: - enabled: true url: https://adaway.org/hosts.txt name: https://adaway.org/hosts.txt id: 1 - enabled: true url: https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt name: https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt id: 2 - enabled: true url: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt name: https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt id: 3 - enabled: true url: https://block.energized.pro/basic/formats/hosts.txt name: https://block.energized.pro/basic/formats/hosts.txt id: 4 - enabled: true url: https://blocklistproject.github.io/Lists/abuse.txt name: https://blocklistproject.github.io/Lists/abuse.txt id: 5 - enabled: true url: https://blocklistproject.github.io/Lists/ads.txt name: https://blocklistproject.github.io/Lists/ads.txt id: 6 - enabled: true url: https://blocklistproject.github.io/Lists/fraud.txt name: https://blocklistproject.github.io/Lists/fraud.txt id: 7 - enabled: true url: https://blocklistproject.github.io/Lists/ransomware.txt name: https://blocklistproject.github.io/Lists/ransomware.txt id: 8 - enabled: true url: https://blocklistproject.github.io/Lists/scam.txt name: https://blocklistproject.github.io/Lists/scam.txt id: 9 - enabled: true url: https://blocklistproject.github.io/Lists/tracking.txt name: https://blocklistproject.github.io/Lists/tracking.txt id: 10 - enabled: true url: https://dbl.oisd.nl/ name: https://dbl.oisd.nl/ id: 11 - enabled: true url: https://gist.githubusercontent.com/BBcan177/b6df57cef74e28d90acf1eec93d62d3b/raw/f0996cf5248657ada2adb396f3636be8716b99eb/MS-4 name: https://gist.githubusercontent.com/BBcan177/b6df57cef74e28d90acf1eec93d62d3b/raw/f0996cf5248657ada2adb396f3636be8716b99eb/MS-4 id: 12 - enabled: true url: https://github.com/AdAway/adaway.github.io/blob/master/hosts.txt name: https://github.com/AdAway/adaway.github.io/blob/master/hosts.txt id: 13 - enabled: true url: https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-hosts.txt name: https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-hosts.txt id: 14 - enabled: true url: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt name: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt id: 15 - enabled: true url: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt name: https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt id: 16 - enabled: true url: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt name: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt id: 17 - enabled: true url: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list.txt name: https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list.txt id: 18 - enabled: true url: https://gnuzilla.gnu.org/filters/blacklist.txt name: https://gnuzilla.gnu.org/filters/blacklist.txt id: 19 - enabled: true url: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt name: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt id: 20 - enabled: true url: https://hosts.nfz.moe/basic/hosts name: https://hosts.nfz.moe/basic/hosts id: 21 - enabled: true url: https://mirror1.malwaredomains.com/files/justdomains name: https://mirror1.malwaredomains.com/files/justdomains id: 22 - enabled: true url: https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt name: https://mirror.cedia.org.ec/malwaredomains/immortal_domains.txt id: 23 - enabled: true url: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt name: https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt id: 24 - enabled: true url: https://paulgb.github.io/BarbBlock/blacklists/domain-list.txt name: https://paulgb.github.io/BarbBlock/blacklists/domain-list.txt id: 25 - enabled: true url: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblockplus&showintro=1&mimetype=plaintext name: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblockplus&showintro=1&mimetype=plaintext id: 26 - enabled: true url: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext name: https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext id: 27 - enabled: true url: https://phishing.army/download/phishing_army_blocklist_extended.txt name: https://phishing.army/download/phishing_army_blocklist_extended.txt id: 28 - enabled: true url: https://raw.github.com/notracking/hosts-blocklists/master/hostnames.txt name: https://raw.github.com/notracking/hosts-blocklists/master/hostnames.txt id: 29 - enabled: true url: https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt name: https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt id: 30 - enabled: true url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/adobeblock.txt name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/adobeblock.txt id: 31 - enabled: true url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/cryptomine.txt name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/cryptomine.txt id: 32 - enabled: true url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/fakenewsde.txt name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/fakenewsde.txt id: 33 - enabled: true url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/gamefake.txt name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/gamefake.txt id: 34 - enabled: true url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/jbfake.txt name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/jbfake.txt id: 35 - enabled: true url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nintendoblock.txt name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nintendoblock.txt id: 36 - enabled: true url: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nomsdata.txt name: https://raw.githubusercontent.com/Akamaru/Pi-Hole-Lists/master/nomsdata.txt id: 37 - enabled: true url: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt name: https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt id: 38 - enabled: true url: https://raw.githubusercontent.com/autinerd/anti-axelspringer-hosts/master/axelspringer-hosts name: https://raw.githubusercontent.com/autinerd/anti-axelspringer-hosts/master/axelspringer-hosts id: 39 - enabled: true url: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts name: https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts id: 40 - enabled: true url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Baidu.txt name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Baidu.txt id: 41 - enabled: true url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/HP.txt name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/HP.txt id: 42 - enabled: true url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/LG.txt name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/LG.txt id: 43 - enabled: true url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Synology.txt name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Synology.txt id: 44 - enabled: true url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Ubisoft.txt name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Ubisoft.txt id: 45 - enabled: true url: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Xiaomi.txt name: https://raw.githubusercontent.com/bloodhunterd/pi-hole-blocklists/master/Xiaomi.txt id: 46 - enabled: true url: https://raw.githubusercontent.com/buggerman/SwitchBlockerForPiHole/master/Paranoid.txt name: https://raw.githubusercontent.com/buggerman/SwitchBlockerForPiHole/master/Paranoid.txt id: 47 - enabled: true url: https://raw.githubusercontent.com/cbuijs/shallalist/master/adv/domains name: https://raw.githubusercontent.com/cbuijs/shallalist/master/adv/domains id: 48 - enabled: true url: https://raw.githubusercontent.com/cbuijs/shallalist/master/costtraps/domains name: https://raw.githubusercontent.com/cbuijs/shallalist/master/costtraps/domains id: 49 - enabled: true url: https://raw.githubusercontent.com/cbuijs/shallalist/master/religion/domains name: https://raw.githubusercontent.com/cbuijs/shallalist/master/religion/domains id: 50 - enabled: true url: https://raw.githubusercontent.com/cbuijs/shallalist/master/spyware/domains name: https://raw.githubusercontent.com/cbuijs/shallalist/master/spyware/domains id: 51 - enabled: true url: https://raw.githubusercontent.com/cbuijs/shallalist/master/tracker/domains name: https://raw.githubusercontent.com/cbuijs/shallalist/master/tracker/domains id: 52 - enabled: true url: https://raw.githubusercontent.com/CHEF-KOCH/Anti-Avast-Telemetry/master/HOSTS.txt name: https://raw.githubusercontent.com/CHEF-KOCH/Anti-Avast-Telemetry/master/HOSTS.txt id: 53 - enabled: true url: https://raw.githubusercontent.com/CHEF-KOCH/NSABlocklist/master/HOSTS/HOSTS name: https://raw.githubusercontent.com/CHEF-KOCH/NSABlocklist/master/HOSTS/HOSTS id: 54 - enabled: true url: https://raw.githubusercontent.com/CHEF-KOCH/PayWall-domains/master/HOSTS.txt name: https://raw.githubusercontent.com/CHEF-KOCH/PayWall-domains/master/HOSTS.txt id: 55 - enabled: true url: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra.txt name: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra.txt id: 56 - enabled: true url: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt name: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt id: 57 - enabled: true url: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/update.txt name: https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/update.txt id: 58 - enabled: true url: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt name: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt id: 59 - enabled: true url: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt name: https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt id: 60 - enabled: true url: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt name: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt id: 61 - enabled: true url: https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/adguard.txt name: https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/adguard.txt id: 62 - enabled: true url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts name: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts id: 63 - enabled: true url: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts name: https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts id: 64 - enabled: true url: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt name: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt id: 65 - enabled: true url: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/nocoin.txt name: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/nocoin.txt id: 66 - enabled: true url: https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/Top500 name: https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/Top500 id: 67 - enabled: true url: https://raw.githubusercontent.com/KurzGedanke/kurzBlock/master/kurzBlock.txt name: https://raw.githubusercontent.com/KurzGedanke/kurzBlock/master/kurzBlock.txt id: 68 - enabled: true url: https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt name: https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt id: 69 - enabled: true url: https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list name: https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list id: 70 - enabled: true url: https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/hosts.txt name: https://raw.githubusercontent.com/mkb2091/blockconvert/master/output/hosts.txt id: 71 - enabled: true url: https://raw.githubusercontent.com/mmotti/adguard-home-filters/master/filters.txt name: https://raw.githubusercontent.com/mmotti/adguard-home-filters/master/filters.txt id: 72 - enabled: true url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt id: 73 - enabled: true url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt id: 74 - enabled: true url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV-AGH.txt name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV-AGH.txt id: 75 - enabled: true url: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt name: https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt id: 76 - enabled: true url: https://raw.githubusercontent.com/pirat28/IHateTracker/master/iHateTracker.txt name: https://raw.githubusercontent.com/pirat28/IHateTracker/master/iHateTracker.txt id: 77 - enabled: true url: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt name: https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts_without_controversies.txt id: 78 - enabled: true url: https://raw.githubusercontent.com/PoorPocketsMcNewHold/steamscamsites/master/steamscamsite.txt name: https://raw.githubusercontent.com/PoorPocketsMcNewHold/steamscamsites/master/steamscamsite.txt id: 79 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Corona-Blocklist name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Corona-Blocklist id: 80 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/crypto name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/crypto id: 81 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Fake-Science name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Fake-Science id: 82 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/gambling name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/gambling id: 83 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/malware name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/malware id: 84 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/notserious name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/notserious id: 85 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Phishing-Angriffe name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Phishing-Angriffe id: 86 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/samsung name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/samsung id: 87 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/spam.mails name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/spam.mails id: 88 - enabled: true url: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Streaming name: https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Streaming id: 89 - enabled: true url: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt name: https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt id: 90 - enabled: true url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts id: 91 - enabled: true url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Risk/hosts id: 92 - enabled: true url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.Spam/hosts id: 93 - enabled: true url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/KADhosts/hosts id: 94 - enabled: true url: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/UncheckyAds/hosts name: https://raw.githubusercontent.com/StevenBlack/hosts/master/data/UncheckyAds/hosts id: 95 - enabled: true url: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts name: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts id: 96 - enabled: true url: https://raw.githubusercontent.com/vokins/yhosts/master/hosts name: https://raw.githubusercontent.com/vokins/yhosts/master/hosts id: 97 - enabled: true url: https://raw.githubusercontent.com/wlqY8gkVb9w1Ck5MVD4lBre9nWJez8/W10TelemetryBlocklist/master/W10TelemetryBlocklist name: https://raw.githubusercontent.com/wlqY8gkVb9w1Ck5MVD4lBre9nWJez8/W10TelemetryBlocklist/master/W10TelemetryBlocklist id: 98 - enabled: true url: https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts name: https://reddestdream.github.io/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts id: 99 - enabled: true url: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt name: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt id: 100 - enabled: true url: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt name: https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt id: 101 - enabled: true url: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt name: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt id: 102 - enabled: true url: https://someonewhocares.org/hosts/zero/hosts name: https://someonewhocares.org/hosts/zero/hosts id: 103 - enabled: true url: https://ssl.bblck.me/blacklists/hosts-file.txt name: https://ssl.bblck.me/blacklists/hosts-file.txt id: 104 - enabled: true url: https://sysctl.org/cameleon/hosts name: https://sysctl.org/cameleon/hosts id: 105 - enabled: true url: https://urlhaus.abuse.ch/downloads/hostfile/ name: https://urlhaus.abuse.ch/downloads/hostfile/ id: 106 - enabled: true url: https://v.firebog.net/hosts/BillStearns.txt name: https://v.firebog.net/hosts/BillStearns.txt id: 107 - enabled: true url: https://v.firebog.net/hosts/Easylist.txt name: https://v.firebog.net/hosts/Easylist.txt id: 108 - enabled: true url: https://v.firebog.net/hosts/Easyprivacy.txt name: https://v.firebog.net/hosts/Easyprivacy.txt id: 109 - enabled: true url: https://v.firebog.net/hosts/Prigent-Ads.txt name: https://v.firebog.net/hosts/Prigent-Ads.txt id: 110 - enabled: true url: https://v.firebog.net/hosts/Prigent-Malware.txt name: https://v.firebog.net/hosts/Prigent-Malware.txt id: 111 - enabled: true url: https://v.firebog.net/hosts/Prigent-Phishing.txt name: https://v.firebog.net/hosts/Prigent-Phishing.txt id: 112 - enabled: true url: https://v.firebog.net/hosts/Shalla-mal.txt name: https://v.firebog.net/hosts/Shalla-mal.txt id: 113 - enabled: true url: https://v.firebog.net/hosts/static/SamsungSmart.txt name: https://v.firebog.net/hosts/static/SamsungSmart.txt id: 114 - enabled: true url: https://v.firebog.net/hosts/static/w3kbl.txt name: https://v.firebog.net/hosts/static/w3kbl.txt id: 115 - enabled: true url: https://winhelp2002.mvps.org/hosts.txt name: https://winhelp2002.mvps.org/hosts.txt id: 116 - enabled: true url: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt name: https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt id: 117 - enabled: true url: https://www.github.developerdan.com/hosts/lists/amp-hosts-extended.txt name: https://www.github.developerdan.com/hosts/lists/amp-hosts-extended.txt id: 118 - enabled: true url: https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt name: https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt id: 119 - enabled: true url: https://www.malwaredomainlist.com/hostslist/hosts.txt name: https://www.malwaredomainlist.com/hostslist/hosts.txt id: 120 - enabled: true url: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser name: https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser id: 121 whitelist_filters: [] user_rules: - [omitted] - '@@||www.hcaptcha.com^$important' - "" dhcp: enabled: false interface_name: "" gateway_ip: "" subnet_mask: "" range_start: "" range_end: "" lease_duration: 86400 icmp_timeout_msec: 1000 clients: [] log_compress: false log_localtime: false log_max_backups: 0 log_max_size: 100 log_max_age: 3 log_file: "" verbose: false schema_version: 6 ```

Additional Information

szolin commented 4 years ago

Of course, entirely not responding via dns during filterlist rebuild - or boot - is less than an optimal experience.

AGH doesn't respond to DNS queries until the first initialization of filtering rules on startup - true. But when filters are being updated, added or removed AGH keeps responding to DNS queries as usual.

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C. Further performance optimization of AGH may take much time to achieve, which means implementing less features. But of course we'll do what we can.

ameshkov commented 4 years ago

@waffshappen Please

  1. What's the average load (requests per second)?
  2. What do you mean by "debugging"? Editing custom filtering rules or something else?

@szolin

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C.

The main part of the difference is due to using an encrypted DNS server vs plain DNS. Other than that, the difference is negligible.

What bothers me more is that we don't handle such configurations (dozens of blocklists with millions of rules) well enough. Let's see what usecases @waffshappen and check if we can avoid full engine re-init.

szolin commented 4 years ago

The main part of the difference is due to using an encrypted DNS server vs plain DNS

Oh, I didn't notice he's comparing plain vs encrypted. In that case C vs Go doesn't matter of course.

waffshappen commented 4 years ago

Of course, entirely not responding via dns during filterlist rebuild - or boot - is less than an optimal experience.

AGH doesn't respond to DNS queries until the first initialization of filtering rules on startup - true. But when filters are being updated, added or removed AGH keeps responding to DNS queries as usual.

Thats precisely the issue. When adding a blocklist while its running it is spinning for a solid minute and not answering any query during the time for me. (Web gui remains active)

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C. Further performance optimization of AGH may take much time to achieve, which means implementing less features. But of course we'll do what we can.

The cpu spiking does look more than just c/go as difference. Something is keeping adguard active the entire time.

@waffshappen Please

  1. What's the average load (requests per second)?

Unless i'm running dnstorment i max out on 10-20 req/sec when all devices are being used in parallel and browse the web. But the entire time you can see up there it was <1req/sec

  1. What do you mean by "debugging"? Editing custom filtering rules or something else?

Any and all actions. Adding allow/blocklist entries, adding or removing rules, anything really. The dns service doesnt respond at all during that time. (Web gui remains up)

@szolin

Regarding higher CPU and memory usage than dnsmasq(pihole) - it's expected, because dnsmasq is written in C.

The main part of the difference is due to using an encrypted DNS server vs plain DNS. Other than that, the difference is negligible.

What bothers me more is that we don't handle such configurations (dozens of blocklists with millions of rules) well enough. Let's see what usecases @waffshappen and check if we can avoid full engine re-init.

The ideal case would be to reduce the constant cpu load - but i am not sure what is generating it. I could try to selfbuild it and get some perf stats on it if needed and if it is not reproducible with my config posted above. And of course remaining responsive during the time it is rebuilding and ideally saving the cached last build so it starts faster.

waffshappen commented 4 years ago

Screenshot during a blocklist update with nslookup timing out

adguardupdateblocklist

ameshkov commented 4 years ago

@szolin

I guess this is actually several issues in one, we should investigate them all and create new tasks on GH for each of them.

  1. Unnecessary blocking on filtering engine reload;
  2. Constant CPU load -- I don't understand where it comes from; Try to repro with the @waffshappen's configuration.
waffshappen commented 4 years ago

Created the two issues, but there's several more points i'd maybe bring up:

I'll selfbuild latest release in a moment and try to track down what keeps spiking.

ameshkov commented 4 years ago

Possible caching of the generated database on reboots?

Possible, but I'd better avoid doing this if possible.

Possibly doing the same as pihole and building a deduplicated database in parallel, iirc they're using sqlite/something custom to do it?

What do you mean by "in parallel"?

And going on a stretch: Would it be possible to lessen database load and build times by adding the check if a domain is (possibly) in the blocklist with a bloom filter? Additionally to deduplicating entries from blocklists.

I think it would, but a straightforward approach to deduplication involves will require quite a lot of RAM.

waffshappen commented 4 years ago

Possible caching of the generated database on reboots?

Possible, but I'd better avoid doing this if possible.

True, but it'd be a short-term solution

Possibly doing the same as pihole and building a deduplicated database in parallel, iirc they're using sqlite/something custom to do it?

What do you mean by "in parallel"?

pihole uses a sqlite database and then blasts the entries into it. That uses significantly less memory and has with free deduplication built into it since its an sql engine.

Plus it separates domains and user filters/regexes. So a change to the latter is basically instant by only requiring a single insert.

https://github.com/pi-hole/pi-hole/blob/master/gravity.sh

This would be doable straight in go, by loading the sqlite handler and swapping it in for all current storage methods using either a on-disk, or even in-memory database, with far less memory use than whatever the current system does, in multiple threads handling several lists at once.

If possible for adguard to support huge filterlists instead of building them i guess straight into memory with several parallel workers why not re-use existing validation filters instead of spawning new objects (as it seems right now from pprof) and then having multiple write references to an in-memory sqlite db, or even on-disk, that can do automatic deduplication of entries (INSERT OR IGNORE etc). That would possibly save a lot of memory, and the less objects spawn the less the need for manually calling memory flush, fixing that issue too.

And going on a stretch: Would it be possible to lessen database load and build times by adding the check if a domain is (possibly) in the blocklist with a bloom filter? Additionally to deduplicating entries from blocklists.

I think it would, but a straightforward approach to deduplication involves will require quite a lot of RAM.

pihole already pulls this off, by using sqlite, with minimal memory overuse. In theory i could try adding this as Minimum Reproduction - replacing all references to the current storage method with sqlite calls and embedding it with go, but that'd also require CGO to be enabled, introducing c bindings and needing to build with musl libc and net_go tag to allow to keep the current amount of supported platforms. (pi-hole solves this by requiring the user to install sqlite and doing it so automatically in their install script)

Alternatively minimal storages like this exist in pure go. Would either request be evaluated or would any CGO requiring sqlite request be dropped?

ameshkov commented 4 years ago

I'd like to avoid using any CGO to keep AGH written in pure Go.

instead of spawning new objects (as it seems right now from pprof) and then having multiple write references to an in-memory sqlite db, or even on-disk

These objects aren't long-living, in the end, we are keeping a simple index:

lookupTable map[uint32][]int64 // map for hosts hashes mapped to the list of rule indexes

Rule indexes point to the rule location in the file.

Unfortunately, Go is not too good at freeing memory. Even if we avoid creating these temporary structs, mere strings will be still allocated and that would still quite a lot of allocations.

Our further actions should depend on what exactly we're going to improve.

  1. Memory allocations on filters initialization: I don't think there's much that can be done about this if we keep AGH written in pure Go.
  2. Filters engine initialization performance:
    • This can be solved, DNSEngine just needs to support serialization/deserialization to a file.
    • Changes to custom filtering rules need to be handled independently and shouldn't lead to reloading the whole filtering engine.
XrosSJ commented 4 years ago

Same question,I'm running in Raspberrypi 2B

sekaiacg commented 4 years ago

@ameshkov 开发者你好 某一天我发现adh高cpu占用,进一步排查发现:上游dns服务器失效的时候,adh会出现高cpu占用,内存使用也随着时间增加而增加。不知道是不是个例。

ameshkov commented 4 years ago

@sekaiacg 上游dns服务器失效的时候,AGH不能解析,它必须等每个解析查询,结果AGH做更多“goroutine“也用更多cpu时间。

如果你用一些上游dns服务器也在dns设置启用负载均衡,AGH就用更快的上游dns服务器。

sekaiacg commented 4 years ago

@sekaiacg 上游dns服务器失效的时候,AGH不能解析,它必须等每个解析查询,结果AGH做更多“goroutine“也用更多cpu时间。

如果你用一些上游dns服务器也在dns设置启用负载均衡,AGH就用更快的上游dns服务器。

好的谢谢解答

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.