search

AdguardTeam / AdGuardHome

Network-wide ads & trackers blocking DNS server
https://adguard.com/adguard-home.html
GNU General Public License v3.0
24.46k stars 1.77k forks source link

Memory leak and device crash #4437

Closed timkgh closed 1 year ago

timkgh commented 2 years ago

Issue Details

Expected Behavior

Run for long periods of time without crashes/memory leak.

Actual Behavior

Leaks memory until the router is out of memory and it crashes.

Screenshots

After about 26 days of uptime:

agh1

After the screenshot above, I restarted AGH and it shows a reasonable memory footprint:

agh2

You can see the memory use on the device drops from 456M to 177M after an AGH restart, but it continues to climb until it eventually exhausts the entire device memory.

Additional Information

Cache settings:

cache_size: 4194304
cache_ttl_min: 300
cache_ttl_max: 86400
cache_optimistic: true
safebrowsing_cache_size: 1048576
safesearch_cache_size: 1048576
parental_cache_size: 1048576
cache_time: 30

Clearing the stats and logs while AGH is running does not help reclaim memory, only an AGH restart does.

jumpsmm7 commented 2 years ago

@timkgh When you call a function, a new "namespace" is allocated on the stack. That's how functions can have local variables. As functions call functions, which in turn call functions, we keep allocating more and more space on the stack to maintain this deep hierarchy of namespaces. To curb programs using massive amounts of stack space, a limit is usually put in place via ulimit -s. If we remove that limit via ulimit -s unlimited, our programs will be able to keep gobbling up RAM for their evergrowing stack until eventually the system runs out of memory entirely.

Usually, using a ton of stack space is accidental or a symptom of very deep recursion that probably should not be relying so much on the stack. Thus the stack limit.

jumpsmm7 commented 2 years ago

@timkgh

I noticed my climb in memory usage when updating the list significantly slows down when I add the ulimit to my setup.

timkgh commented 2 years ago

@jumpsmm7 I doubt AGH deals with any deep stacks or recursion. The stack should be freed anyway when a function call returns.

jumpsmm7 commented 2 years ago

@jumpsmm7 I doubt AGH deals with any deep stacks or recursion. The stack should be freed anyway when a function call returns.

agreed "should be freed", but where is the evidence that it is if you are running out of memory from its usage. As your post above, it might actually have something to do with GO itself on mips and it could be stack related if it is a symptom of the conditions.

timkgh commented 2 years ago

@jumpsmm7 I doubt AGH deals with any deep stacks or recursion. The stack should be freed anyway when a function call returns.

agreed "should be freed", but where is the evidence that it is if you are running out of memory from its usage. As your post above, it might actually have something to do with GO itself on mips.

You may be thinking heap instead of stack. The stack is not garbage collected, the heap is.

jumpsmm7 commented 2 years ago

Both are dictated by the kernel typically and in this scenario, Go may be creating all sorts of implications with kernel version of mips you are using.

jumpsmm7 commented 2 years ago

@timkgh

you can use "ulimit -d #some-value" to manage size of heap, also you can use "ulimit -v #some-value" to control the total memory allotment which includes stack, heap, and any other types of memory allocations acquired by the program.

timkgh commented 2 years ago

@EugeneOne1

Looking at pprof multiple times over the past few days the heap use is about constant.

File: AdGuardHome
Type: inuse_space
Time: May 4, 2022 at 7:28am (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 15628.04kB, 88.41% of 17676.32kB total
Showing top 10 nodes out of 92
      flat  flat%   sum%        cum   cum%
 8623.44kB 48.79% 48.79%  8623.44kB 48.79%  github.com/AdguardTeam/urlfilter/lookup.(*ShortcutsTable).TryAdd
 1536.61kB  8.69% 57.48%  1536.61kB  8.69%  runtime.malg
 1184.27kB  6.70% 64.18% 10319.83kB 58.38%  github.com/AdguardTeam/urlfilter.NewDNSEngine
 1026.81kB  5.81% 69.99%  1026.81kB  5.81%  github.com/AdguardTeam/AdGuardHome/internal/filtering.(*sbCtx).setCache
  632.14kB  3.58% 73.56%   632.14kB  3.58%  github.com/AdguardTeam/golibs/cache.(*cache).Set
  570.04kB  3.22% 76.79%  1594.21kB  9.02%  github.com/AdguardTeam/urlfilter/filterlist.(*RuleStorage).RetrieveRule
     515kB  2.91% 79.70%      515kB  2.91%  github.com/AdguardTeam/AdGuardHome/internal/home.NewRDNS
     515kB  2.91% 82.62%      515kB  2.91%  golang.org/x/net/http2/hpack.(*headerFieldTable).addEntry
  512.56kB  2.90% 85.51%   512.56kB  2.90%  runtime.allocm
  512.16kB  2.90% 88.41%   512.16kB  2.90%  github.com/AdguardTeam/urlfilter/rules.NewNetworkRule

I don't know where the memory is going, the RES(ident) size of the process keeps climbing up.

Am I missing something in the way I'm looking at the pprof dump?

jumpsmm7 commented 2 years ago

@timkgh

@EugeneOne1

Looking at pprof multiple times over the past few days the heap use is about constant.

File: AdGuardHome
Type: inuse_space
Time: May 4, 2022 at 7:28am (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 15628.04kB, 88.41% of 17676.32kB total
Showing top 10 nodes out of 92
      flat  flat%   sum%        cum   cum%
 8623.44kB 48.79% 48.79%  8623.44kB 48.79%  github.com/AdguardTeam/urlfilter/lookup.(*ShortcutsTable).TryAdd
 1536.61kB  8.69% 57.48%  1536.61kB  8.69%  runtime.malg
 1184.27kB  6.70% 64.18% 10319.83kB 58.38%  github.com/AdguardTeam/urlfilter.NewDNSEngine
 1026.81kB  5.81% 69.99%  1026.81kB  5.81%  github.com/AdguardTeam/AdGuardHome/internal/filtering.(*sbCtx).setCache
  632.14kB  3.58% 73.56%   632.14kB  3.58%  github.com/AdguardTeam/golibs/cache.(*cache).Set
  570.04kB  3.22% 76.79%  1594.21kB  9.02%  github.com/AdguardTeam/urlfilter/filterlist.(*RuleStorage).RetrieveRule
     515kB  2.91% 79.70%      515kB  2.91%  github.com/AdguardTeam/AdGuardHome/internal/home.NewRDNS
     515kB  2.91% 82.62%      515kB  2.91%  golang.org/x/net/http2/hpack.(*headerFieldTable).addEntry
  512.56kB  2.90% 85.51%   512.56kB  2.90%  runtime.allocm
  512.16kB  2.90% 88.41%   512.16kB  2.90%  github.com/AdguardTeam/urlfilter/rules.NewNetworkRule

I don't know where the memory is going, the RES(ident) size of the process keeps climbing up.

Am I missing something in the way I'm looking at the pprof dump?

this kind of reminds me of this issue.

https://github.com/golang/go/issues/38257

and here is a resource tool that might help you... https://blog.detectify.com/2019/09/05/how-we-tracked-down-a-memory-leak-in-one-of-our-go-microservices/

image

jumpsmm7 commented 2 years ago

@timkgh

The default go behavior is mafree. Maybe it is not an actual memory leak, however maybe it is go not releasing memory at the time indications it is suppose to. Supposedly, the go program should start to let go of memory approximately five minutes from the time it first gains hold of it, and determines it is not being used. This is assuming the go process has reached a point of "calm". Options like GODEBUG=madvdontneed=1 would change this behavior allowing for the RSS to be released almost instantly upon determining memory is no longer needed.

timkgh commented 2 years ago

It doesn't seem to ever release memory, the process keeps growing and growing until the router is out of memory and reboots, though lately the process has been growing very slowly. It's possible something is messed up on the mips architecture. I'll try to keep on running like this and occasionally look at pprof. I'll then try GODEBUG=madvdontneed=1.

jumpsmm7 commented 2 years ago

It doesn't seem to ever release memory, the process keeps growing and growing until the router is out of memory and reboots, though lately the process has been growing very slowly. It's possible something is messed up on the mips architecture. I'll try to keep on running like this and occasionally look at pprof. I'll then try GODEBUG=madvdontneed=1.

@timkgh Yea this whole study I posted above was based on the similar premise that RSS kept growing until the Out of memory situation was present. Originally the incident only happen on ios platforms but the writers of the of the research were also experiencing the same with Linux OS. The problem was, RSS kept growing with no evidence of an actual leak from the pprof tools. While pprof is great for finding actual leaks, it does not account for a hidden flaw in the go itself.

jumpsmm7 commented 2 years ago

I am looking into investigating the pprof with promethius now.

timkgh commented 2 years ago

The size of the heap has grown a bit but that seems normal due to more cached entries and the cache size seems to stay at the 4MB limit.

File: AdGuardHome
Type: inuse_space
Time: May 11, 2022 at 6:41pm (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 19897.28kB, 92.83% of 21433.33kB total
Showing top 10 nodes out of 88
      flat  flat%   sum%        cum   cum%
 7184.10kB 33.52% 33.52%  7184.10kB 33.52%  github.com/AdguardTeam/urlfilter/lookup.(*ShortcutsTable).TryAdd
 3585.83kB 16.73% 50.25%  4097.86kB 19.12%  github.com/AdguardTeam/AdGuardHome/internal/filtering.(*sbCtx).setCache
 2048.81kB  9.56% 59.81%  2048.81kB  9.56%  runtime.malg
 1792.43kB  8.36% 68.17%  1792.43kB  8.36%  github.com/AdguardTeam/golibs/cache.(*cache).Set
 1696.29kB  7.91% 76.08%  8880.39kB 41.43%  github.com/AdguardTeam/urlfilter.NewDNSEngine
 1536.43kB  7.17% 83.25%  1536.43kB  7.17%  github.com/AdguardTeam/dnsproxy/proxy.(*cacheItem).pack
     515kB  2.40% 85.66%      515kB  2.40%  github.com/AdguardTeam/AdGuardHome/internal/home.NewRDNS
  513.69kB  2.40% 88.05%   513.69kB  2.40%  bytes.makeSlice
  512.56kB  2.39% 90.44%   512.56kB  2.39%  runtime.allocm
  512.14kB  2.39% 92.83%   512.14kB  2.39%  github.com/AdguardTeam/AdGuardHome/internal/dnsforward.msgToSignature

However, the RES(ident) process size has grown from about 50MB to 102MB now.

timkgh commented 2 years ago

@EugeneOne1 any indications of memory leaks? The heap is growing somewhat.

File: AdGuardHome
Type: inuse_space
Time: May 18, 2022 at 7:18am (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 25553.75kB, 92.58% of 27602.48kB total
Showing top 10 nodes out of 98
      flat  flat%   sum%        cum   cum%
 9751.09kB 35.33% 35.33%  9751.09kB 35.33%  github.com/AdguardTeam/urlfilter/lookup.(*ShortcutsTable).TryAdd
 5124.76kB 18.57% 53.89%  5124.76kB 18.57%  github.com/AdguardTeam/AdGuardHome/internal/filtering.(*sbCtx).setCache
 3072.98kB 11.13% 65.03%  3072.98kB 11.13%  github.com/AdguardTeam/dnsproxy/proxy.(*cacheItem).pack
 2048.81kB  7.42% 72.45%  2048.81kB  7.42%  runtime.malg
 1280.29kB  4.64% 77.09%  1280.29kB  4.64%  github.com/AdguardTeam/golibs/cache.(*cache).Set
 1184.27kB  4.29% 81.38% 10935.37kB 39.62%  github.com/AdguardTeam/urlfilter.NewDNSEngine
 1024.31kB  3.71% 85.09%  1024.31kB  3.71%  github.com/AdguardTeam/urlfilter/rules.NewNetworkRule
 1024.06kB  3.71% 88.80%  1024.06kB  3.71%  encoding/hex.DecodeString
  528.17kB  1.91% 90.71%   528.17kB  1.91%  regexp.(*bitState).reset
     515kB  1.87% 92.58%      515kB  1.87%  github.com/AdguardTeam/AdGuardHome/internal/home.NewRDNS
lex-obrien commented 2 years ago

I just wanted to add my name / info here that, and I experiencing the same issue running AdGuard v0.107.6 (armv7) on OpenWRT. Not happy that the issue is happening, but so happy that there is so much good discussion on the topic!

Some quick pprofing shows very similar heap:

File: AdGuardHome
Type: inuse_space
Time: May 20, 2022 at 12:50am (CDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 6590.64kB, 100% of 6590.64kB total
Showing top 10 nodes out of 34
      flat  flat%   sum%        cum   cum%
 1805.17kB 27.39% 27.39%  1805.17kB 27.39%  compress/flate.NewWriter
 1414.59kB 21.46% 48.85%  1414.59kB 21.46%  github.com/AdguardTeam/urlfilter/lookup.(*ShortcutsTable).TryAdd
 1320.74kB 20.04% 68.89%  3759.35kB 57.04%  github.com/AdguardTeam/urlfilter.NewDNSEngine
 1024.02kB 15.54% 84.43%  1024.02kB 15.54%  github.com/AdguardTeam/urlfilter.(*DNSEngine).addRule
     514kB  7.80% 92.23%      514kB  7.80%  bufio.NewReaderSize (inline)
  512.12kB  7.77%   100%   512.12kB  7.77%  runtime.malg
         0     0%   100%      514kB  7.80%  bufio.NewReader
         0     0%   100%  1805.17kB 27.39%  compress/gzip.(*Writer).Write
         0     0%   100%  1805.17kB 27.39%  encoding/json.(*Encoder).Encode
         0     0%   100%  3759.35kB 57.04%  github.com/AdguardTeam/AdGuardHome/internal/filtering.(*DNSFilter).SetFilters
File: AdGuardHome
Type: inuse_space
Time: May 20, 2022 at 12:57am (CDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 7754.79kB, 100% of 7754.79kB total
Showing top 10 nodes out of 71
      flat  flat%   sum%        cum   cum%
 1414.59kB 18.24% 18.24%  1414.59kB 18.24%  github.com/AdguardTeam/urlfilter/lookup.(*ShortcutsTable).TryAdd
 1320.74kB 17.03% 35.27%  3759.35kB 48.48%  github.com/AdguardTeam/urlfilter.NewDNSEngine
 1024.02kB 13.20% 48.48%  1024.02kB 13.20%  github.com/AdguardTeam/urlfilter.(*DNSEngine).addRule
  902.59kB 11.64% 60.12%   902.59kB 11.64%  compress/flate.NewWriter
  532.26kB  6.86% 66.98%  1044.35kB 13.47%  github.com/AdguardTeam/urlfilter/filterlist.(*RuleStorage).RetrieveRule
  512.34kB  6.61% 73.59%   512.34kB  6.61%  crypto/x509.parseCertificate
  512.12kB  6.60% 80.19%   512.12kB  6.60%  runtime.malg
  512.09kB  6.60% 86.79%   512.09kB  6.60%  github.com/AdguardTeam/urlfilter/rules.NewNetworkRule
  512.03kB  6.60% 93.40%   512.03kB  6.60%  encoding/hex.DecodeString
  512.02kB  6.60%   100%   512.02kB  6.60%  github.com/AdguardTeam/dnsproxy/proxy.msgToKey
trinib commented 2 years ago

I thought I'd share a live preview of memory rising. It is running a python script(@EugeneOne1 maybe use this to reproduce issue quickly) for adding blocklist URLs to AGH.

AdGuard Home v0.107.6
Raspberry Pi 3B+
Raspberry OS 64bit

Come on AGH team, this needs to be solved !🙏. This is serious issue

https://user-images.githubusercontent.com/18756975/169598692-55250e21-f822-4821-a18a-cb89f5425eae.mp4

trinib commented 2 years ago

If anyone decides to use script for issue reproduction. use this with more URLs than the previous one I posted. It actually freezes Pi until python request shows connection closed from AGH memory lockup, it goes above 90%. memory usage. Even after reboot it rises up quickly for no reason.

htop

SCRIPT ``` import requests import ssl import json from requests.adapters import HTTPAdapter from requests.packages.urllib3.poolmanager import PoolManager # +++ A SCRIPT TO ADD MULTIPLE BLOCKLIST AT ONCE IN ADGUARD +++ # ### RUN SCRIPT ### # sudo python3 bulkurls.py ### SET IP / USERNAME / PASSWORD USING *YOUR* ADGUARD CREDENTIALS ### host = "http://your_pi_ip:80" # Your user name userName = "Yours" # Your password password = "Yours" # My blocklist (add your own: https://github.com/T145/black-mirror) urls = [ "https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt", "https://adaway.org/hosts.txt", "https://someonewhocares.org/hosts/zero/hosts", "https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt", "https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hosts", "https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV-AGH.txt", "https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt", "https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblockplus&showintro=1&mimetype=plaintext", "https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt", "https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/adguard.txt", "https://curben.gitlab.io/malware-filter/urlhaus-filter-agh-online.txt", "https://paulgb.github.io/BarbBlock/blacklists/hosts-file.txt", "https://abp.oisd.nl/", "https://v.firebog.net/hosts/Easyprivacy.txt", "https://v.firebog.net/hosts/Prigent-Ads.txt", "https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt", "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts", "https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt", "https://v.firebog.net/hosts/static/w3kbl.txt", "https://v.firebog.net/hosts/AdguardDNS.txt", "https://v.firebog.net/hosts/Admiral.txt", "https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt", "https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt", "https://v.firebog.net/hosts/Easylist.txt", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext", "https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts", "https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/TOP_EU_US_Ads_Trackers_HOST", "https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt", "https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt", "https://v.firebog.net/hosts/Prigent-Crypto.txt", "https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt", "https://phishing.army/download/phishing_army_blocklist_extended.txt", "https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt", "https://v.firebog.net/hosts/Shalla-mal.txt", "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts", "https://urlhaus.abuse.ch/downloads/hostfile/", "https://raw.githubusercontent.com/neodevpro/neodevhost/master/lite_host", "https://raw.githubusercontent.com/neodevpro/neodevhost/master/lite_adblocker", "https://perflyst.github.io/PiHoleBlocklist/AmazonFireTV.txt", "https://perflyst.github.io/PiHoleBlocklist/android-tracking.txt", "https://perflyst.github.io/PiHoleBlocklist/regex.list", "https://perflyst.github.io/PiHoleBlocklist/SessionReplay.txt", "https://blocklistproject.github.io/Lists/phishing.txt", "https://blocklistproject.github.io/Lists/tracking.txt", "https://raw.githubusercontent.com/ftpmorph/ftprivacy/master/blocklists/smartphone-ads-tracking.txt", "https://raw.githubusercontent.com/CleanMachine1/AdlistTXTS/main/adservers.txt", "https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt", "https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_15_DnsFilter/filter.txt", "https://easylist-downloads.adblockplus.org/easylist.txt", "https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_11_Mobile/filter.txt", "https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_3_Spyware/filter.txt", "https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_2_English/filter.txt", "https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_4_Social/filter.txt", "https://easylist.to/easylist/easyprivacy.txt", "https://easylist.to/easylist/fanboy-annoyance.txt", "https://easylist.to/easylist/fanboy-social.txt", "https://anti-ad.net/easylist.txt", "https://hosts.anudeep.me/mirror/adservers.txt", "https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/abp_light.txt", "https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dbl_light.txt", "https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/hosts_light.txt", "https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dnsmasq_light.txt", "https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/rpz_basic.rpz", "https://raw.githubusercontent.com/smed79/blacklist/master/hosts.txt", "https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt", "https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt", "https://raw.githubusercontent.com/mitchellkrogza/Phishing.Database/master/phishing-domains-ACTIVE.txt", "https://gitlab.com/quidsup/notrack-blocklists/-/raw/master/malware.list", "https://gitlab.com/quidsup/notrack-blocklists/-/raw/master/trackers.list", "https://raw.githubusercontent.com/yous/YousList/master/hosts.txt", "https://zerodot1.gitlab.io/CoinBlockerLists/list.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/list_browser.txt", "https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/domains", "https://easylist-downloads.adblockplus.org/easyprivacy.txt", "https://raw.githubusercontent.com/privacy-protection-tools/anti-AD/master/anti-ad-easylist.txt", "https://www.i-dont-care-about-cookies.eu/abp/", "https://hblock.molinero.dev/hosts", "http://winhelp2002.mvps.org/hosts.txt", "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts", "https://raw.githubusercontent.com/VeleSila/yhosts/master/hosts.txt", "https://iplists.firehol.org/files/alienvault_reputation.ipset", "https://orca.pet/notonmyshift/domains.txt", "https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/firewall/spy.txt", "https://raw.githubusercontent.com/d43m0nhLInt3r/socialblocklists/master/Windows/windowstelemetryblocklist.txt", "https://raw.githubusercontent.com/d43m0nhLInt3r/socialblocklists/master/MobileAppAds/appadsblocklist.txt", "https://raw.githubusercontent.com/d43m0nhLInt3r/socialblocklists/master/Android/androidblocklist.txt", "https://raw.githubusercontent.com/twcau/AdblockRules/master/MurdochList", "https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews/hosts", "https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/gambling/hosts", "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts", "https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts", "https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts", "https://winhelp2002.mvps.org/hosts.txt", "https://raw.githubusercontent.com/notracking/hosts-blocklists/master/adblock/adblock.txt", "https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/unbound.conf", "https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/hosts.txt", "https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/adblock.txt", "https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/hosts.win", "https://blocklistproject.github.io/Lists/adguard/ransomware-ags.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt", "https://curben.gitlab.io/malware-filter/urlhaus-filter.txt", "https://abp.oisd.nl/basic/", "https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareAdGuardHome.txt", "https://curben.gitlab.io/malware-filter/urlhaus-filter-ag.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/resource-abuse.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt", "https://raw.githubusercontent.com/easylist/easylist/master/easyprivacy/easyprivacy_general.txt", "https://secure.fanboy.co.nz/fanboy-cookiemonster.txt", "https://easylist-downloads.adblockplus.org/easyprivacy+easylist.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/annoyances.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters-2022.txt", "https://secure.fanboy.co.nz/fanboy-annoyance.txt", "https://easylist.to/easylist/easylist.txt", "https://curbengh.github.io/malware-filter/urlhaus-filter.txt", "https://curben.gitlab.io/malware-filter/urlhaus-filter-vivaldi.txt", "https://curben.gitlab.io/malware-filter/urlhaus-filter-domains.txt", "https://curben.gitlab.io/malware-filter/urlhaus-filter-agh.txt", "https://curben.gitlab.io/malware-filter/urlhaus-filter-hosts.txt", "https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_adservers.txt", "https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_general_block.txt", "https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_adservers_popup.txt", "https://raw.githubusercontent.com/easylist/easylist/master/easyprivacy/easyprivacy_trackingservers.txt", "https://raw.githubusercontent.com/angad/webPriority/master/easylist/easylist/easylist_general_block.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/thirdparties/easylist-downloads.adblockplus.org/easylist.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/thirdparties/easylist-downloads.adblockplus.org/easyprivacy.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters-2021.txt", "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters-2020.txt", "https://raw.githubusercontent.com/easylist/easylist/master/easyprivacy/easyprivacy_thirdparty.txt", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblock&showintro=0&mimetype=plaintext", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=littlesnitch&showintro=0&mimetype=plaintext", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=junkbuster&showintro=0&mimetype=plaintext", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=unbound&showintro=0&mimetype=plaintext", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=iptables&showintro=0&mimetype=plaintext", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=msfilter&showintro=0&mimetype=plaintext", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=operafilter&showintro=0&mimetype=plaintext", "https://pgl.yoyo.org/adservers/serverlist.php?hostformat=machosts&showintro=0&mimetype=plaintext", "https://raw.githubusercontent.com/BlackJack8/iOSAdblockList/master/Regular%20Hosts.txt", "https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock.txt", "https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock_lite.txt", "https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock_plus.txt", "https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock_privacy.txt", "https://raw.githubusercontent.com/TonyRL/blocklist/master/hosts", "https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt", "https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt", "https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt", "https://raw.githubusercontent.com/ShadowWhisperer/BlockLists/master/Lists/Tracking", "https://raw.githubusercontent.com/ShadowWhisperer/BlockLists/master/Lists/Malware", "https://raw.githubusercontent.com/hemiipatu/PiHoleBlocklists/master/blocklists/advertisement.txt", "https://raw.githubusercontent.com/hemiipatu/PiHoleBlocklists/master/blocklists/phishing.txt", "https://raw.githubusercontent.com/DandelionSprout/adfilt/master/AdGuard%20Home%20Compilation%20List/AdGuardHomeCompilationList.txt", "https://raw.githubusercontent.com/ftpmorph/ftpihole/master/blocklists/smartphone-ads-tracking.txt", "https://raw.githubusercontent.com/mhxion/pornaway/master/hosts/porn_ads.txt", "https://raw.githubusercontent.com/privacy-protection-tools/anti-AD/master/anti-ad-domains.txt", "https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Hosts/GoodbyeAds.txt", "http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-CIF-latest.txt", "https://anti-ad.net/domains.txt", "https://block.energized.pro/assets/sources/filter/better-fyi-trackers.txt", "https://block.energized.pro/assets/sources/filter/sinfonietta-porn.txt", "https://block.energized.pro/extensions/xtreme/formats/domains.txt", "https://blocklistproject.github.io/Lists/alt-version/everything-nl.txt", "https://blocklistproject.github.io/Lists/alt-version/smart-tv-nl.txt", "https://blocklistproject.github.io/Lists/alt-version/vaping-nl.txt", "https://blocklistproject.github.io/Lists/alt-version/whatsapp-nl.txt", "https://blokada.org/blocklists/ddgtrackerradar/standard/hosts.txt", "https://cinsscore.com/list/ci-badguys.txt", "https://data.netlab.360.com/feeds/dga/dga.txt", "https://dataplane.org/sipinvitation.txt", "https://dataplane.org/sipquery.txt", "https://dataplane.org/sipregistration.txt", "https://dataplane.org/sshpwauth.txt", "https://dataplane.org/vncrfb.txt", "https://fanboy.co.nz/fanboy-antifonts.txt", "https://fanboy.co.nz/fanboy-cookiemonster.txt", "https://fanboy.co.nz/fanboy-indian.txt", "https://fanboy.co.nz/r/fanboy-ultimate.txt", "https://feeds.dshield.org/block.txt", "https://filter.futa.gg/hosts.txt", "https://filtri-dns.ga/filtri.txt", "https://getblackbird.net/blacklist/hosts.txt", "https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/Extreme%20lists/The_Quantum_Extreme_Youtube_Ad-List.txt", "https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Abuse-List.txt", "https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Privacy-list.txt", "https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Simply-ads-list.txt", "https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Youtube-Ads-List.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/list.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/list_optional.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/white_list.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_optional.txt", "https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/white_list.txt", "https://gitlab.com/quidsup/ntrk-tracker-radar/-/raw/master/ddg_tracker_radar_confirmed.txt", "https://hostfiles.frogeye.fr/firstparty-trackers.txt", "https://hostfiles.frogeye.fr/multiparty-trackers.txt", "https://hosts.anudeep.me/mirror/CoinMiner.txt", "https://hosts.anudeep.me/mirror/facebook.txt", "https://hosts.anudeep.me/mirror/miscellaneous/false-positives.txt", "https://kriskintel.com/feeds/ktip_covid_domains.txt", "https://kriskintel.com/feeds/ktip_malicious_Ips.txt", "https://kriskintel.com/feeds/ktip_malicious_domains.txt", "https://kriskintel.com/feeds/ktip_ransomware_feeds.txt", "https://lists.blocklist.de/lists/all.txt", "https://malsilo.gitlab.io/feeds/dumps/domain_list.txt", "https://mirai.security.gives/data/ip_list.txt", "https://myip.ms/files/bots/live_webcrawlers.txt", "https://openphish.com/feed.txt", "https://orca.pet/notonmyshift/hosts.txt", "https://osint.digitalside.it/Threat-Intel/lists/latestips.txt", "https://public-dns.info/nameservers.txt", "https://raw.githubusercontent.com/4skinSkywalker/Anti-Porn-HOSTS-File/master/HOSTS.txt", "https://raw.githubusercontent.com/AdguardTeam/cname-trackers/master/combined_disguised_trackers_justdomains.txt", "https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/activation.txt", "https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/adservers-and-trackers.txt", "https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/fake-domains.txt", "https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/search-blacklist.txt", "https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/unwanted-iranian.txt", "https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareDomains.txt", "https://raw.githubusercontent.com/DandelionSprout/adfilt/master/NorwegianList.txt", "https://raw.githubusercontent.com/EnergizedProtection/unblock/master/basic/formats/domains.txt", "https://raw.githubusercontent.com/Ewpratten/youtube_ad_blocklist/master/blocklist.txt", "https://raw.githubusercontent.com/FiltersHeroes/KADhosts/master/KADomains.txt", "https://raw.githubusercontent.com/FutaGuard/FutaFilter/master/hosts.txt", "https://raw.githubusercontent.com/List-KR/List-KR/master/filter.txt", "https://raw.githubusercontent.com/MajkiIT/polish-ads-filter/master/polish-pihole-filters/hole_cert_pl.txt", "https://raw.githubusercontent.com/MajkiIT/polish-ads-filter/master/polish-pihole-filters/hostfile.txt", "https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SessionReplay.txt", "https://raw.githubusercontent.com/TheAntiSocialEngineer/AntiSocial-BlockList-UK-Community/main/UK-Community.txt", "https://raw.githubusercontent.com/allendema/noplaylist/main/NoPlayList.txt", "https://raw.githubusercontent.com/anudeepND/blacklist/master/CoinMiner.txt", "https://raw.githubusercontent.com/anudeepND/blacklist/master/facebook.txt", "https://raw.githubusercontent.com/anudeepND/blacklist/master/miscellaneous/false-positives.txt", "https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt", "https://raw.githubusercontent.com/badmojr/1Hosts/master/submit_here/exclude_for_all.txt", "https://raw.githubusercontent.com/badmojr/1Hosts/master/submit_here/include_for_Pro_Xtra_only.txt", "https://raw.githubusercontent.com/badmojr/1Hosts/master/submit_here/include_for_all.txt", "https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/everything-nl.txt", "https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/smart-tv-nl.txt", "https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/vaping-nl.txt", "https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/whatsapp-nl.txt", "https://raw.githubusercontent.com/blokadaorg/landing-github-pages/master/blocklists/ddgtrackerradar/standard/hosts.txt", "https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/BlacklistedDomains.txt", "https://raw.githubusercontent.com/craiu/mobiletrackers/master/list.txt", "https://raw.githubusercontent.com/craiu/mobiletrackers/master/windowslist.txt", "https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra_v6.txt", "https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy_v6.txt", "https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/update_v6.txt", "https://raw.githubusercontent.com/davidonzo/Threat-Intel/master/lists/latestdomains.txt", "https://raw.githubusercontent.com/davidonzo/Threat-Intel/master/lists/latestips.txt", "https://raw.githubusercontent.com/durablenapkin/block/master/luminati.txt", "https://raw.githubusercontent.com/durablenapkin/block/master/streaming.txt", "https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/hosts.txt", "https://raw.githubusercontent.com/finnish-easylist-addition/finnish-easylist-addition/master/Finland_adb.txt", "https://raw.githubusercontent.com/gioxx/xfiles/master/filtri.txt", "https://raw.githubusercontent.com/gioxx/xfiles/master/siteblock.txt", "https://raw.githubusercontent.com/heradhis/indonesianadblockrules/master/subscriptions/abpindo.txt", "https://raw.githubusercontent.com/infinitytec/blocklists/master/ads-and-trackers.txt", "https://raw.githubusercontent.com/infinitytec/blocklists/master/medicalpseudoscience.txt", "https://raw.githubusercontent.com/infinitytec/blocklists/master/scams-and-phishing.txt", "https://raw.githubusercontent.com/jakejarvis/ios-trackers/master/blocklist.txt", "https://raw.githubusercontent.com/lassekongo83/Frellwits-filter-lists/master/Frellwits-Swedish-Hosts-File.txt", "https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/ads-and-tracking-extended.txt", "https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/amp-hosts-extended.txt", "https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/facebook-extended.txt", "https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/hate-and-junk-extended.txt", "https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/tracking-aggressive-extended.txt", "https://raw.githubusercontent.com/marcusminus/Orthrus-BlockList/master/domains.txt", "https://raw.githubusercontent.com/matomo-org/referrer-spam-list/master/spammers.txt", "https://raw.githubusercontent.com/mitchellkrogza/Phishing.Database/master/phishing-IPs-ACTIVE.txt", "https://raw.githubusercontent.com/notracking/hosts-blocklists/master/dnscrypt-proxy/dnscrypt-proxy.blacklist.txt", "https://raw.githubusercontent.com/rimu/no-qanon/master/etc_hosts.txt", "https://raw.githubusercontent.com/scafroglia93/blocklists/master/blocklists-certego.txt", "https://raw.githubusercontent.com/stamparm/aux/master/maltrail-malware-domains.txt", "https://raw.githubusercontent.com/stamparm/ipsum/master/ipsum.txt", "https://rescure.me/covid.txt", "https://rescure.me/malware/clop.txt", "https://rescure.me/malware/ekans.txt", "https://rescure.me/malware/maze.txt", "https://rescure.me/malware/netwalker.txt", "https://rescure.me/malware/ryuk.txt", "https://rescure.me/malware/sodinokibi.txt", "https://rescure.me/malware/try2cry.txt", "https://rescure.me/malware/wastedlocker.txt", "https://rescure.me/rescure_blacklist.txt", "https://rescure.me/rescure_domain_blacklist.txt", "https://schakal.ru/hosts/alive_hosts.txt", "https://secure.fanboy.co.nz/fanboy-antifonts.txt", "https://secure.fanboy.co.nz/fanboy-indian.txt", "https://secure.fanboy.co.nz/r/fanboy-ultimate.txt", "https://tgc.cloud/downloads/hosts.alive.txt", "https://threatcrowd.org/feeds/domains.txt", "https://threatcrowd.org/feeds/ips.txt", "https://threatcrowd.org/inc/blacklist.txt", "https://www.binarydefense.com/banlist.txt", "https://www.blocklist.de/downloads/export-ips_all.txt", "https://www.botvrij.eu/data/misp.text_ip-dst.ADMIN.txt", "https://www.github.developerdan.com/hosts/lists/amp-hosts-extended.txt", "https://www.github.developerdan.com/hosts/lists/facebook-extended.txt", "https://www.github.developerdan.com/hosts/lists/hate-and-junk-extended.txt", "https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt", "https://www.joewein.net/dl/bl/dom-bl.txt", "https://www.spamhaus.org/drop/drop.txt", "https://www.spamhaus.org/drop/edrop.txt", "https://www.sunshine.it/blacklist.txt", "https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt", "https://www.threatsourcing.com/dnall-free.txt", "https://www.threatsourcing.com/ipall-free.txt", "https://zerodot1.gitlab.io/CoinBlockerLists/list_browser.txt", "https://zerodot1.gitlab.io/CoinBlockerLists/list_optional.txt", "https://zerodot1.gitlab.io/CoinBlockerLists/white_list.txt", "https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Hosts/GoodbyeAds.txt" ] ############ End Of Edits ################# # Open TLSv1 Adapter class MyAdapter(HTTPAdapter): def init_poolmanager(self, connections, maxsize, block=False): self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize, block=block, ssl_version=ssl.PROTOCOL_TLSv1) headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0'} s = requests.Session() s.mount(host, MyAdapter()) x = s.post(host + "/control/login", json.dumps({"name": userName, "password" : password}), headers=headers ) print(x.text) for u in urls: filterObj = json.dumps({'url':u, "name":u,"whitelist":False}) print(filterObj) x = s.post(host + "/control/filtering/add_url", data = filterObj, headers=headers) print(x.text) ```
jumpsmm7 commented 2 years ago

If anyone decides to use script for issue reproduction. use this with more URLs than the previous one I posted. It actually freezes Pi until python request shows connection closed from AGH memory lockup, it goes above 90%. memory usage. Even after reboot it rises up quickly for no reason.

htop SCRIPT

import requests
import ssl
import json
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager

# +++ A SCRIPT TO ADD MULTIPLE BLOCKLIST AT ONCE IN ADGUARD +++ #

### RUN SCRIPT ###
# sudo python3 bulkurls.py

### SET IP / USERNAME / PASSWORD USING *YOUR* ADGUARD CREDENTIALS ###
host = "http://your_pi_ip:80" 
# Your user name
userName = "Yours"
# Your password
password = "Yours"

# My blocklist (add your own: https://github.com/T145/black-mirror) 
urls = [
"https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt",
"https://adaway.org/hosts.txt",
"https://someonewhocares.org/hosts/zero/hosts",
"https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt",
"https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hosts",
"https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV-AGH.txt",
"https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt",
"https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblockplus&showintro=1&mimetype=plaintext",
"https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt",
"https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/adguard.txt",
"https://curben.gitlab.io/malware-filter/urlhaus-filter-agh-online.txt",
"https://paulgb.github.io/BarbBlock/blacklists/hosts-file.txt",
"https://abp.oisd.nl/",
"https://v.firebog.net/hosts/Easyprivacy.txt",
"https://v.firebog.net/hosts/Prigent-Ads.txt",
"https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt",
"https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts",
"https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt",
"https://v.firebog.net/hosts/static/w3kbl.txt",
"https://v.firebog.net/hosts/AdguardDNS.txt",
"https://v.firebog.net/hosts/Admiral.txt",
"https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt",
"https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt",
"https://v.firebog.net/hosts/Easylist.txt",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext",
"https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts",
"https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/TOP_EU_US_Ads_Trackers_HOST",
"https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt",
"https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt",
"https://v.firebog.net/hosts/Prigent-Crypto.txt",
"https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt",
"https://phishing.army/download/phishing_army_blocklist_extended.txt",
"https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt",
"https://v.firebog.net/hosts/Shalla-mal.txt",
"https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts",
"https://urlhaus.abuse.ch/downloads/hostfile/",
"https://raw.githubusercontent.com/neodevpro/neodevhost/master/lite_host",
"https://raw.githubusercontent.com/neodevpro/neodevhost/master/lite_adblocker",
"https://perflyst.github.io/PiHoleBlocklist/AmazonFireTV.txt",
"https://perflyst.github.io/PiHoleBlocklist/android-tracking.txt",
"https://perflyst.github.io/PiHoleBlocklist/regex.list",
"https://perflyst.github.io/PiHoleBlocklist/SessionReplay.txt",
"https://blocklistproject.github.io/Lists/phishing.txt",
"https://blocklistproject.github.io/Lists/tracking.txt",
"https://raw.githubusercontent.com/ftpmorph/ftprivacy/master/blocklists/smartphone-ads-tracking.txt",
"https://raw.githubusercontent.com/CleanMachine1/AdlistTXTS/main/adservers.txt",
"https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt",
"https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_15_DnsFilter/filter.txt",
"https://easylist-downloads.adblockplus.org/easylist.txt",
"https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_11_Mobile/filter.txt",
"https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_3_Spyware/filter.txt",
"https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_2_English/filter.txt",
"https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/master/filters/filter_4_Social/filter.txt",
"https://easylist.to/easylist/easyprivacy.txt",
"https://easylist.to/easylist/fanboy-annoyance.txt",
"https://easylist.to/easylist/fanboy-social.txt",
"https://anti-ad.net/easylist.txt",
"https://hosts.anudeep.me/mirror/adservers.txt",
"https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/abp_light.txt",
"https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dbl_light.txt",
"https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/hosts_light.txt",
"https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dnsmasq_light.txt",
"https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/rpz_basic.rpz",
"https://raw.githubusercontent.com/smed79/blacklist/master/hosts.txt",
"https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt",
"https://raw.githubusercontent.com/azet12/KADhosts/master/KADhosts.txt",
"https://raw.githubusercontent.com/mitchellkrogza/Phishing.Database/master/phishing-domains-ACTIVE.txt",
"https://gitlab.com/quidsup/notrack-blocklists/-/raw/master/malware.list",
"https://gitlab.com/quidsup/notrack-blocklists/-/raw/master/trackers.list",
"https://raw.githubusercontent.com/yous/YousList/master/hosts.txt",
"https://zerodot1.gitlab.io/CoinBlockerLists/list.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/list_browser.txt",
"https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/domains",
"https://easylist-downloads.adblockplus.org/easyprivacy.txt",
"https://raw.githubusercontent.com/privacy-protection-tools/anti-AD/master/anti-ad-easylist.txt",
"https://www.i-dont-care-about-cookies.eu/abp/",
"https://hblock.molinero.dev/hosts",
"http://winhelp2002.mvps.org/hosts.txt",
"https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts",
"https://raw.githubusercontent.com/VeleSila/yhosts/master/hosts.txt",
"https://iplists.firehol.org/files/alienvault_reputation.ipset",
"https://orca.pet/notonmyshift/domains.txt",
"https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/firewall/spy.txt",
"https://raw.githubusercontent.com/d43m0nhLInt3r/socialblocklists/master/Windows/windowstelemetryblocklist.txt",
"https://raw.githubusercontent.com/d43m0nhLInt3r/socialblocklists/master/MobileAppAds/appadsblocklist.txt",
"https://raw.githubusercontent.com/d43m0nhLInt3r/socialblocklists/master/Android/androidblocklist.txt",
"https://raw.githubusercontent.com/twcau/AdblockRules/master/MurdochList",
"https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews/hosts",
"https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/gambling/hosts",
"https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Dead/hosts",
"https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts",
"https://raw.githubusercontent.com/mitchellkrogza/Badd-Boyz-Hosts/master/hosts",
"https://winhelp2002.mvps.org/hosts.txt",
"https://raw.githubusercontent.com/notracking/hosts-blocklists/master/adblock/adblock.txt",
"https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/unbound.conf",
"https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/hosts.txt",
"https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/adblock.txt",
"https://raw.githubusercontent.com/badmojr/1Hosts/master/mini/hosts.win",
"https://blocklistproject.github.io/Lists/adguard/ransomware-ags.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt",
"https://curben.gitlab.io/malware-filter/urlhaus-filter.txt",
"https://abp.oisd.nl/basic/",
"https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareAdGuardHome.txt",
"https://curben.gitlab.io/malware-filter/urlhaus-filter-ag.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/resource-abuse.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt",
"https://raw.githubusercontent.com/easylist/easylist/master/easyprivacy/easyprivacy_general.txt",
"https://secure.fanboy.co.nz/fanboy-cookiemonster.txt",
"https://easylist-downloads.adblockplus.org/easyprivacy+easylist.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/annoyances.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters-2022.txt",
"https://secure.fanboy.co.nz/fanboy-annoyance.txt",
"https://easylist.to/easylist/easylist.txt",
"https://curbengh.github.io/malware-filter/urlhaus-filter.txt",
"https://curben.gitlab.io/malware-filter/urlhaus-filter-vivaldi.txt",
"https://curben.gitlab.io/malware-filter/urlhaus-filter-domains.txt",
"https://curben.gitlab.io/malware-filter/urlhaus-filter-agh.txt",
"https://curben.gitlab.io/malware-filter/urlhaus-filter-hosts.txt",
"https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_adservers.txt",
"https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_general_block.txt",
"https://raw.githubusercontent.com/easylist/easylist/master/easylist/easylist_adservers_popup.txt",
"https://raw.githubusercontent.com/easylist/easylist/master/easyprivacy/easyprivacy_trackingservers.txt",
"https://raw.githubusercontent.com/angad/webPriority/master/easylist/easylist/easylist_general_block.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/thirdparties/easylist-downloads.adblockplus.org/easylist.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/thirdparties/easylist-downloads.adblockplus.org/easyprivacy.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters-2021.txt",
"https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters-2020.txt",
"https://raw.githubusercontent.com/easylist/easylist/master/easyprivacy/easyprivacy_thirdparty.txt",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=adblock&showintro=0&mimetype=plaintext",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=littlesnitch&showintro=0&mimetype=plaintext",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=junkbuster&showintro=0&mimetype=plaintext",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=unbound&showintro=0&mimetype=plaintext",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=iptables&showintro=0&mimetype=plaintext",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=msfilter&showintro=0&mimetype=plaintext",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=operafilter&showintro=0&mimetype=plaintext",
"https://pgl.yoyo.org/adservers/serverlist.php?hostformat=machosts&showintro=0&mimetype=plaintext",
"https://raw.githubusercontent.com/BlackJack8/iOSAdblockList/master/Regular%20Hosts.txt",
"https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock.txt",
"https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock_lite.txt",
"https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock_plus.txt",
"https://raw.githubusercontent.com/uniartisan/adblock_list/master/adblock_privacy.txt",
"https://raw.githubusercontent.com/TonyRL/blocklist/master/hosts",
"https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt",
"https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt",
"https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt",
"https://raw.githubusercontent.com/ShadowWhisperer/BlockLists/master/Lists/Tracking",
"https://raw.githubusercontent.com/ShadowWhisperer/BlockLists/master/Lists/Malware",
"https://raw.githubusercontent.com/hemiipatu/PiHoleBlocklists/master/blocklists/advertisement.txt",
"https://raw.githubusercontent.com/hemiipatu/PiHoleBlocklists/master/blocklists/phishing.txt",
"https://raw.githubusercontent.com/DandelionSprout/adfilt/master/AdGuard%20Home%20Compilation%20List/AdGuardHomeCompilationList.txt",
"https://raw.githubusercontent.com/ftpmorph/ftpihole/master/blocklists/smartphone-ads-tracking.txt",
"https://raw.githubusercontent.com/mhxion/pornaway/master/hosts/porn_ads.txt",
"https://raw.githubusercontent.com/privacy-protection-tools/anti-AD/master/anti-ad-domains.txt",
"https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Hosts/GoodbyeAds.txt",
"http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-CIF-latest.txt",
"https://anti-ad.net/domains.txt",
"https://block.energized.pro/assets/sources/filter/better-fyi-trackers.txt",
"https://block.energized.pro/assets/sources/filter/sinfonietta-porn.txt",
"https://block.energized.pro/extensions/xtreme/formats/domains.txt",
"https://blocklistproject.github.io/Lists/alt-version/everything-nl.txt",
"https://blocklistproject.github.io/Lists/alt-version/smart-tv-nl.txt",
"https://blocklistproject.github.io/Lists/alt-version/vaping-nl.txt",
"https://blocklistproject.github.io/Lists/alt-version/whatsapp-nl.txt",
"https://blokada.org/blocklists/ddgtrackerradar/standard/hosts.txt",
"https://cinsscore.com/list/ci-badguys.txt",
"https://data.netlab.360.com/feeds/dga/dga.txt",
"https://dataplane.org/sipinvitation.txt",
"https://dataplane.org/sipquery.txt",
"https://dataplane.org/sipregistration.txt",
"https://dataplane.org/sshpwauth.txt",
"https://dataplane.org/vncrfb.txt",
"https://fanboy.co.nz/fanboy-antifonts.txt",
"https://fanboy.co.nz/fanboy-cookiemonster.txt",
"https://fanboy.co.nz/fanboy-indian.txt",
"https://fanboy.co.nz/r/fanboy-ultimate.txt",
"https://feeds.dshield.org/block.txt",
"https://filter.futa.gg/hosts.txt",
"https://filtri-dns.ga/filtri.txt",
"https://getblackbird.net/blacklist/hosts.txt",
"https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/Extreme%20lists/The_Quantum_Extreme_Youtube_Ad-List.txt",
"https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Abuse-List.txt",
"https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Privacy-list.txt",
"https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Simply-ads-list.txt",
"https://gitlab.com/The_Quantum_Alpha/the-quantum-ad-list/-/raw/master/Individual%20lists/The_Quantum_Youtube-Ads-List.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/list.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/list_optional.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/-/raw/master/white_list.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_browser.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/list_optional.txt",
"https://gitlab.com/ZeroDot1/CoinBlockerLists/raw/master/white_list.txt",
"https://gitlab.com/quidsup/ntrk-tracker-radar/-/raw/master/ddg_tracker_radar_confirmed.txt",
"https://hostfiles.frogeye.fr/firstparty-trackers.txt",
"https://hostfiles.frogeye.fr/multiparty-trackers.txt",
"https://hosts.anudeep.me/mirror/CoinMiner.txt",
"https://hosts.anudeep.me/mirror/facebook.txt",
"https://hosts.anudeep.me/mirror/miscellaneous/false-positives.txt",
"https://kriskintel.com/feeds/ktip_covid_domains.txt",
"https://kriskintel.com/feeds/ktip_malicious_Ips.txt",
"https://kriskintel.com/feeds/ktip_malicious_domains.txt",
"https://kriskintel.com/feeds/ktip_ransomware_feeds.txt",
"https://lists.blocklist.de/lists/all.txt",
"https://malsilo.gitlab.io/feeds/dumps/domain_list.txt",
"https://mirai.security.gives/data/ip_list.txt",
"https://myip.ms/files/bots/live_webcrawlers.txt",
"https://openphish.com/feed.txt",
"https://orca.pet/notonmyshift/hosts.txt",
"https://osint.digitalside.it/Threat-Intel/lists/latestips.txt",
"https://public-dns.info/nameservers.txt",
"https://raw.githubusercontent.com/4skinSkywalker/Anti-Porn-HOSTS-File/master/HOSTS.txt",
"https://raw.githubusercontent.com/AdguardTeam/cname-trackers/master/combined_disguised_trackers_justdomains.txt",
"https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/activation.txt",
"https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/adservers-and-trackers.txt",
"https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/fake-domains.txt",
"https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/search-blacklist.txt",
"https://raw.githubusercontent.com/DRSDavidSoft/additional-hosts/master/domains/blacklist/unwanted-iranian.txt",
"https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareDomains.txt",
"https://raw.githubusercontent.com/DandelionSprout/adfilt/master/NorwegianList.txt",
"https://raw.githubusercontent.com/EnergizedProtection/unblock/master/basic/formats/domains.txt",
"https://raw.githubusercontent.com/Ewpratten/youtube_ad_blocklist/master/blocklist.txt",
"https://raw.githubusercontent.com/FiltersHeroes/KADhosts/master/KADomains.txt",
"https://raw.githubusercontent.com/FutaGuard/FutaFilter/master/hosts.txt",
"https://raw.githubusercontent.com/List-KR/List-KR/master/filter.txt",
"https://raw.githubusercontent.com/MajkiIT/polish-ads-filter/master/polish-pihole-filters/hole_cert_pl.txt",
"https://raw.githubusercontent.com/MajkiIT/polish-ads-filter/master/polish-pihole-filters/hostfile.txt",
"https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SessionReplay.txt",
"https://raw.githubusercontent.com/TheAntiSocialEngineer/AntiSocial-BlockList-UK-Community/main/UK-Community.txt",
"https://raw.githubusercontent.com/allendema/noplaylist/main/NoPlayList.txt",
"https://raw.githubusercontent.com/anudeepND/blacklist/master/CoinMiner.txt",
"https://raw.githubusercontent.com/anudeepND/blacklist/master/facebook.txt",
"https://raw.githubusercontent.com/anudeepND/blacklist/master/miscellaneous/false-positives.txt",
"https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt",
"https://raw.githubusercontent.com/badmojr/1Hosts/master/submit_here/exclude_for_all.txt",
"https://raw.githubusercontent.com/badmojr/1Hosts/master/submit_here/include_for_Pro_Xtra_only.txt",
"https://raw.githubusercontent.com/badmojr/1Hosts/master/submit_here/include_for_all.txt",
"https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/everything-nl.txt",
"https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/smart-tv-nl.txt",
"https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/vaping-nl.txt",
"https://raw.githubusercontent.com/blocklistproject/Lists/master/alt-version/whatsapp-nl.txt",
"https://raw.githubusercontent.com/blokadaorg/landing-github-pages/master/blocklists/ddgtrackerradar/standard/hosts.txt",
"https://raw.githubusercontent.com/bongochong/CombinedPrivacyBlockLists/master/NoFormatting/BlacklistedDomains.txt",
"https://raw.githubusercontent.com/craiu/mobiletrackers/master/list.txt",
"https://raw.githubusercontent.com/craiu/mobiletrackers/master/windowslist.txt",
"https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra_v6.txt",
"https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy_v6.txt",
"https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/update_v6.txt",
"https://raw.githubusercontent.com/davidonzo/Threat-Intel/master/lists/latestdomains.txt",
"https://raw.githubusercontent.com/davidonzo/Threat-Intel/master/lists/latestips.txt",
"https://raw.githubusercontent.com/durablenapkin/block/master/luminati.txt",
"https://raw.githubusercontent.com/durablenapkin/block/master/streaming.txt",
"https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/hosts.txt",
"https://raw.githubusercontent.com/finnish-easylist-addition/finnish-easylist-addition/master/Finland_adb.txt",
"https://raw.githubusercontent.com/gioxx/xfiles/master/filtri.txt",
"https://raw.githubusercontent.com/gioxx/xfiles/master/siteblock.txt",
"https://raw.githubusercontent.com/heradhis/indonesianadblockrules/master/subscriptions/abpindo.txt",
"https://raw.githubusercontent.com/infinitytec/blocklists/master/ads-and-trackers.txt",
"https://raw.githubusercontent.com/infinitytec/blocklists/master/medicalpseudoscience.txt",
"https://raw.githubusercontent.com/infinitytec/blocklists/master/scams-and-phishing.txt",
"https://raw.githubusercontent.com/jakejarvis/ios-trackers/master/blocklist.txt",
"https://raw.githubusercontent.com/lassekongo83/Frellwits-filter-lists/master/Frellwits-Swedish-Hosts-File.txt",
"https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/ads-and-tracking-extended.txt",
"https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/amp-hosts-extended.txt",
"https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/facebook-extended.txt",
"https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/hate-and-junk-extended.txt",
"https://raw.githubusercontent.com/lightswitch05/hosts/master/docs/lists/tracking-aggressive-extended.txt",
"https://raw.githubusercontent.com/marcusminus/Orthrus-BlockList/master/domains.txt",
"https://raw.githubusercontent.com/matomo-org/referrer-spam-list/master/spammers.txt",
"https://raw.githubusercontent.com/mitchellkrogza/Phishing.Database/master/phishing-IPs-ACTIVE.txt",
"https://raw.githubusercontent.com/notracking/hosts-blocklists/master/dnscrypt-proxy/dnscrypt-proxy.blacklist.txt",
"https://raw.githubusercontent.com/rimu/no-qanon/master/etc_hosts.txt",
"https://raw.githubusercontent.com/scafroglia93/blocklists/master/blocklists-certego.txt",
"https://raw.githubusercontent.com/stamparm/aux/master/maltrail-malware-domains.txt",
"https://raw.githubusercontent.com/stamparm/ipsum/master/ipsum.txt",
"https://rescure.me/covid.txt",
"https://rescure.me/malware/clop.txt",
"https://rescure.me/malware/ekans.txt",
"https://rescure.me/malware/maze.txt",
"https://rescure.me/malware/netwalker.txt",
"https://rescure.me/malware/ryuk.txt",
"https://rescure.me/malware/sodinokibi.txt",
"https://rescure.me/malware/try2cry.txt",
"https://rescure.me/malware/wastedlocker.txt",
"https://rescure.me/rescure_blacklist.txt",
"https://rescure.me/rescure_domain_blacklist.txt",
"https://schakal.ru/hosts/alive_hosts.txt",
"https://secure.fanboy.co.nz/fanboy-antifonts.txt",
"https://secure.fanboy.co.nz/fanboy-indian.txt",
"https://secure.fanboy.co.nz/r/fanboy-ultimate.txt",
"https://tgc.cloud/downloads/hosts.alive.txt",
"https://threatcrowd.org/feeds/domains.txt",
"https://threatcrowd.org/feeds/ips.txt",
"https://threatcrowd.org/inc/blacklist.txt",
"https://www.binarydefense.com/banlist.txt",
"https://www.blocklist.de/downloads/export-ips_all.txt",
"https://www.botvrij.eu/data/misp.text_ip-dst.ADMIN.txt",
"https://www.github.developerdan.com/hosts/lists/amp-hosts-extended.txt",
"https://www.github.developerdan.com/hosts/lists/facebook-extended.txt",
"https://www.github.developerdan.com/hosts/lists/hate-and-junk-extended.txt",
"https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt",
"https://www.joewein.net/dl/bl/dom-bl.txt",
"https://www.spamhaus.org/drop/drop.txt",
"https://www.spamhaus.org/drop/edrop.txt",
"https://www.sunshine.it/blacklist.txt",
"https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt",
"https://www.threatsourcing.com/dnall-free.txt",
"https://www.threatsourcing.com/ipall-free.txt",
"https://zerodot1.gitlab.io/CoinBlockerLists/list_browser.txt",
"https://zerodot1.gitlab.io/CoinBlockerLists/list_optional.txt",
"https://zerodot1.gitlab.io/CoinBlockerLists/white_list.txt",
"https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Hosts/GoodbyeAds.txt"
]

############ End Of Edits #################

# Open TLSv1 Adapter
class MyAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       ssl_version=ssl.PROTOCOL_TLSv1)

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0'}     

s = requests.Session()
s.mount(host, MyAdapter())
x = s.post(host + "/control/login", json.dumps({"name": userName, "password" : password}), headers=headers )
print(x.text)

for u in urls:
  filterObj = json.dumps({'url':u, "name":u,"whitelist":False})
  print(filterObj)
  x = s.post(host + "/control/filtering/add_url", data = filterObj, headers=headers)
  print(x.text)

Yea I imagined if you loaded that many list into memory with less than 1gb of ram space and hardly any swap space, then you would be suffering from some sort of memory issue.

trinib commented 2 years ago

Yea I imagined if you loaded that many list into memory with less than 1gb of ram space and hardly any swap space, then you would be suffering from some sort of memory issue.

It is not pi's entire fault. AGH memory do not even get freed when python script stops. I think their problem is not they cannot fix it, but about finding it.

trinib commented 2 years ago

@jumpsmm7 i tried it on A VPS and the python script runs without freezing but AGH memory do not drop image

after reboot image

trinib commented 2 years ago

@jumpsmm7 @SiNONiMiTY @EugeneOne1 After 7 hours just doing nothing IT STILL SHOWS HIGH MEMORY on vps linux 1gb ram. And AGH sticks on "DNS server starting up"

image

trinib commented 2 years ago

hmmm I added blocklist with fewer URLs(still alot) and after reboot it remains around 35% ram and AGH starts up and it's stable. So I guess it is AGH blocklist url part cause memory issue🤷‍♂️

image

jumpsmm7 commented 2 years ago

@trinib do me a favor, once you get the memory usage high, try using a different program that also demands alot of memory, then tell me if adguardhome starts releasing memory. From my understanding, this is a behavior of Go itself. GO does not free memory until it is taken back.

trinib commented 2 years ago

@jumpsmm7 It does goes down running memory stress test but it rises right back up after stopping it.

https://user-images.githubusercontent.com/18756975/169733842-fb08d4e4-44c7-4073-ae6d-3fe92c8f1752.mp4

trinib commented 2 years ago

I would like to add, from a clean install I used backup yaml file with all the blocklist URLs links instead of python script, rebooted pi and memory still went high. I deleted all the URLs in yaml without a clean install, rebooted pi and AGH memory was very low again.

Why memory gets & stays so high when processing lots of blocklist filters or logs filling up from days/weeks of build up. 🤔🤔🤔

jumpsmm7 commented 2 years ago

@trinib how much of the memory consumption is cache memory and how much of it is actually being used?

trinib commented 2 years ago

@trinib how much of the memory consumption is cache memory and how much of it is actually being used?

@jumpsmm7 Use command free ?

trinib commented 2 years ago

image

jumpsmm7 commented 2 years ago

@trinib

Here is mine running on a router that uses busybox.

image

and the arch

image

AdGuardHome at the bottom of my htop

image

trinib commented 2 years ago

@trinib

Here is mine running on a router that uses busybox.

image

and the arch

image

AdGuardHome at the bottom of my htop

image

I do not understand . what this suppose to mean? Did you do something to get lower memory ? did you ran script with all the URLs ? Cause correct me if i am wrong i see 1gb of ram and 829 in use and AGH is 26% ? please tell me you ran script with those results . if so what tweak you did ?

ainar-g commented 2 years ago

@trinib, thank you for the script!

@EugeneOne1, please investigate using the script.

jumpsmm7 commented 2 years ago

@trinib Here is me using your script. (this is durring and after.) image image image image image image image image image

trinib commented 2 years ago

@jumpsmm7 do you have any thing change in AGH ? You should check out sharex.

jumpsmm7 commented 2 years ago

@jumpsmm7 do you have any thing change in AGH ? You should check out sharex.

Nothing out the ordinary other than a large memory usage from a high amount of filters having been loaded into memory from adguard home

jumpsmm7 commented 2 years ago

Adguardhome seems to struggle to know how to commit memory in the presence of this many filters having been added.

trinib commented 2 years ago

Adguardhome seems to struggle to know how to commit memory in the presence of this many filters having been added.

hopefully it can help point out quicker by adding many urls to see where AGH has memory issue

cxw620 commented 2 years ago

Well it seems I meet with the same problem. I have about 30 filters included some official ones. My 2GB-MEM VPS crashed when updating them, though my 4GB-MEM VPS with same configuration works well. And I also enable optimistic caching. Is there any further solutions now?

jkellerer commented 2 years ago

Could it be related to GC in go taking too much memory on small devices when not limited? Had AGH pile up to 1.5G on a 4G VM until I limited it to 384m (using docker) now it runs fine at 350m per average.

Go itself is meant to introduce a soft limit via ENV GOMEMLIMIT in 1.19, until this is released ulimit (e.g. via docker) can potentially help.

ainar-g commented 1 year ago

@timkgh, is this still an issue? Because we could never really reproduce this.

jumpsmm7 commented 1 year ago

The only thing I can say when using an embedded or small device with a small amount of memory, you definitely need a swap and a good regard for how many filters/and or lists you use. If you expect to have alot of filters, then expect to have a swap incase there is the opportunity for an out of memory occurrence. If on a VM, you need to allocate an adequate amount of memory if you expect to be loading alot of filters. All of these types of blocklists are loading into memory. The more filters or block domains, the more caution needs to be taken in regards to memory needed. The memory doesn't fill indefinitely, however it may appear that it does when using memory stores that are inadequate in size for the amount of lists being used. This is not something that can be controlled; however, it relies on the user's good judgement and understanding of the system from which they operate.

timkgh commented 1 year ago

@timkgh, is this still an issue? Because we could never really reproduce this.

I turned off optimistic caching because of other issues (e.g. Chrome can cache stale DNS responses for 1 min which is annoying) and I haven't seen any memory problems.

We can probably close this issue.