Closed Wizballs closed 3 months ago
hagezi
commented
4 months ago Hi @Wizballs, I can't say much about the RAM consumption with X entries, but the configuration is conclusive for me, I wouldn't do it any differently - very good.
With Small 128mb routers the fake bothers me, here a TIF mini instead of the fake would be the better option - but the list doesn't exist yet. I'm thinking about what a TIF mini could look like.
Wizballs
commented
4 months ago Ok great, thanks Hagezi for reviewing this! For the time being, I'll remove fake from small. But I'm also asking a few people to test on 128mb and 256mb routers to see if we can find out how many lines these can take comfortably. If we can recommend more lines for those routers, than great also. Cheers
hagezi
commented
4 months ago @Wizballs Great, if you have more information about the RAM, let me know and we'll see what we can do.
hagezi
commented
4 months ago Wizballs
commented
4 months ago @hagezi Oh wow that was fast, and also thank you for putting that together! I've added TIF-mini to the small list recommendations above. I know these combinations will be safe to run within the listed ram limits, so I think we'll go ahead and list these in the docs. It could take a while to get any feedback if bigger lists can be run per router hardware, but this is absolutely great to list these as is now.
lynxthecat
commented
3 months ago @Wizballs list suggestions look good to me.
It seems difficult to determine corresponding values for:
# Mininum number of lines of any individual downloaded blocklist part
min_blocklist_file_part_line_count=1
# Maximum size of any individual downloaded blocklist part
max_blocklist_file_part_size_KB=20000
# Maximum total size of combined, processed blocklist
max_blocklist_file_size_KB=30000
# Minimum number of good lines in final postprocessed blocklist
min_good_line_count=100000@lynxthecat From the main forum I get the feeling that new & less experienced users are just looking for recommendations on what blocklists and config to run so they don't have to trial & error. These should be quite safe and stable configs to start with. min_blocklist_file_part_line_count=1 should be fine for all of them - this is partly for individuals fine tuning?
Mini 64mb routers. Aim for <100k entries. TOTAL: 83k entries blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.mini.txt" min_blocklist_file_part_line_count=1 max_blocklist_file_part_size_KB=4000 max_blocklist_file_size_KB=4000 min_good_line_count=40000
Small 128mb routers. Aim for <300k entries. TOTAL: 249k entries blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif.mini.txt" min_blocklist_file_part_line_count=1 max_blocklist_file_part_size_KB=7000 max_blocklist_file_size_KB=10000 min_good_line_count=100000
Medium 256mb routers. Aim for <600k entries. TOTAL: 430k entries blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif.medium.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/popupads.txt" min_blocklist_file_part_line_count=1 max_blocklist_file_part_size_KB=10000 max_blocklist_file_size_KB=20000 min_good_line_count=200000
Large =>512mb routers. Aim for <1,200k entries. TOTAL: 913k entries blocklist_urls="https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/tif.txt https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/pro.txt" min_blocklist_file_part_line_count=1 max_blocklist_file_part_size_KB=30000 max_blocklist_file_size_KB=50000 min_good_line_count=400000
Wizballs
commented
3 months ago @hagezi Have a request if we may please :) And put me back in my box if I'm out of it... ;) We have some testing and feedback on the 256mb router list. We can comfortably take more lines on this size router.
Would it be possible to create a new tif-med list, or upgrade the existing tif-med list to:
Looping in @friendly-bits
hagezi
commented
3 months ago @Wizballs Currently the Popupads list are split between Pro or higher and TIF full, see here:
[!NOTE] In the combination of the Pro or higher and additionally the Threat Intelligence Feeds (TIF), all domains from the Pop-Up Ads list are included. This means that if you use the Pro or higher and also the TIF full, you no longer need to add this list separately.
With the TIF medium I have to stay below 200000 entries so that the list remains compatible for all adblockers. That was the reason why I created the TIF medium. With TIF lists it should also be noted that they can vary greatly in size, as the feeds sometimes vary in size from 10,000 to 30,000 domains. It is difficult to maintain the size. I'll see what I can manage if I include the malicious part of the popup ads in the medium ...
hagezi
commented
3 months ago @Wizballs I think I get that in, could still optimize something in the TIF medium so that the not Pro part of the popupads list fits into the TIF medium. I'll end up with around 185000 domains for the TIF medium and 165000 domains for the Pro. There will still be a few overlaps between Pro and TIF medium, but this cannot be avoided, as I would have to completely remove the TIF medium domains from the Pro, which would make the Pro incomplete for users who cannot use a TIF version (e.g. NextDNS).
I will push the customized TIF medium right away. There will be a few more shifts in the next release, which will be available at around 18:00 UTC.
hagezi
commented
3 months ago @Wizballs made new TIF medium available in https://github.com/hagezi/dns-blocklists/commit/0f8109fcda077a82be433822871cd5fc0a274e04
Wizballs
commented
3 months ago Hagezi thankyou muchly again. I know it's not a perfect science, but we are all just trying to block as much internet ads/crap as possible. In my mind this configuration makes more sense now, as adb-lean 'large' users (maybe others) won't be wondering why popupads aren't included as they were in 'medium'.
BTW do you have a preferred download location/mirror ie something you are counting downloads etc? At them moment we are just using https://raw.githubusercontent.com/hagezi/dns-blocklists/main/dnsmasq/.... but can easily change to anything else preferred.
hagezi
commented
3 months ago No problem, I'm glad to be able to help if I can.
No, the downloads are not counted. With the Github RAW link you are on the right track.
Wizballs
commented
3 months ago @hagezi looks like the other two adblocking solutions have added or requested your pro list: https://forum.openwrt.org/t/adblock-fast-ad-blocking-service-for-dnsmasq-smartdns-and-unbound/170530/274?u=wizballs https://forum.openwrt.org/t/adblock-support-thread/507/2729?u=wizballs
Always nice to know your work is being used. But thanks again for the above. I'll close this out now, our recommended lists seem to be working great.
Wizballs
commented
3 months ago complete
Hi Hagezi, your input on this would be great if possible. I want to determine the best recommended Hagezi combination lists for different ram configurations.
@lynxthecat What do you think - any adjustments to the linecount limits etc?
So an arbitrary start would be as follows, and I'm really on guessing on these. The adblock-lean default aim is to have maximum protection against ads/scams/telemetry/tracking/whatever while breaking as few websites as possible.
Also, Is there preference to running either or both: Thread intelligence feeds and/or TIF-IPS? (TIF-IPS is 219k entries)
Mini 64mb routers. Aim for <100k entries. Multi PRO mini: 83k entries TOTAL: 83k entries
Small 128mb routers. Aim for <300k entries Multi PRO: 160k entries Threat Intelligence Feed - Mini version: 89k entries TOTAL: 249k entries
Medium 256mb routers. Aim for <600k entries Multi PRO: 164k entries Threat Intelligence Feeds - Medium version: 183k entries TOTAL: 347k entries
Large 512mb routers. Aim for <1,200k entries Multi PRO: 160k entries Threat Intelligence Feeds: 753k entries TOTAL: 913k entries