hagezi
commented
6 months ago Great work!
I have optimised the size of the normal lists so that they can be used by all AdBlockers without losing effectiveness. The goal is <200000 "compressed domains". To achieve this, I have "moved" many malicious domains to the TIF medium and TIF, also to virtually eliminate duplicates between the normal and TIF lists. This does not mean that the normal lists no longer contain any TIF domains, but they do contain mainly Ads, Tracker, Telemetry and Analytics domains.
Some TIF sources vary greatly in size, sometimes from update to update, which makes it impossible to keep the normal lists stable in size. Furthermore, some domains are already dead before they even land on the list and thus inflate the lists unnecessarily.
You have seen this with your scam list, for months there were < 10000 domains and suddenly there were > 30000 domains.
Everything has been reorganised so that the user can assemble what they need from the available lists in a modular way. I made the switch after the TIF list was available in almost all DNS cloud services. For AdBlockers that can't cope with the large TIF (e.g. AdGuard iOS), I created the smaller TIF medium so that every user is able to combine the normal lists with a TIF version.
Your Light version has an acceptable size today, so I could easily add it from Normal to Ultimate without exceeding the target of 200000 domains. But, will it stay that way? Probably not, this list will grow too, right?
I know that NextDNS users have the disadvantage of not being able to use the TIF version. Which for me would be the only reason to include your Light Scam version in the normal lists and not just in the TIFs. With all other services and self-hosting, the normal lists can be combined with a TIF version of your choice. But if I start doing that, we'll soon have the huge lists from a few months ago again because there will be more requests as to why this and that source is not also included. NextDNS doesn't want to include the TIF because they have their "own feeds" and, according to them, everything is covered. If that's the case, it shouldn't be a problem. I know it's not like that, but I don't organise my lists according to a single service. Since the OISD also contains a lot of "TIF domains", an integration there might be possible @sjhgvr? Then NextDNS users could also benefit from your Light version by using the OISD.
I will add your Light to the TIF medium, the TIF will still contain the full version. This gives the TIF medium an acceptable size again, it has been properly inflated by the full version of your scam list.
jarelllama
sjhgvr
Hi @hagezi I recently finished adding the last bit of features to my scam blocklist. I still want to wait a week of automated builds to see if there are any issues before I make an announcement on my repo. But in the mean time, I introduced a light version of my blocklist seeing how adding the now 30,000 domains from the original list would be impractical for most of your lists.
The light version currently has 2100 domains and I was hoping you could test it in your Ultimate list?
A quick rundown on the new features:
The light list, while a lot smaller than the full list, still meets my goal of collating scam domains as close to their registration date as possible. Hopefully this makes it a practical blocklist to include. I would love to hear your thoughts!
More about the new features and the light version can be found in my readme.