Aggressive blocklists are ruining user experience

[Tomatoide]

Hello, it seems like every few days a major website is broken and I think the whitelisting approach without addressing the main issue of detecting and removing the aggressive blocklists is counterproductive, so right now there is issues with syncing microsoft apps, so I suggest comparing the domains in the whitelist to the blocklists and seeing which list has blocked these sites and removing it. thank you and keep up the good work.

ps. also there are a lot of duplicates in the blocklists due to some of them being aggregates of the others

[ragibkl]

Thanks for the report @Tomatoide .

Hello, it seems like every few days a major website is broken and I think the whitelisting approach without addressing the main issue of detecting and removing the aggressive blocklists is counterproductive, so right now there is issues with syncing microsoft apps,

I'm really sorry that some sites are broken.

Every time I push some code changes to this repo, a new version of the adblock dns gets built and deployed automatically. So you'll see changes in the behavour of the adblock dns every few days if I'm active on this project. Changes depends also on upstream blocklist sources, which might cause unintentional changes.

In this case, if an upstream source included some legitimate microsoft syncing servers in their list, it would be included in the final blocklist, which causes websites to break.

so I suggest comparing the domains in the whitelist to the blocklists and seeing which list has blocked these sites and removing it.

Actually, the compiler script already does this for us automatically. It downloads all the blocklist, compare it to the whitelist that we defined, and removes any matching entries. I'm actually in the process of improving the whitelisting mechanism by adding regex support, so that we can have more concise whitelisting.

ps. also there are a lot of duplicates in the blocklists due to some of them being aggregates of the others

I think we don't have to worry too much about duplicates. The compiler script combines them all and runs a quick deduplication before output of the final blocklist. I'll admit that it is wasteful to download duplicate sources, but it shouldn't cause problems for us.

I haven't personally audited the sources, but if you are familiar with them, and you know which ones are duplicates and can be removed, I'm open to update/remove them.

[ragibkl]

Hi @Tomatoide ,

so right now there is issues with syncing microsoft apps

I don't personally use Microsoft apps or any of their syncing features. So I won't notice any broken features. Can you help me debug this one?

I assume you are using the server in France. Try the following links.

http://51.158.99.7:8080/
http://[2001:bc8:600:705::1]:8080/

Can you see what sites are being blocked? Maybe just copy them and paste them here. You don't have to paste everything. Just the ones that might be related to microsoft.

[Tomatoide]

so I suggest comparing the domains in the whitelist to the blocklists and seeing which list has blocked these sites and removing it.

Actually, the compiler script already does this for us automatically. It downloads all the blocklist, compare it to the whitelist that we defined, and removes any matching entries. I'm actually in the process of improving the whitelisting mechanism by adding regex support, so that we can have more concise whitelisting.

sorry wasn't clear enough, what I meant is I think there is one or maybe two blocklists that is causing all these issues, so if it has the whitelisted entries (by comparing whitelist vs every upstream blocklist), we remove the culprit list(s).

[Tomatoide]

I did some investigation and I think this list is the issue https://blokada.org/blocklists/ddgtrackerradar/standard/hosts.txt

[ragibkl]

I did some investigation and I think this list is the issue https://blokada.org/blocklists/ddgtrackerradar/standard/hosts.txt

Thanks for the suggestion. I'll exclude this list for now. However, if ads start to appear again, we should probably add it back and add the required sites to whitelist.

BTW, did you manage to find which Microsoft sync domains were blocked for you? If you do, share them with me so that I can whitelist them anyway.

[Tomatoide]

Thank you, now the issues seem to be fixed. 👍 About the ddg list, I doubt removing it result in ads showing up, it's a relatively small list and doesn't really focus on ads, also I read advise against using it in domain filtering form due to reliability issues, which is clear. It blocked a microsoft login server which was like login.msa. something something