search

ragibkl / adblock-dns-server

Adblock DNS Server powered by Bancuh DNS and dnsdist-acme
https://bancuh.com/
MIT License
68 stars 14 forks source link

Add Popup Scam Filter #41

Closed TechComputerServiceDunkerque closed 4 years ago

TechComputerServiceDunkerque commented 4 years ago

Here are different lists to block scam popups, since a while new popups have appeared and Bancuh doesn't filter them, so I allow myself to give you different filters for the dns :)

Merci

ragibkl commented 4 years ago

Hi @TechComputerServiceDunkerque ,

I've added these sources and deployed them to live servers.

Thanks for the suggestion.

ragibkl commented 4 years ago

I think, let's keep this ticket open for a bit. Feel free to close it after few days of testing.

TechComputerServiceDunkerque commented 4 years ago

Is it possible to add anti-tracking filters:

https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt https://www.github.developerdan.com/hosts/lists/tracking-aggressive-extended.txt https://raw.githubusercontent.com/NanoMeow/UltimateMirror/master/archive/hphosts.txt

Filter Ublock Hosts

https://raw.githubusercontent.com/mtxadmin/ublock/master/hosts.txt https://raw.githubusercontent.com/ReddestDream/reddestdream.github.io/master/Projects/MinimalHosts/etc/MinimalHostsBlocker/minimalhosts

Filter Mobile Android/Iphone

https://hostsmobile.oisd.nl https://raw.githubusercontent.com/DataMaster-2501/DataMaster-Android-AdBlock-Hosts/master/hosts

this one is very complete and well organized I find

https://raw.githubusercontent.com/kowith337/PersonalFilterListCollection/master/hosts/hosts_leftover.txt

I've checked that everything works before, can you add them and test them. However, I don't remember the ones I provided you with in the past. So there may be duplicates. Otherwise the other filters I gave you before will work fine.

I hope you are well, in France we are confined until May 11th because of COVID-19.

ragibkl commented 4 years ago

Hi @TechComputerServiceDunkerque ,

If you are interested, I configure the upstream block list at here. I will compare the list, and remove the duplicates.

I am fine to add the filters you suggested. I'll remove duplicates if any. If something breaks, we can always undo them. As always, let's test it together and report back here if anything breaks.

I hope you are well, in France we are confined until May 11th because of COVID-19.

I hope you are well also, both mental and physical. This confinement is also making people feel isolated. We also have confinement here in Malaysia, until 28th April. But it might be extended if the virus is still spreading. Stay safe and stay healthy.

ragibkl commented 4 years ago

Status

TechComputerServiceDunkerque commented 4 years ago

Thank you yes, let's keep testing the hosts, see if everything is ok. COVID-19 is very hard in France, the French don't respect the laws at all and go out of their house... Otherwise for me, everything is fine, it's just complicated to be away from family, friends and outings are also missing, it's nice weather at the moment on the side of Dunkirk in the North of France moreover, so it's quite hard.

Take care of yourself too :)

TechComputerServiceDunkerque commented 4 years ago

is it possible to block: https://syndication.optimizesrv.com/splash.php?type=8&idzone=1023 (every time I go to a site, it opens this page)

TechComputerServiceDunkerque commented 4 years ago

Small problem, Microsoft's servers for windows and xbox are blocked, impossible to make an update.Other problems, I can no longer read my facebook messages on messenger. Otherwise everything works fine and the dns is faster.

ragibkl commented 4 years ago

is it possible to block: https://syndication.optimizesrv.com/splash.php?type=8&idzone=1023 (every time I go to a site, it opens this page)

I added this just now.

Small problem, Microsoft's servers for windows and xbox are blocked, impossible to make an update.Other problems, I can no longer read my facebook messages on messenger. Otherwise everything works fine and the dns is faster.

In the past, we had some issues with MS products and FB messenger. But I believe we fixed it by whitelisting all the required domains. pihole whitelist

I'm not sure what domains I need to whitelist. I don't have any MS products, and I don't use FB anymore, so it's a bit difficult to test. If you know what to whitelist, let me know.

I saw some domains like facebook.com.gateway.huawei.net being blocked. If you also happen to own a Huawei phone, I suspect you are affected because of that.

Otherwise, we can just revert the changes.

TechComputerServiceDunkerque commented 4 years ago

I'm going to test on Microsoft, but messenger is working again, I have a meizu phone. I think the pihole white list can be used to solve the problem. I test, I tell you what, otherwise I got feedback from my friends who have your dns, they are satisfied since you added the filters. They're almost out of advertising, popup.

ragibkl commented 4 years ago

Since this morning, I added 2 whitelists:

  1. https://raw.githubusercontent.com/notracking/hosts-blocklists-scripts/master/hostnames.whitelist.txt
  2. https://github.com/silv3rr/pihole-scripts/blob/master/pihole-whitelist.txt

no 2 also includes all the whitelist in that pihole whitelist. I might have missed some domains last time.

I'm not sure if this recent change made messenger function again. Do report back any testing results.

TechComputerServiceDunkerque commented 4 years ago

Messenger continues to malfunction, I have to be in 4g to receive messages on Android, I'll look tomorrow in the hosts to see what's wrong.

ragibkl commented 4 years ago

I added some whitelist for FB here. I hope that works.

TechComputerServiceDunkerque commented 4 years ago

Thanks I'm going to test, so I just found a hosts file I was using when I root my android mobile. I share it with you, it blocks 99.9 ℅ ads, spam. If it was possible to add it, we would have peace with some ads that persist.

https://gist.github.com/sgeto/36bd4b93fc23db9936be06fa8c6a5dc3

ragibkl commented 4 years ago

https://gist.github.com/sgeto/36bd4b93fc23db9936be06fa8c6a5dc3

I think I can add that in. However, let's wait until after we resolve the fb messenger and ms updates issue.

TechComputerServiceDunkerque commented 4 years ago

Hello ragibkl,

I'm busy working on new lists for your dns, I've noticed some anomalies with some lists, I'm currently busy doing a lot of tests with these lists, they will block several things (Mobile Advertisements, Spam, porn etc...), I just tested Messenger with these lists, for the moment I don't have any block on video or voice calls. I wanted to know also, which DNS you use behind your server configuration (Google DNS or others ???) because if you do, I have a good dns to propose you to put on your machine in the background (I don't know much about pihole server config). I'll give you the list after many tests, you'll just have to make a backup of the old one and delete it and test mine. Being confined, I have all my time to help you and improve the good work you already did :)

Translated with www.DeepL.com/Translator (free version)

TechComputerServiceDunkerque commented 4 years ago

I spent my day testing with these lists there and I admit that the navigation is more fluid, much more ads blocked, however, I could not test with the windows and xbox updates because the list must be added on the dns. I tested with nano adblocker by disabling all adblock filters in the nano list, I installed only lists in hosts. No problem with Messenger and calls for the moment on android (I tested with Blokada with all filters), snapchat, Tiktok works too, however it is impossible to remove facebook ads (we'll find a solution one day who knows (^^). But otherwise, your users should be happy with the blocks (scam, phishing, ads, trackers.

TechComputerServiceDunkerque commented 4 years ago

Here are the lists:

Android/IOS:

https://gist.githubusercontent.com/sgeto/36bd4b93fc23db9936be06fa8c6a5dc3/raw/6ec2966dd7960f5bb58f29db4f8aed25e0f92e8f/lphosts (67 235 blocked)

Pornoblocking:

ADS Complete

Filter DNS (Blah DNS, Warden DNS)

Privacy

Trackers, spam, scam, phishing, malware)

I hope you will like these lists, they are complete, some sites may be blocked by mistake, but as I know you already have a white list, I think it's useless to remove it. Let me know if everything's okay with you?

if that's possible, how can we create an easylist? Because some ads get stuck with this one. Thanks

https://easylist-downloads.adblockplus.org/liste_fr.txt

I tested the filters with blokada and the next dns: ns0.fdn.fr : 80.67.169.12 ou 2001:910:800::12 ns1.fdn.fr : 80.67.169.40 ou 2001:910:800::40

ragibkl commented 4 years ago

  1. I wanted to know also, which DNS you use behind your server configuration (Google DNS or others ???) because if you do, I have a good dns to propose you to put on your machine in the background (I don't know much about pihole server config).

I am currently using google DNS (8.8.8.8, 8.8.4.4). Google does not do any adblocking, it is just a way to get regular dns service working.

I have plans to maybe remove this, and do dns resolution against the root name servers. I worry that Google DNS might have performance rate limit on my servers.

We can try with Next DNS, but I prefer we control the adblocking on our side, as we are doing now. That makes it easier to whitelist more reliably.

By the way, I am not using pihole server. I am using Bind on Linux. :smile:

  1. I saw the list that you provided, it looks like a lot. What do you want me to do with your list?: a. To remove my existing list and replace with your suggestions? b. To add/combine your suggested list with my current ones?

I think it is better to do b.. Other users might depend on the current set of lists that I'm using. It is better to block more ads, then to release some existing ones. Let me know what you think.

  1. I hope you will like these lists, they are complete, some sites may be blocked by mistake, but as I know you already have a white list, I think it's useless to remove it. Let me know if everything's okay with you?

I think it is worth to try this list. Yes, sometimes upstream list providers will mistakenly block legitimate sites. We have to rely on whitelist to unblock them. My existing whitelist will continue to work, but we just need to keep adding more sites to the whitelist as we go.

  1. Did you have enough time to test FB messenger? Everything working fine? I think it is better if we whitelist FB messenger properly, so that we have less problems when we add more adblock list.
ragibkl commented 4 years ago

if that's possible, how can we create an easylist? Because some ads get stuck with this one. Thanks https://easylist-downloads.adblockplus.org/liste_fr.txt

I am not sure what you want here, you want to generate an easylist for use in adblockplus? In that case, I don't really know how to do that.

TechComputerServiceDunkerque commented 4 years ago

You can try to change the lists by the ones I gave you, just make a backup of your old list on your machine and see if everything is ok, I made some tests, some sites will be blocked (just put them in white list), I think the other users will be happy because most of the lists I gave you above are international and block a lot of things. For messenger, I did some tests with your dns, it still blocks. But here with the lists I tested, nothing is blocked and everything is optimized, even the speed of android (I'm even quite surprised by the result). Why don't you change the google dns on the server to FDN's? They are uncensored too. For the easylist list, let's drop it, it blocks too many things, I just wanted to know if we could make an easylist host.

Translated with www.DeepL.com/Translator (free version)

TechComputerServiceDunkerque commented 4 years ago

Here are the lists:

Android/IOS:

https://gist.githubusercontent.com/sgeto/36bd4b93fc23db9936be06fa8c6a5dc3/raw/6ec2966dd7960f5bb58f29db4f8aed25e0f92e8f/lphosts (67 235 blocked)

Pornoblocking:

ADS Complete

Filter DNS (Blah DNS, Warden DNS)

Privacy

Trackers, spam, scam, phishing, malware)

I hope you will like these lists, they are complete, some sites may be blocked by mistake, but as I know you already have a white list, I think it's useless to remove it. Let me know if everything's okay with you?

if that's possible, how can we create an easylist? Because some ads get stuck with this one. Thanks

https://easylist-downloads.adblockplus.org/liste_fr.txt

I tested the filters with blokada and the next dns: ns0.fdn.fr : 80.67.169.12 ou 2001:910:800::12 ns1.fdn.fr : 80.67.169.40 ou 2001:910:800::40

Also I notice that with the old filters, ads on android are back (google ads, admob). While with the new lists I gave you, all is blocked correctly. So I'm forced to use Nixnet DNS on my router for the moment to avoid these invasive ads while waiting for your update...

Bancuh S00422-06251718

TechComputerServiceDunkerque commented 4 years ago

Nixnet dns (pub delete)

S00422-07591788

TechComputerServiceDunkerque commented 4 years ago

thanks to you for the work you do :)

ragibkl commented 4 years ago

I tried to make this, I had some issues, but I hope it is fixed now. Please test it and let me know.

TechComputerServiceDunkerque commented 4 years ago

Hello, yes the google ads are well blocked there :). Have you thought about what I suggested with the lists or unless you analyze the lists to see if they're not too heavy?

ragibkl commented 4 years ago

Hello, yes the google ads are well blocked there :). Have you thought about what I suggested with the lists or unless you analyze the lists to see if they're not too heavy?

Adblock Coverage

I actually have not tested the DNS recently. I'm out of town right now, and don't have them configured for my use right now. If you say the experience is better, then I'll take that as an improvement.

The final block list is about 2 million sites. Before this recent change, it was only about ~ 1.5 million sites. This is an improvement from before.

As long as the list helps users, and does not block legitimate sites, I'm happy to keep them.

Whitelist requirements

This requires a bit more testing, but from your report, it seems like these new sources don't block any other legitimate sites. So I think we can just keep using them.

The existing whitelist still works. We just need to keep monitoring for any malfunctions, and add whitelist as needed.

Performance and server loads

Currently, the DNS servers are consuming about 1.2 GiB of RAM. The servers are configured for 2 GiB, so I think it is still within limits. Before this, I think it was ~ 1 GiB.

So far there's no drop in performance, so I think we are fine here.

Re-adding sources from original list

I think, most of the recent issues with FB and MS started when I added the sources that was provided in this ticket. Otherwise, I think the sources that I've used before that, did not cause as much issues.

I think I want to re-add them. There is no reason to suspect that they caused any breakages. Let me know what you think.

BTW, if there's nothing breaking after few days, I think we can close this issue. I still want to do item re-adding, maybe I'll track that in a separate ticket.

TechComputerServiceDunkerque commented 4 years ago

Hello, yes no worries otherwise I can reopen a ticket in the coming days with the list I provided you. Your dns is becoming popular in France since I put it first on my blog's ranking and I'm happy to help you, I like challenges, to succeed in blocking everything would be a luxury 😃 but there is still work to do. After certainly in this list, some sites will be blocked, but it will be enough to unblock them, I will also create a host of my own that I will give you for your dns, I have some sites that manage to pass through the nets of adblock.

ragibkl commented 4 years ago

@TechComputerServiceDunkerque

  1. I re-added some previous sources into the list. These are the ones that I had originally. Nothing should break, cause we know they worked before. But just be aware when testing.

  2. I wanted to try setting up a server in France. Please go here and help me test it.

TechComputerServiceDunkerque commented 4 years ago

Hello ragibkl,

this is a host that blocks advertising overlay in the world.

https://someonewhocares.org/hosts/hosts

https://pgl.yoyo.org/as/serverlist.php?hostformat=hosts

I'll let you watch.

ragibkl commented 4 years ago

Hi @TechComputerServiceDunkerque , thanks for the suggestion. I'll try to add those into my list as well.

On a side note, I think if you have new suggestions or a new request like this, it is better to open a new issue ticket. It allows us to better track the progress of the new suggestion separate from the old issue.

With that being said, I will close this issue and open a new ticket separately. I'll update any progress on the new ticket instead.

ragibkl commented 4 years ago

Closing this ticket. Created a new ticket to track this here: https://github.com/ragibkl/adblock-dns-server/issues/56 Let's continue the discussion there instead.