StevenBlack / hosts

🔒 Consolidating and extending hosts files from several well-curated sources. Optionally pick extensions for porn, social media, and other categories.
MIT License
26.65k stars 2.21k forks source link

Some ads can't be blocked? #1075

Closed BugHunt3rM4x closed 4 years ago

BugHunt3rM4x commented 4 years ago

Despite being blocked and dns cache being wiped I can't block ads from these hosts. I can't go to the sites but they still show ads. news.freegames66.com universesearch.net Other custom blocks work such as these ad/malware networks I've found. 0.0.0.0 get.stream-all.com 0.0.0.0 airartapt.site 0.0.0.0 arcaptart.site 0.0.0.0 allashark.site 0.0.0.0 bumcapale.site

welcome[bot] commented 4 years ago

Hello! Thank you for opening your first issue in this repo. It’s people like you who make these host files better!

spirillen commented 4 years ago

From which site do you see ads from the listed resources?

Can you post some examples urls?

BugHunt3rM4x commented 4 years ago

From which site do you see ads from the listed resources?

Can you post some examples urls?

kimcartoon.to sitewide.

Martii commented 4 years ago

kimcartoon.to

What an obnoxious site. Typed just that above into the addy bar and there's some sort of privacy pass.... noticed that its http to https doesn't work properly so https://kimcartoon.to actually takes you there.

news.freegames66.com

Doesn't appear to be in my list with an update of today of block-able items but will try a manual entry.

Rule:

local-zone: "news.freegames66.com" always_nxdomain

... with restart ...

$ sudo unbound-control -c /etc/unbound/unbound.conf reload

... and browser refresh (soft and hard) ...

... Additional flush check(s)...

$ sudo unbound-control flush_zone news.freegames66.com
ok removed 0 rrsets, 0 messages and 0 key entries

$ sudo unbound-control flush_zone kimcartoon.to
ok removed 2 rrsets, 2 messages and 0 key entries

Definitely still seeing ads text and some pictures in unfettered browser on left and right side of actual content (in iframes)... presuming everything was done correctly (pasted directly from terminal so hope so).

spirillen commented 4 years ago

Hi @Martii Nice explanation of what you have done and how you did it :100:

The right way to let other replicate and find eventual failures :+1:

A little schooling for enhanging your knowledge to find the "&/%(/&¤%¤/" banners

In your browser you opens dev tool ( mosts browsers I can think of uses F12 as hotkey)

My next reply will be my investigation of the OP's Q.

Screenshot_20191028_213656

spirillen commented 4 years ago

First off, from a very new and very clean FF 70 profile I see no banners what so ever....

But here are some spooky urls that might require a bit more investigation, but I would not test these directly from my working computer

https://keapeiros.xyz/1clkn/10526
https://balvalur.com/pntne

And some absolutely differently ads urls #269

https://services.bilsyndication.com/adv1/?d=850
https://biltag.bilsyndication.com/jsv1/1572284506/?d=850&n=
https://assets.bilsyndication.com/plugins/cmpv2/cmp.complete.bundle.js
https://assets.bilsyndication.com/prebid/default/prebid-v2.38.0.js
https://assets.bilsyndication.com/plugins/vlPlayer/min/viPlayer_v24.js
https://assets.bilsyndication.com/plugins/safeframe/src/js/sf_host.min.js

Privacy violating Tracking urls:

https://platform.twitter.com/widgets.js
https://platform.twitter.com/widgets/widget_iframe.2d991e3dfc9abb2549972ce8b64c5d85.html?origin=https%3A%2F%2Fkimcartoon.to
https://syndication.twitter.com/settings
https://platform.twitter.com/js/button.d941c9a422e2e3faf474b82a1f39e936.js
https://platform.twitter.com/widgets/follow_button.2d991e3dfc9abb2549972ce8b64c5d85.en.html#dnt=false&id=twitter-widget-0&lang=en&screen_name=kimcartoonweb&show_count=false&show_screen_name=false&size=m&time=1572290057363
https://syndication.twitter.com/i/jot?l=%7B%22widget_origin%22%3A%22https%3A%2F%2Fkimcartoon.to%2F%22%2C%22widget_frame%22%3Afalse%2C%22language%22%3A%22en%22%2C%22message%22%3A%22m%3Awithcount%3A%22%2C%22_category_%22%3A%22tfw_client_event%22%2C%22triggered_on%22%3A1572290057781%2C%22dnt%22%3Afalse%2C%22client_version%22%3A%223541749%3A1571780739496%22%2C%22format_version%22%3A1%2C%22event_namespace%22%3A%7B%22client%22%3A%22tfw%22%2C%22page%22%3A%22button%22%2C%22section%22%3A%22follow%22%2C%22action%22%3A%22impression%22%7D%7D
https://www.google-analytics.com/analytics.js
https://imasdk.googleapis.com/js/sdkloader/ima3.js
https://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fwww.facebook.com%2Fkimcartoonfp&send=false&layout=button_count&width=100&show_faces=false&action=like&colorscheme=dark&font&height=21
https://www.facebook.com/plugins/likebox.php?href=http%3A%2F%2Fwww.facebook.com%2FKimCartoon-1500686783296635&width=300&colorscheme=dark&show_faces=true&stream=false&header=false&height=160

Other:

https://vendorlist.consensu.org/vendorlist.json; /var/lib/unbound/someonewhocares.db

consensu.org

Screenshot_2019-10-28 Save BIG with $9 99 COMs from GoDaddy

The some of this site investigation is: Not reproducible, as I can't find any queries to news.freegames66.com on the site in question, However there is a very high number of spooky domain queries that would make my think twice about visiting that site again.

ScreenShots from test Screenshot_2019-10-28 KimCartoon - Watch cartoons online FREE Screenshot_20191028_201449 cross profile notification Screenshot_20191028_201549 privacy violating notification and ads spam warning

Next test would be with the Tor Browser Screenshot_2019-10-28 The Tor Project Privacy Freedom Online

Nope, still no banners, but also again, most content is blocked by the hostler

Screenshot_20191028_204032 via tor

Final conclusion

news.freegames66.com is not in any interaction with kimcartoon.to

funilrys commented 4 years ago

I tried to hunt it ... Well, good luck!

Putting assets.bilsyndication.com in my iptables (Yes I know we are talking about hosts files 🙄 But it was the quickest way from where I was...) make the ads disappear from a machine with nothing ....

Martii commented 4 years ago

@spirillen

news.freegames66.com is not in any interaction with kimcartoon.to

You might be forgetting region specific targeting. It's definitely present, and interacting, as screen shotted here, including Fx:

news.freegames66.com iframes

@funilrys

assets.bilsyndication.com

Will give that a try momentarily. EDIT Nope... still present with the iframes.

Refs: Rule...

local-zone: "assets.bilsyndication.com" always_nxdomain

Cache...

$ sudo unbound-control flush_zone news.freegames66.com
ok removed 0 rrsets, 0 messages and 0 key entries
$ sudo unbound-control flush_zone kimcartoon.to
ok removed 2 rrsets, 2 messages and 0 key entries
$ sudo unbound-control flush_zone assets.bilsyndication.com
ok removed 0 rrsets, 0 messages and 0 key entries

The seemingly to be web dot image from appears to be filtered directly in the address bar but not the iframes loading... so that's possibly what the reporter meant by news.freegames66.com. Still interesting that you aren't getting those though. Probably region targeting.

On my little soap box, if anyone advertises to me in the browser there's is almost always a 100% chance that I won't ever click or buy anything that way. I prefer to do my own research and find my own sources. :smile_cat: ... besides they can't figure out what I like anyhow because it's on a need to know basis and do some shopping for other people. ;)

spirillen commented 4 years ago

Hi @Martii

Did you remember to reload Unbound and clear browser cache?

Firefox cache dns queries.... Screenshot_20191028_221609

You should block all of bilsyndicate by:

local-zone: "bilsyndication.com" always_nxdomain

I'm fully aware of the localization, but without stripping the js, my guess is that at least one of them is bypassing your local dns setting and goes strait for google's dns server

Could you or @funilrys try to have a look on your network traffic for traffic on port :53 and :853

On my little soap box, if anyone advertises to me in the browser there's is almost always a 100% chance that I won't ever click or buy anything that way. I prefer to do my own research and find my own sources. smile_cat ... besides they can't figure out what I like anyhow because it's on a need to know basis and do some shopping for other people. ;) :+1:

I love good challenges :champagne:

spirillen commented 4 years ago

Maybe try to open the page-source and see if the are any direct queries from kimcartoon.to to news.freegames66.com....

funilrys commented 4 years ago

While hunting, I had to open/close the private/incognito window because they put everything in the cache. If you give them 1s of traffic, they put everything in cache and never leave until you properly clean your cache/cookies and reload.

Martii commented 4 years ago

@spirillen

You should block all of bilsyndicate by ...

This did the trick for the iframes themselves: ( EDIT: But not sure if the content is playable... since I'm unfamiliar with the site)

local-zone: "bilsyndication.com" always_nxdomain

(Once posted above it's always implied that I reload unbound otherwise the rule wouldn't go into effect plus Ctrl + Shift + Del is my friend and rote rehearsal ingrained :wolf: ;)

port :53 and :853

Even if they use those ports on dev it's routed to unbound/DoT and the filters here (usually try.conf for what I try from you or someone else) afaik (last tested in February and since the filters are working it should be that still between OS updates). If any browser bypasses local security that's an issue as we've already discussed. As you mentioned it's a rather "spooky" site... I usually don't visit that site but thought this issue could use a little more detail than the original reporter did... and I think we hit the same FQDN in viewing.

spirillen commented 4 years ago

We are defiantly on the same page about braking down this.

But what we haven't found is the how do there come a call to news.freegames66.com which bypass the hosts file.... and/or a local DNS recursor and since @funilrys got it in first step, by blocking on firewall level...

The call to news.freegames66.com most comes from one of the .js from bilsyndication.com that bypass local settings and using external DNS e.g. cloudflare.com as both freegames and bilsyndication is hosted there

spirillen commented 4 years ago

Just ran a test of https://keapeiros.xyz/1clkn/10526 on virustotal :wink:

https://balvalur.com/pntne not looking any better VirusTotal

Better get them in my own list...

Martii commented 4 years ago

@spirillen

... a local DNS recursor ...

As I added below the hr above a few minutes later before your reply... it's catching the highlighted image in the try.conf just not the iframes until you mentioned the whole domain blockage to remove the iframes themselves. Here's an example of what it "normally" looks like and it's url:

Highlighted image source.

Can't speak for the hosts since I'm not using that for educational testing atm.

spirillen commented 4 years ago

AAhh that was the reference to try.conf :smile: well spottet :100: :medal_sports:

I thought it was a local script of yours :scroll:

spirillen commented 4 years ago

@BugHunt3rM4x are your problem solved by adding

0.0.0.0 services.bilsyndication.com
0.0.0.0 biltag.bilsyndication.com
0.0.0.0 assets.bilsyndication.com

to your hosts file?