anudeepND / blacklist

Curated and well-maintained hostfile to block ads, tracking, cryptomining, and more! Updated regularly. ⚡🔒
https://hosts.anudeep.me/mirror/adservers.txt
MIT License
1.13k stars 111 forks source link

Google Fonts is blocked #57

Closed hectorm closed 5 years ago

hectorm commented 5 years ago

I don't know if this is intentional, but the domain googleadapis.l.google.com was added in 62b873ddbd7170ae1c6964186eec0a0d984ab7ef and is causing the problem, since fonts.googleapis.com contains a CNAME record to this domain.

dnmTX commented 5 years ago

@hectorm can you post a URL sample so it could be inspected and confirmed.Without it it's just a baseless claim so far. Thank you.

hectorm commented 5 years ago

Code snippets generated at fonts.google.com use the domain fonts.googleapis.com.

With dig we can see how fonts.googleapis.com has a CNAME record with value googleadapis.l.google.com.

$ dig fonts.googleapis.com @1.1.1.1

; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> fonts.googleapis.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5103
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;fonts.googleapis.com.      IN  A

;; ANSWER SECTION:
fonts.googleapis.com.   901 IN  CNAME   googleadapis.l.google.com.
googleadapis.l.google.com. 216  IN  A   172.217.17.10

;; Query time: 9 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Tue Aug 27 22:05:59 CEST 2019
;; MSG SIZE  rcvd: 101

Other blocklists have already erroneously blocked Google Fonts in the past (https://github.com/StevenBlack/hosts/issues/505, https://github.com/AdguardTeam/AdGuardHome/issues/460).

dnmTX commented 5 years ago

fonts.googleapis.com is not blocked and it doesn't matter that one of it's CNAMES is googleadapis.l.google.com. I have yet to see google fonts being blocked because of it,so far everything works fine on my end.So,again,either provide specific URL where you can prove that the domain in question indeed creates problems by being blocked or i can guarantee you that your request will be denied and the issue closed.

hectorm commented 5 years ago

Reproducing the problem depends on how you are using this blocklist.

When you use the /etc/hosts file this problem does not occur, but when you implement a DNS block, such as using Knot Resolver with static hints (e.g., hints['googleadapis.l.google.com'] = '0.0.0.0'), any client using this resolver will get the following answer:

$ dig fonts.googleapis.com @127.0.0.153

; <<>> DiG 9.11.3-1ubuntu1.8-Ubuntu <<>> fonts.googleapis.com @127.0.0.153
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30148
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;fonts.googleapis.com.      IN  A

;; ANSWER SECTION:
fonts.googleapis.com.   362 IN  CNAME   googleadapis.l.google.com.
googleadapis.l.google.com. 5    IN  A   0.0.0.0

;; Query time: 0 msec
;; SERVER: 127.0.0.153#53(127.0.0.153)
;; WHEN: Wed Aug 28 00:15:00 CEST 2019
;; MSG SIZE  rcvd: 101

You can reproduce the problem in any environment that uses this specific setup.

dnmTX commented 5 years ago

@hectorm i see now what you mean and would advise you to contact the developer(s) there and explain all this to them and see what they can do about it. CNAMES in many cases are used not just by one domain and some are blocked by hosts file projects like this one here and some not but the bottom line is that no regular dns resolver will block CNAME on any domain or the domain with that CNAME because one is blocked(very confusing to explain...i know). DNS over TLS are more complex and i know that when in use they using SOCKS 5 protocol which completely bypassing the hosts file so having such a resolver doing both creates problems,still.

anudeepND commented 5 years ago

@hectorm Interesting.... Google Fonts works for me when using pi-hole. But I don't know what is delivered over googleadapis.l.google.com, it was in my pi-hole log. It is better to remove it from the list, what do you think @dnmTX I don't want to be too aggressive with the list

dnmTX commented 5 years ago

@anudeepND the problem @hectorm is having is different.It's not about being too aggressive or not but it's about the dns resolver that he's using. His resolver will block not just the domain name from your or any other lists like yours but it's CNAME as well,or if the CNAME(only) is blocked in your list the resolver still will block it's domain name with it,no matter what.Where any other normal dns resolver will block only the domain name. Removing the domain in question from your list will solve his problem for now but i can guarantee you that he'll be back reporting other domains(here or in some other repo,doesn't matter). I'll leave it to you to decide but keep in mind if @hectorm continuing to report domains because of the same reason we need to make sure that removing it wont cause more harm then good.

hectorm commented 5 years ago

This behavior is common in DNS resolvers. For example systemd-resolved, the resolver included by default in most distributions that use systemd, behaves in the same way when there is an entry of this type in the /etc/hosts file.

anudeepND commented 5 years ago

I will keep the domain in the list, I will remove it if I get more reports on this domain. Thanks @hectorm and @dnmTX

DRSDavidSoft commented 5 years ago

@hectorm I can confirm the same behavior in Unbound, another DNS resolver and cache server that I use in my setup (DNScrypt + Unbound).

@anudeepND I also think that googleadapis.l.google.com should not be in the adservers.txt list, since it's a general domain and is not exclusively used to serve ads.

Personally I have the googleadapis.l.google.com entry in my whitelist, since I've had other issues with that domain being blocked, other than as a CNAME to something like fonts.google.com.

I highly encourage you to remove any domains that is not exclusively used to serve ads, or used only to track users. If a domain (or its CNAME) is known to serve useful content, it should not be blocked.

As a side note, if anyone's wondering why I block CNAMEs to a blocked domain, click here for a detailed reason. 1. Consider an obvious domain like ads.some-evil-corp.com that anyone knows about, and block it in their hosts list. 2. When a user's browser tries to connect to that particular domain, the request fails because the domain name exists in their blacklist. 3. Either the person who runs the website, or guys behind `some-evil-corp.com`, register a new domain like: files.innocent-name.com. 4. Now, since the new domain is not in the user's blacklist, when the browser tries to make a request, it succeeds and the ads are displayed!
I also block all wildcards, in addition to CNAMEs. click here for explanation as to why. So, if some-ads-server.com is blocked, all requests to the following will be blocked as well:
s1.some-ads-server.com
s2.some-ads-server.com
....
s9999.some-ads-server.com
and so on.
anudeepND commented 5 years ago

@DRSDavidSoft Thanks for your valuable info, currently no ads are being served from this domain. I will remove it the next update