Add cloudflare IPs to exceptions? weboost.com (assets.wilsonelectronics.com) blocked

[poisonsnak]

Today I noticed weboost.com hangs while loading because the images that come from assets.wilsonelectronics.com are blocked.

assets.wilsonelectronics.com resolves to Cloudflare IPs for me:

104.26.2.137 104.26.3.137 172.67.69.149

and these are all in the list but not in the exception list. So I guess Cloudflare is providing DOH as well as hosting websites on these IPs? I'm not sure how the IPs were added to the list but I guess they must be DOH servers. Cloudflare's official DOH servers are all IPs like 1.1.1.1, 1.0.0.1, etc.

There was a similar issue opened here but the user closed it and decided to allowlist them himself https://github.com/jpgpi250/piholemanual/issues/12 . Also some similar discussion here https://github.com/jpgpi250/piholemanual/issues/3 which led to you creating the exception lists.

How do you feel about adding the Cloudflare IP ranges to the exception lists (https://www.cloudflare.com/ips/) ? A lot of websites use Cloudflare and I guess any of them could end up hosted on one of these IPs. But if Cloudflare is also using these for DOH then you have a huge gap in your blocklist. Sucks that they strongarm you into not blocking them by hosting websites on them. They don't include their official DOH IPs (1.1.1.1 etc.) in the IP lists I linked.

Personally I don't use the exception list and just write in my own exceptions case by case but I thought I'd share and get your view on it.

[jpgpi250]

first, how I got the IP addresses you mentioned: 104.26.2.137 blitz.ahadns.com 104.26.2.137 blitz-setup.ahadns.com

104.26.3.137 blitz.ahadns.com 104.26.3.137 blitz-setup.ahadns.com

172.67.69.149 blitz.ahadns.com 172.67.69.149 blitz-setup.ahadns.com

second, lists where these domains appear in: blitz.ahadns.com: 7, 11, 12, 15 blitz-setup.ahadns.com: 11, 12

7: https://raw.githubusercontent.com/wiki/curl/curl/DNS-over-HTTPS.md 11: https://raw.githubusercontent.com/dibdot/DoH-IP-blocklists/master/doh-ipv4.txt 12: https://raw.githubusercontent.com/dibdot/DoH-IP-blocklists/master/doh-ipv6.txt 15: https://raw.githubusercontent.com/jbaggs/doh-intel/master/doh.intel

only list 15 is new, since 2022-08-02 ,see NOTICE here

third, IPv6 addresses, associated with this domain: 2606:4700:20::681a:289 blitz.ahadns.com 2606:4700:20::681a:389 blitz.ahadns.com 2606:4700:20::ac43:4595 blitz.ahadns.com

2606:4700:20::681a:289 blitz-setup.ahadns.com 2606:4700:20::681a:389 blitz-setup.ahadns.com 2606:4700:20::ac43:4595 blitz-setup.ahadns.com

It is a very good idea to maintain your own exception list, this prevents the excessive growht of exceptions, using the generic list. I will (a soon as possible) update the doc to encourage new users to create a personal exception list, this to avoid entries that don't apply to the users environment / region, and ever growing exception list(s). The DoH IP addresses, mentioned above, will thus NOT be added to the exceptions, the exceptions lists are considered deprecated, the files will remain on GitHub, this to ensure the block rules doesn't cause problems in existing environments.

I strongly recommend, if you are using unbound (or bind, knot resolver, ...), to look into and add the RPZ (response policy zone) feature (mentioned in de DoH manual, explained in full in a separate manual). The idea is, even though you have whitelisted (exception) a specific IP address, the DNS name is still blocked. A client using a different name than the DoH name will not be hindered, getting the required resources (images in this specific case), a client trying to find the address for a DoH service, using the DNS name will be blocked.

Of course, if a client (app) is using IP addresses to get to the DoH servers, the RPZ will do nothing, the exceptions will make the DoH request succeed...

edit I don't think adding the Cloudflare IP ranges (https://www.cloudflare.com/ips/) addresses (ranges) to the IP execption lists is a good idea, due to the amount of IP addresses that would be allowed. I very much like your approach to create customized (your own) exception lists, adding exceptions case by case. /edit

edit2 from the current IPv4 exception list ( 37 entries) only 2 are in the cloudflare ranges (15) from the IPv4 list ( 418 entries), 28 are in the cloudflare ranges

from the current IPv6 exception list ( 9 entries) only 2 are in the cloudflare ranges (7) from the IPv6 list ( 281 entries), 36 are in the cloudflare ranges

tested with grepcidr (example usage https://unix.stackexchange.com/questions/274330/check-ip-is-in-range-of-whitelist-array - accepted solution) /edit2

Your thoughts are welcome....

[jpgpi250]

I've been working (thinking) almost all day to find the correct solution for this issue, and decided your suggestion (create a personal exception list) is the best solution.

I've editted my previous comment several times to get to the final version, you might want to reread it.

As explained (and also in the open NOTICE, I will consider the exceptionlists as deprecated.

Question for you: what method do you use to define the exceptions? options: 1: IP alias, has the advantage you can add a comment, everything in one place. 2: URL alias, requires extra files (v4 & v6) on a web server, also allows for comments (# entry on a separate line)

what method would be the best (consider NOT all users are as advanced as you are (with your great ideas!!).

Thanks for your time and effort, hoping to hear your thoughts.

[poisonsnak]

Thanks for the detailed reply. It's good to know they are actual DoH servers.

Using the RPZ feature to block DNS names is a good idea. I forgot that DoH usually depends on "regular" DNS to get going. I'll be sure to block all the known hostnames for DoH servers. I use nextdns as my upstream server and I think they do this for me already with the "block bypass methods" toggle but I'll double check.

Thanks for using that grepcidr tool to figure out which IPs on your lists are in the Cloudflare ranges. I'd never heard of grepcidr before but it sounds handy. For me, on the one hand I want to block as many DoH servers as possible, but on the other hand I don't want to cause problems for my users. If it was just me I wouldn't add the full Cloudflare ranges to my exception list but I run a small business and the staff get kind of upset when "the internet" doesn't work haha. So I ended up adding the Cloudflare ranges to my exception list.

For the method I use to define the exception list, my router is a Ubiquiti Edgerouter so I just have a firewall rule that rejects outbound requests to servers on your list, but then I insert a rule just before it to allow access to IPs on my exception list. The Edgerouter allows one comment per rule so I can keep track of them that way. The way you've described it I think the IP alias sounds the best since it's simpler for those users that don't have as much experience.

Thanks again

[poisonsnak]

So as I was looking into AhaDNS a bit more, I believe blitz.ahadns.com and blitz-setup.ahadns.com are just web sites that explain how to set up AhaDNS. Their actual DoH servers are set up in the format https://doh.[LOCATION].ahadns.net/dns-query . You can see the full list here https://ahadns.com/dns-over-https/ . So you might be able to remove those IPs for blitz.ahadns.com and blitz-setup.ahadns.com after all. The ahadns.net servers are all in your list so we're covered there.

As I thought about it and felt somewhat nervous adding Cloudflare's huge IP ranges to my exception list, I wondered if Cloudflare would actually proxy DoH requests. AhaDNS's instructions don't list blitz.ahadns.com or blitz-setup.ahadns.com as usable DoH servers. Do you know of a way to test a DoH server? I tried adding it to Firefox's settings but even if I put a server there that I know is invalid, DNS still works so I can't use it as a test. I know dig recently got DoH support (https://www.isc.org/blogs/bind-doh-update-2021/) but the version I have in Debian Stable isn't new enough

[jpgpi250]

The DoH rules prevent me from checking https://ahadns.com/dns-over-https/ right now, I'll look into that later, and if possible, add the list (check the doc for all lists used), this to ensure the ahadns servers remain in the block list. As explained, I only parse the lists I can find, extracting domains the authors have marked as (o)DoH servers. I never test if they are actual DoH servers, only try to get the IPs and add these to the block list. New domains are added, but never removed, althoug it is possible to keep track of removed entries, by verifying the timestamp of the entries in the database (see sqlite3 queries in the doc).

There is something called "oblivious DNS over HTTPS" wich uses a proxy. The proxy knows the IP of the requestor, this to be able to send the answer, the actual DoH server recieves all requests from the proxy, thus no client identification possible. Those providers that support oDoH all claim the proxy doesn't keep logs... Don't know if cloudflare supports oDoH.

There are lots of DoH clients on GitHub, I've tried some, in order to be able to test if it's really a DoH server, none of them stand out, they all have pro / cons and don't always work. Most of the lists only provide a domain (example ahadns.com), you need the URL to test (https://ahadns.com/dns-over-https/), so it's not always clear what to test.

You can install a windows version of dig, following this guide. Since the link you provided says "as of March, 2021" and the latest downloadable windows version says "July 2022", this could possibly work (not tested yet)

I'll update the manual as soon as possible, using the alias IP as the method to create "personal" exceptions, marking the GitHub exceptionlists as deprecated. The database version will increase to v4, given the new policy (create local exception lists), the exceptions table becomes obsolete.

[jpgpi250]

All done, documentation, scripts, ... generated new lists, this to ensure everything still works...

• time required to make the RPZ list (from database): 1m49s
• time required to make the DOH lists (from source lists): 51m50s

ever considered writing a manual to implement this on Ubiquiti Edgerouter? Would increase the user base...

[poisonsnak]

Unfortunately ISC stopped providing Windows builds and you have to install 9.16.31 (but 9.17.10 is when they added DoH support). I have a machine running Debian Testing that should be able to install 9.18.4 so I will give that a go next time I have access to it, or worst case I can try one of those clients floating around github.

For the Edgerouter manual, I would but Ubiquiti has discontinued their EdgeMAX line. They aren't officially discontinued but they don't release updates for them any more - 2.0.9 is the current version for Edgerouter and it was released in 2020 (although it has gotten a few basic hotfixes since then - just things like updating openssl). You can't buy them anywhere either - I have one Edgeswitch I need and it's been on "back order" for 18 months. It's a shame because they are a really good product but Ubiquiti has gotten really weird the past few years. Their UniFi and AirMAX products are still available and see regular updates, but they aren't as easy to customize as the EdgeMAX products so it would be a lot harder to use your list on them (if it's even possible).

I use the script here modified to my own situation https://github.com/WaterByWind/edgeos-bl-mgmt and there's a megathread on the community here https://community.ui.com/questions/Emerging-Threats-Blacklist/62a9549e-ddae-4631-941d-b0878b2a13e0 . It pulls your list in once every couple days along with a bunch of other lists I use and updates them in the firewall automatically. So if you have Edgerouter users that are interested I'd send them in that direction.

[jpgpi250]

The domain entries from https://ahadns.com/dns-over-https/ have been added, this to ensure the DoH servers remain blocked, even if removed from other lists.

[jpgpi250]

added a section / method (manual) to limit the number of exceptions, can be used for CIDR addresses, such as cloudflare, google, ...