HTTP(S) connections (without DNS) to certain hosts are blocked (by ISP?)

[yuukiAme]

Creating a bug report/issue

• [x] I have searched the existing open and closed issues

Required Information

• DietPi version | cat /boot/dietpi/.version

root@DietPi:~# cat /boot/dietpi/.version
cat /boot/dietpi/.version
G_DIETPI_VERSION_CORE=9
G_DIETPI_VERSION_SUB=3
G_DIETPI_VERSION_RC=0
G_GITBRANCH='master'
G_GITOWNER='MichaIng'

• Distro version | echo $G_DISTRO_NAME $G_RASPBIAN

root@DietPi:~# echo $G_DISTRO_NAME $G_RASPBIAN
bookworm 0

• Kernel version | uname -a

root@DietPi:~# uname -a
Linux DietPi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

• SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)

root@DietPi:~# echo $G_HW_MODEL_NAME
RPi 4 Model B (aarch64)

• Power supply used | (EG: 5V 1A RAVpower)

Raspberry Pi Power supply 5V 3A

• SD card used | (EG: SanDisk ultra)

SanDisk Ultra 64G

Additional Information (if applicable)

• Software title

PiVPN

• Was the software title installed freshly or updated/migrated?

Freshly installed

• Can this issue be replicated on a fresh installation of DietPi?

Not sure.

• Bug report ID | echo $G_HW_UUID

root@DietPi:~# echo $G_HW_UUID
c16aa61b-b23a-4d7c-adcd-91591412a10b

Steps to reproduce

1. install Dietpi image on RPi

2. apt update && apt upgrade -y

3. dietpi-launcher

4. install Pi-hole, Unbound, activate DoT with the conf from Dietpi Docs

5. install pivpn, wireguard, setup DNS support with Pi-Hole ,add user, scan QR code

6. add conf file to a device (e.g: an Android phone) to connect to pivpn OUTSIDE of the network

7. check for the websites that is KNOWN to be blocked like 18+ sites, see if it returns query on unbound, load the content on the browser within my Local Network, meaning the DoT is working because my DNS query is encrypted and returned to Unbound with no issue.

8. Turn on Wireguard VPN on the device in Step 6

9. check if the known blocked website works and contents are loaded in the browser.

Expected behaviour

It should load all the contents of the website in the browser.

Actual behaviour

The browser will not load any content and return an error like "Unable to connect". Like an unencrypt query was rejected by the ISP DNS.

Extra details

I can use Pi-Hole with Unbound (activated DoT (DNS over TLS)). I can browse normally to any websites I want other than the block list inside Pi-Hole. My country required ISP to blocks certain websites from their own DNS. So ISP will allow your network to query the website, but it will reject the content of the website.

Help me to troubleshoot if PiVPN is leaking the query because when I access my network over PiVPN, it behaves exactly like when I access the blocked websites WITHOUT Pi-Hole and Unbound DoT which mean my query to unencrypted and intercept by my ISP.

My question is:

How do I check/troubleshoot PiVPN whether it is encrypting my query to Pi-Hole and Unbound on port 5335 or leaving the query unencrypted like port 53?

[MichaIng]

First of all, of course, when you disable the DNS setting (i.e. not use Pi-hole>Unbound>DoT) in the WireGuard config, you need to enable DoH in browsers. The idea was to test whether, when DoH is used, toggling the VPN alone toggles the issue. Since Pi-hole does not support DoH, you can test this with any upstream DoH provider, or test again with AdGuard Home.

Of course pi.hole is only available, when using Pi-hole for DNS resolution, hence this only works when enforcing Pi-hole as DNS via VPN config or without VPN via LAN/DHCP server, and DoH must be disabled. However, this is irrelevant for the pornhub issue.

When you remove the "DNS" setting and stop the VPN, it is possible that you need to disconnect and reconnect to your LAN network, to get the DNS server from the DHCP server. I am not sure whether it has the original DNS server still stored somewhere internally to recover it on VPN stop. However, this should be a one-time action, since the VPN won't touch /etc/resolv.conf anymore on startup now. This would explain why accessing any hostname, including Google, failed.

Regarding the traceroute, interesting to see the exact routes for the two test cases of my first sentence: VPN enabled + DoH, VPN disabled + DoH, both when connected to the home LAN I assume that, while DoH is used, enabling the VPN breaks pornhub access, disabling the VPN fixes it. And the route (after home router) is the only difference I can think of, while I have no idea how using the VPN from within your home LAN can have an effect on it.

[yuukiAme]

when you disable the DNS setting (i.e. not use Pi-hole>Unbound>DoT) in the WireGuard config, you need to enable DoH in browsers. The idea was to test whether, when DoH is used, toggling the VPN alone toggles the issue.

Well, I redid the test like you suggested above,

• deleted DNS entry in conf file and saved the file
• change Firefox Settings > DNS over HTTPS > Max protection > custom > my own AGH with ClientID

outcome:

• Browser cannot use the internet (i.e. www.google.com)
• Browser can access local network like the coffee shop's internet default Gateway (192.168.1.1)
• AGH query log has no entry from Macbook ClientID at all.
• using nslookup www.google.com in Terminal returned a google public IP address (141.251.220.14).

changed Max protection > Cloudflare (default)

• Browser cannot use the internet (i.e. www.google.com)
• Browser can access local network like the coffee shop's internet default Gateway (192.168.1.1)

Same outcome.

Since Pi-hole does not support DoH, you can test this with any upstream DoH provider, or test again with AdGuard Home.

It seems like without the DNS entry in the config file, the VPN will use the default DNS by the router which has the ISP DNS and block my query immediately if it's banned in my country.

I have nothing else to add related to matters above.

But one interesting tidbits, my country blocked https://store.steampowered.com this entire time since April 2024 - now, but I never noticed it because of Pi-hole, Unbound DoT, they unblock everything for me. I'm so glad to be using DietPi to assist me with all this.

I didn't believe it until I tried nslookup https://store.steampowered.com with default DNS provided by my ISP and it returned 127.0.0.1 . But with PiVPN on Macbook, nslookup https://store.steampowered.com returned the Steam Public IP.

I guess my ISP is cracking down harder on www.pornhub.com traffics than https://store.steampowered.com. But I still have no idea why my PC (O11) is working as intended but my other devices cannot access www.pornhub.com.

[MichaIng]

It seems like without the DNS entry in the config file, the VPN will use the default DNS by the router which has the ISP DNS and block my query immediately if it's banned in my country.

Hmm, yes, the "system" uses the default DNS from the ISP or whatever you configured in your LAN. However, with DoH enabled, the browser overrides this/does not use the system DNS. Hence if you cannot access Google with whichever DoH you configured, then your system seems to have no Internet access at all. Does pinging or curling the plain Google IP 141.251.220.14 work?

What I cannot derive from your info is whether you had WireGuard enabled or not in which case. I guess all tests were with VPN enabled, while with VPN disabled (and DoH) it works? If Internet access itself is broken once the VPN is enabled, then probably IP forwarding is missing at the VPN server? Can you check this on the VPN server system:

sysctl net.ipv4.conf.wg0.forwarding net.ipv4.conf.eth0.forwarding
iptables -L FORWARD
iptables -L POSTROUTING -t nat

And to verify that the VPN client really is connected:

wg

However, this did work before (all but pornhub), didn't it? Probably best to test from home LAN, to take out differences of the coffee shop Internet for now.

[yuukiAme]

Hmm, yes, the "system" uses the default DNS from the ISP or whatever you configured in your LAN. However, with DoH enabled, the browser overrides this/does not use the system DNS. Hence if you cannot access Google with whichever DoH you configured, then your system seems to have no Internet access at all. Does pinging or curling the plain Google IP 141.251.220.14 work?

Yeah. I couldn't figure it out either. So I rebooted the Macbook just to be sure nothing was misconfigured.

Steps I did:

1. removed the DNS entry again from the conf file and saved the file
2. change Firefox Settings > DNS over HTTPS > Max protection > Cloudflare

outcome:

• I have internet again.
• google.com works
• nslookup google works
• curl 142.250.207.78 works > https://www.google.com
• local gateway works in the browser

it might have been a bad cache or improper setting during the whole up/down from the wg0 interface.

If Internet access itself is broken once the VPN is enabled, then probably IP forwarding is missing at the VPN server? Can you check this on the VPN server system:

root@DietPi:~# sysctl net.ipv4.conf.wg0.forwarding net.ipv4.conf.eth0.forwarding
iptables -L FORWARD
iptables -L POSTROUTING -t nat
net.ipv4.conf.wg0.forwarding = 1
net.ipv4.conf.eth0.forwarding = 1
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  10.9.82.0/24         anywhere             /* wireguard-nat-rule */

I don't see the wg you're referring to in the output. But I can confirm that 10.9.82.0/24 is used by my PiVPN WG

I'm back at home. I will use my secondary WAN (no Pi-hole, unbound DoT, AGH) as an outside network to VPN if you want. Or should I use the 1st network (with Pi-hole, etc) to test the VPN and www.pornhub.com again?

[yuukiAme]

Extra comment related to this test above

Steps I did:

removed the DNS entry again from the conf file and saved the file change Firefox Settings > DNS over HTTPS > Max protection > Cloudflare outcome:

I have internet again. google.com works nslookup google works curl 142.250.207.78 works > https://www.google.com/ local gateway works in the browser

All this over PiVPN Wireguard without the DNS entry in the conf file.

nslookup www.pornhub.com Address: 127.0.0.1 which is blocked by default DNS (expected)

nslookup store.steampowered.com Address: 127.0.0.1

nslookup xhamster.com Address: 127.0.0.1

Firefox browser - enabled Max Protection > Cloudflare

website	status
pornhub.com	timed out
xhamster.com	working as intended
store.steampowered.com	working as intended

So my ISP really did something to pornhub.com

[MichaIng]

Okay, so now everything but pornhub works with VPN enabled and DoH. And when you disable the VPN now, can pornhub be accessed via browser?

What I want to find out is whether the VPN alone, taking DNS out of the game, has an effect on whether pornhub can be accessed or not.

wg is a command from WireGuard to see which clients was or is connected.

[yuukiAme]

And when you disable the VPN now, can pornhub be accessed via browser?

Steps I did:

sudo wg-quick down wg0 - so VPN is down.

nslookup google.com to confirm it's the DNS switched back to Default DNS by ISP. It did and returned 142.250.66.142

Good. I have internet.

Back to Firefox with DNS over HTTPS > Cloudflare

outcome - it timed out and never loaded on Firefox browser.

Just to be sure, I checked with xhamster.com and store.steampowered.com. They both working as intended in the browser.

as for the wg command, I turned on Wireguard again (Macbook)

ran the command, output below

sh-3.2# wg
interface: utun2
  public key: +REDACTED_LONG_STRING=
  private key: (hidden)
  listening port: 65446

peer: REDACTED_LONG_STRING=
  preshared key: (hidden)
  endpoint: <PUBLIC IP network 1 with Pi-Hole>:51820
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 12 seconds ago
  transfer: 40.92 KiB received, 20.14 KiB sent

wg on Server

peer: +REDACTED_LONG_STRING=
  preshared key: (hidden)
  endpoint: <PUBLIC IP from Network 2>:65446
  allowed ips: 10.9.82.3/32
  latest handshake: 3 minutes, 18 seconds ago
  transfer: 15.73 MiB received, 164.20 MiB sent

[MichaIng]

Hmm, so pornhub is not accessible on the Mac? Was it every accessible there? So instead of comparing VPN with no VPN, we should probably compare Mac and PC, as the PC is the only device where pornhub was ever accessible?

[yuukiAme]

so pornhub is not accessible on the Mac?

True. But if I use Proton VPN, it will work.

Was it every accessible there?

Not sure what this comment means.

we should probably compare Mac and PC, as the PC is the only device where pornhub was ever accessible?

I can confirm that my Xiaomi (Android) and Samsung Tablet (Android) did open www.pornhub.com in Firefox browser on BOTH LAN and VPN connection. Work just like my PC (Windows).

Ignore this.

Test: Devices on Pi-Hole network with Unbound, 3 websites being tested - pornhub, steam, xhamster

Device	Status
O11(Windows)	Pornhub✔Steam✔Xhamster✔
iPhone 10(iOS)	Pornhub❌Steam✔Xhamster✔
iPhone 14(iOS)	Pornhub❌Steam✔Xhamster✔
Xiaomi(Android)	Pornhub✔Steam✔Xhamster✔
Samsung(Android)	Pornhub❌Steam✔Xhamster✔
Macbook(MacOS)	Pornhub❌Steam✔Xhamster✔

Note: Earlier today, Xiaomi could not open www.pornhub.com. But it opens Steam just fine because I use the Steam app on Android. OMG. I hate these tests. Why is it full of variety? It's so fickle. So much inconsistent.

I only have a laptop that is a Mac and it uses similar commands to Linux and I can move it around to different locations so that's why I chose it to test out VPN vs no VPN.

[MichaIng]

True. But if I use Proton VPN, it will work.

Makes sense, as your ISP then does not see any traffic from your home to pornhub (IP).

Not sure what this comment means.

I mean whether pornhub was ever accessible on the Mac (when not using ProtonVPN or any other public provider).

I only have a laptop that is a Mac and it uses similar commands to Linux and I can move it around to different locations so that's why I chose it to test out VPN vs no VPN.

Good, so we know the (home) VPN is not the reason for the issue, but it also does not prevent it. And encrypted DNS does not resolve it either, since it is the HTTP(S) connection itself which fails. And for now, based on earlier curl and openssl output, pornhub themselves get and answer the request, but the TLS handshake it is somehow not successful.

EDIT: Okay, some more devices are affected, interesting that one Android works, the other one not. Also the DietPi server is affected, getting a 404 response with curl, instead of a redirect. However, so we need to find out what the difference between requests from those devices is.

Can you run the traceroute on Windows like this:

tracert www.pornhub.com

And on the Mac and/or DietPi like this:

apt install mtr
mtr www.pornhub.com

and paste the results? I want to see whether the routes differ at any point. You can mask IPs/hostnames from your LAN, of course.

mtr instead of traceroute on Linux, since in my case the latter has issues finding the route to pornhub, while mtr and tracert (on Windows) succeed, and curl delivers the fill pornhub HTML document successfully.

[yuukiAme]

I mean whether pornhub was ever accessible on the Mac (when not using ProtonVPN or any other public provider).

Never. I hardly ever use this laptop. It's an older Macbook. It has like 4 GB of RAM and Intel duo cores. Super slow. So I never used it to access pornhub.com before yesterday.

traceroute on Windows

C:\Users\PC>tracert www.pornhub.com

Tracing route to pornhub.com [66.254.114.41]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  tplinkwifi.net [192.168.16.1]
  2    21 ms    19 ms    19 ms  O11 [2x.xx.xxx.xxx]
  3     2 ms     1 ms     1 ms  10.255.39.245
  4    21 ms    21 ms    21 ms  O11 [2x.xx.xxx.xxx]
  5    20 ms    24 ms    22 ms  O11 [2x.xx.xxx.xxx]
  6    21 ms    20 ms    20 ms  O11 [2x.xx.xxx.xxx]
  7    46 ms    44 ms    44 ms  O11 [2x.xx.xxx.xxx]
  8    40 ms    40 ms    75 ms  125.xxx.xxx.xxx.isp.domain.com  [125.xxx.xxx.xxx]
  9    39 ms    39 ms    39 ms  213.248.97.50
 10     *        *        *     Request timed out.
 11    40 ms    40 ms    40 ms  62.115.137.243
 12    76 ms    76 ms    76 ms  hnk-b4-link.ip.twelve99.net [62.115.112.223]
 13   245 ms   245 ms   245 ms  tky-b3-link.ip.twelve99.net [62.115.134.187]
 14   242 ms   240 ms   241 ms  lax-b23-link.ip.twelve99.net [62.115.137.108]
 15   241 ms   240 ms   240 ms  lax-b22-link.ip.twelve99.net [62.115.143.39]
 16   245 ms   245 ms   245 ms  haproxy-ic-328269.ip.twelve99-cust.net [213.248.75.179]
 17   230 ms   230 ms   229 ms  cust-reflected-svc11502.ip.reflected.net [64.210.157.119]
 18   245 ms   245 ms   244 ms  reflectededge.reflected.net [66.254.114.41]

Trace complete.

mtr www.pornhub.com

Wow. that's a cool application/software. I like it a lot. Thanks for introducing me to mtr.

output

DietPi (192.168.16.2) -> www.pornhub.com (66.254.114.41)                                       2024-05-08T02:38:53+0700
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                               Packets               Pings
 Host                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.16.1                                                              0.0%     5    0.4   0.6   0.4   0.6   0.1
 2. localhost                                                                 0.0%     5    2.0   1.8   1.4   2.0   0.2
 3. 10.255.39.245                                                             0.0%     5    1.7   1.9   1.6   2.6   0.4
 4. localhost                                                                20.0%     5   20.1  21.5  20.1  24.6   2.1
 5. localhost                                                                 0.0%     5   22.0  24.1  20.7  27.5   3.0
 6. localhost                                                                 0.0%     4   20.3  20.5  20.3  20.9   0.3
 7. localhost                                                                 0.0%     4   44.4  44.8  44.4  45.0   0.3
 8. 125.xxx.xxx.xxx.isp.domain.com                                           0.0%     4   40.1  47.8  40.1  70.0  14.8
 9. (waiting for reply)
10. (waiting for reply)
11. 62.115.137.243                                                            0.0%     4   40.3  40.3  40.0  40.5   0.2
12. hnk-b4-link.ip.twelve99.net                                               0.0%     4   76.6  76.9  76.5  77.2   0.4
13. tky-b3-link.ip.twelve99.net                                               0.0%     4  245.7 245.7 245.6 245.8   0.1
14. lax-b23-link.ip.twelve99.net                                              0.0%     4  241.1 241.1 241.0 241.2   0.1
15. lax-b22-link.ip.twelve99.net                                              0.0%     4  245.6 245.4 245.2 245.6   0.2
16. haproxy-ic-328269.ip.twelve99-cust.net                                    0.0%     4  245.8 245.7 245.4 246.1   0.3
17. cust-reflected-svc11502.ip.reflected.net                                  0.0%     4  229.9 232.6 229.8 240.3   5.2
18. reflectededge.reflected.net

They both seem to work. Note: When I ran mtr on dietpi, the 125.xxx.xxx.xxx.isp.domain.com entry has a high rate of Loss - hovering around 75%-80%. No clue what that means.

Another test mtr on DietPi if I leave it on longer than 30 seconds

                                                 My traceroute  [v0.95]
DietPi (192.168.16.2) -> www.pornhub.com (66.254.114.41)                                       2024-05-08T03:08:42+0700
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                               Packets               Pings
 Host                                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.16.1                                                              0.0%    27    0.5   0.6   0.4   4.5   0.8
 2. localhost                                                                 0.0%    27    2.3   3.8   1.4  47.8   8.8
 3. 10.255.39.245                                                             0.0%    27    2.0   1.8   1.5   2.4   0.2
 4. localhost                                                                 0.0%    27   21.4  20.9  19.8  21.9   0.6
 5. localhost                                                                 0.0%    27   19.9  22.8  19.5  25.8   2.1
 6. localhost                                                                 0.0%    27   20.6  20.6  20.2  21.6   0.3
 7. localhost                                                                 0.0%    27   44.6  45.0  44.1  46.3   0.7
 8. 125.xxx.xxx.xxx.isp.domain.com                                             0.0%    26   40.2  40.1  39.7  40.9   0.3
 9. 213.248.97.50                                                            84.0%    26   39.5  39.5  39.4  39.7   0.1
10. sng-b7-link.ip.twelve99.net                                              96.0%    26   40.4  40.4  40.4  40.4   0.0
11. 62.115.137.243                                                            0.0%    26   40.4  40.4  39.8  42.9   0.6
12. hnk-b4-link.ip.twelve99.net                                               0.0%    26   76.5  77.3  76.2  81.4   1.5
13. tky-b3-link.ip.twelve99.net                                               0.0%    26  241.6 241.5 241.2 242.0   0.2
14. lax-b23-link.ip.twelve99.net                                              0.0%    26  241.0 241.1 240.6 243.5   0.7
15. lax-b22-link.ip.twelve99.net                                              0.0%    26  241.5 241.1 240.7 241.5   0.2
16. haproxy-ic-328269.ip.twelve99-cust.net                                    0.0%    26  240.9 247.2 240.9 255.9   3.8
17. cust-reflected-svc11502.ip.reflected.net                                  0.0%    26  227.8 227.7 225.1 258.2   6.4
18. reflectededge.reflected.net                                               0.0%    26  241.2 244.7 240.9 245.3   1.1

[yuukiAme]

After brew install mtr and dealing with mtr command not found - I had to create ~/.zshrc to deal with it. I'm good

output for Macbook - without VPN on Pi-Hole network

mac.192 (192.168.16.32) -> www.pornhub.com (66.254.114.41)       2024-05-08T02:50:42
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                      Packets               Pings
 Host                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. tplinkwifi.net                                   0.0%    16    2.2   1.6   1.2   2.3   0.3
 2. localhost                                        0.0%    16    3.2  13.9   2.5  62.3  16.9
 3. 10.255.39.245                                    0.0%    16    3.3   3.2   2.6   4.9   0.5
 4. localhost                                        0.0%    16   21.7  22.4  21.3  23.6   0.7
 5. localhost                                        0.0%    15   28.3  26.1  22.3  28.6   1.9
 6. localhost                                        0.0%    15   21.8  21.8  21.3  23.1   0.5
 7. localhost                                        0.0%    15   47.0  47.1  45.8  52.3   1.6
 8. 125.xxx.xxx.xxx.isp.domain.com                  0.0%    15   45.6  46.6  45.1  56.4   2.7
 9. 213.248.97.50                                   57.1%    15   44.9  45.1  44.9  45.2   0.1
10. (waiting for reply)
11. 62.115.137.243                                   0.0%    15   45.2  42.0  41.0  45.3   1.4
12. hnk-b4-link.ip.twelve99.net                      0.0%    15   77.5  78.5  77.4  84.4   1.8
13. tky-b3-link.ip.twelve99.net                      6.7%    15  246.4 243.5 242.4 247.4   1.7
14. lax-b23-link.ip.twelve99.net                     6.7%    15  246.2 246.4 245.9 247.2   0.4
15. lax-b22-link.ip.twelve99.net                     0.0%    15  242.9 242.2 241.5 243.0   0.4
16. haproxy-ic-328269.ip.twelve99-cust.net           0.0%    15  247.7 250.3 246.1 276.6   8.0
17. cust-reflected-svc11502.ip.reflected.net         0.0%    15  226.3 228.3 226.3 234.9   3.1
18. reflectededge.reflected.net                      0.0%    15  246.0 243.0 241.8 246.3   1.6

output for Macbook - with VPN on Pi-Hole network

mac.192 (10.9.82.3) -> www.pornhub.com (66.254.114.41)              2024-05-08T02:51:57
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                         Packets               Pings
 Host                                                  Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. pi.hole                                             0.0%    12    2.0   2.5   2.0   3.2   0.4
 2. tplinkwifi.net                                      0.0%    12    3.1   2.7   2.3   3.5   0.3
 3. localhost                                           0.0%    12    3.8   5.5   3.6  15.7   3.4
 4. 10.255.39.245                                       0.0%    12    4.2   4.2   3.7   4.9   0.4
 5. localhost                                           0.0%    12   24.3  23.6  22.1  24.6   0.7
 6. localhost                                           0.0%    12   29.1  27.0  22.7  29.9   1.9
 7. localhost                                           0.0%    12   23.1  23.0  22.4  23.7   0.4
 8. localhost                                           0.0%    11   46.8  47.5  46.1  48.9   0.9
 9. 125.xxx.xxx.xxx.isp.domain.com                     0.0%    11   42.2  42.9  41.7  43.9   0.7
10. 213.248.97.50                                      90.0%    11   46.0  46.0  46.0  46.0   0.0
11. (waiting for reply)
12. 62.115.137.243                                      0.0%    11   49.8  47.8  46.6  52.1   1.7
13. hnk-b4-link.ip.twelve99.net                         0.0%    11   78.7  80.6  78.5  90.7   3.8
14. tky-b3-link.ip.twelve99.net                         0.0%    11  257.1 249.0 247.6 257.1   2.7
15. lax-b23-link.ip.twelve99.net                        0.0%    11  245.6 243.6 242.9 245.6   0.7
16. lax-b22-link.ip.twelve99.net                        0.0%    11  243.1 243.4 242.9 243.9   0.3
17. haproxy-ic-328269.ip.twelve99-cust.net              0.0%    11  281.5 251.3 247.6 281.5  10.1
18. cust-reflected-svc11502.ip.reflected.net            0.0%    11  228.8 228.1 227.2 229.4   0.6
19. reflectededge.reflected.net                         0.0%    11  247.3 247.2 247.0 247.7   0.2

[yuukiAme]

Extra tests from Macbook

Test 3: mtr number 3 on network number 2 (not Pi-hole) no VPN - default DNS

192.168.1.106 (127.0.0.1) -> www.pornhub.com (127.0.0.1)                          2024-05-08T03:00:37
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                  Packets               Pings
 Host                                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. localhost                                                    0.0%    15    0.3   0.2   0.1   0.5   0.1

Test 4: mtr number 4 network number 2 with vpn no dns entry in conf file

192.168.1.106 (127.0.0.1) -> www.pornhub.com (127.0.0.1)                            2024-05-08T03:01:31
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                    Packets               Pings
 Host                                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. localhost                                                      0.0%    11    0.1   0.1   0.1   0.2   0.0

Test 5 : mtr number 5 network number 2 with vpn with dns entry 10.9.82.1

mac.local (10.9.82.3) -> www.pornhub.com (66.254.114.41)               2024-05-08T03:02:46
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                    Packets               Pings
 Host                                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. pi.hole                                                        0.0%    11    3.4   3.8   3.4   4.7   0.5
 2. tplinkwifi.net                                                 0.0%    11    4.2   4.2   3.5   5.1   0.4
 3. localhost                                                      0.0%    11    4.8  12.2   4.4  42.8  11.6
 4. 10.255.39.245                                                  0.0%    11    5.3   5.5   5.0   7.2   0.6
 5. localhost                                                      0.0%    11   24.0  24.6  23.6  25.9   0.9
 6. localhost                                                      0.0%    11   26.6  27.6  23.5  32.3   2.6
 7. localhost                                                      0.0%    11   23.9  24.2  23.5  26.0   0.7
 8. localhost                                                      0.0%    10   44.6  44.9  44.0  45.7   0.5
 9. 125.xxx.xxx.xxx.isp.domain.com                                 0.0%    10   49.4  51.0  47.6  71.9   7.4
10. 213.248.97.50                                                 88.9%    10   42.9  42.9  42.9  42.9   0.0
11. (waiting for reply)
12. 62.115.137.243                                                 0.0%    10   44.8  45.5  43.6  49.4   2.3
13. hnk-b4-link.ip.twelve99.net                                    0.0%    10   82.2  81.0  80.0  85.0   1.5
14. tky-b3-link.ip.twelve99.net                                    0.0%    10  249.0 249.2 248.5 250.0   0.5
15. lax-b23-link.ip.twelve99.net                                   0.0%    10  248.7 248.7 248.5 249.2   0.2
16. lax-b22-link.ip.twelve99.net                                   0.0%    10  248.8 249.0 248.4 250.2   0.5
17. haproxy-ic-328269.ip.twelve99-cust.net                         0.0%    10  244.5 244.9 244.3 246.0   0.5
18. cust-reflected-svc11502.ip.reflected.net                       0.0%    10  229.5 234.4 228.8 262.5  10.7
19. reflectededge.reflected.net                                    0.0%    10  248.0 248.3 247.9 249.1   0.4

[MichaIng]

Ah sorry, right, on Mac of course the apt install was nonsense. However, not bad to have the result from the DietPi/Linux system as well.

Yeah, mtr seems to be better than the classic traceroute, not only as it continues to monitor all nodes to get averages, but also since at least in this particular case, from the systems I can test this, traceroute was simply not able to find the route to pornhub.

ISP DNS blocks *.pornhub.com, hence all ends at localhost, makes sense. With any other DNS, the VPN adds the VPN/Pi-hole server as first hop, otherwise has no effect. And also the routes on all devices without VPN are identical. So that is not the difference. Probably it is not even the ISP which blocks this in a way, but the TLS libraries used on those systems somehow have an issue in your particular case. But I am pretty much out of ideas how to further debug.

Just one last test, on the DietPi system, which seem to have been affected as well. Can you test again with curl as well as wget? Both use different TLS libraries. If not the case, use Pi-hole for DNS resolving, as we know it is otherwise blocked by ISP DNS:

cp -a /etc/resolv.conf{,.bak}
echo 'nameserver 127.0.0.1' > /etc/resolv.conf
curl -L https://www.pornhub.com
wget -O- https://www.pornhub.com
mv /etc/resolv.conf{.bak,}

In case of success, both return the full HTML document, which is huge, so do not wonder, and no need to paste that 😄. If it ends with </html>, it was a success, that is all we need to know.

Not sure whether someone on our forum has further ideas how to debug, or in general how requests can be (made) fail in such a way. Otherwise, probably people on some StackExchange site, like https://superuser.com/, have an idea. In case, it makes sense to leave the (home) VPN out of the game. It was a reasonable idea to test whether it helps, but we showed that it is irrelevant in regards of the issue. But that using an encrypted public DNS (like DoH in browser) is required in any case, as the ISP DNS blocks it, is of course important, and that using a public VPN provider fixes the issue on the affected systems, is relevant as well. So it is somehow these particular systems or clients sending the HTTP(S) request through this particular ISP route, having issues, while other systems/clients, sending the same request through the same route, succeed. EDIT: Ah, you said the Mac fails as well when trying from the coffee shop internet access, right? So the same clients are affected as well when sending from other access points in the same region (?).

An interesting case, so in case you're doing to further debug, I am interested about the outcome. You can ping me on superuser as well (same username).

[yuukiAme]

Just one last test, on the DietPi system, which seem to have been affected as well. Can you test again with curl as well as wget? Both use different TLS libraries. If not the case, use Pi-hole for DNS resolving, as we know it is otherwise blocked by ISP DNS:

cp -a /etc/resolv.conf{,.bak}
echo 'nameserver 127.0.0.1' > /etc/resolv.conf
curl -L https://www.pornhub.com
wget -O- https://www.pornhub.com
mv /etc/resolv.conf{.bak,}

Output on DietPi

root@DietPi:~# cp -a /etc/resolv.conf{,.bak}
echo 'nameserver 127.0.0.1' > /etc/resolv.conf
curl -L https://www.pornhub.com
wget -O- https://www.pornhub.com
mv /etc/resolv.conf{.bak,}
curl: (35) Recv failure: Connection reset by peer
--2024-05-08 04:13:11--  https://www.pornhub.com/
Resolving www.pornhub.com (www.pornhub.com)... 66.254.114.41
Connecting to www.pornhub.com (www.pornhub.com)|66.254.114.41|:443... connected.
GnuTLS: The TLS connection was non-properly terminated.
Unable to establish SSL connection.

With a blinking cursor. Should I Ctrl+C to cancel the operation?

What now? I certainly could make a superuser account and ask a question there. I don't think anyone else other than you, @MichaIng is willing to debug/troubleshooting this far into an issue.

Thank you so much for your help.

[yuukiAme]

Ah, you said the Mac fails as well when trying from the coffee shop internet access, right? So the same clients are affected as well when sending from other access points in the same region (?).

Yup. Mac fails at the Coffee Shop too. Another problem you happened to remind me is my country has 3 major ISP so their DNS and DNS Filter implementation are all different. That could be why my results are varies.

What a pain. All I want is to self-host and be in control of my data yet. To have privacy. Apparently it's too much to ask in my country. Super lame.

[MichaIng]

Okay, so OpenSSL and GnuTLS both run into the same issue. So the TLS library is unrelated as well. EDIT: Fits to the 404/408 response you got with curl pornhub.com, where no TLS was involved yet, but a redirect expected.

What now? I certainly could make a superuser account and ask a question there. I don't think anyone else other than you, @MichaIng is willing to debug/troubleshooting this far into an issue.

There is no guarantee, of course, and on StackExchange sites it is a bit RNG and timing, whether someone with interest and knowledge sees the question or not while it is new/highlighted. But there are certainly people around willing to help/answer questions, with much more knowledge/experience than me. So it is worth an attempt, if you do not mind the time to create an account, read a bit into how to ask a question properly and sort/sum up the relevant results we found.

What a pain. All I want is to self-host and be in control of my data yet. To have privacy. Apparently it's too much to ask in my country. Super lame.

Well, DNS and probably packet filtering by ISP is not directly related to whether you can self-host things or not? You can self-host a VPN and a DNS for DoT or DoH perfectly fine. So you can bypass DNS filtering that way, or use DoH with a public provider instead. Just sadly it does not help with the HTTPS request issue, whatever is causing this.

However, yeah, any kind of Internet filtering can be nasty. I was in China for a week, and it was interesting how many websites were blocked, not only raw.githubusercontent.com, while github.com works well (same in India, as far as I read, weird somehow), but also all Google services, including Gmail, Cloudflare, and even DuckDuckGo (my default search machine), for whatever reason. Microsoft services and Bing however do work. One can bypass the issue with a VPN, which is not strictly forbidden, especially for western visitors who often simply require it (our DietPi mails go through Gmail, currently). But even then, for some reasons, some websites are suspiciously slow, nearly unusable (when using DoH to bypass DNS block), while others work perfectly fine. Whether intended or not, at least I had the impression that it was intended, and your issue reminded me about this.

[yuukiAme]

@MichaIng I’m reading up on StackExchanger Superuser rules and tags.

Which tags do you think this issue belongs to? Because I know their mods will remove issues or could straight up ban me if I tried to resubmit and commit a “spam” offense against their site.

[yuukiAme]

@MichaIng

Superuser question posted

They closed it really quick with off-topic. The 2nd comment gave me some thoughts. Is Cloudflare respecting my country content filtering as well? How do I switch all my query to Quad9 to test DNS-over-HTTPS properly?

[Joulinar]

How do I switch all my query to Quad9 to test DNS-over-HTTPS

I guess you would need to use cloudflared instead of Unbound. https://docs.pi-hole.net/guides/dns/cloudflared/

[yuukiAme]

@Joulinar Thanks. That was my thoughts after that answer. Been researching through our DietPi old issues. Someone mentioned in 2018, Cloudflared was being shaddy with their CDN and bad for privacy. I'm not sure if their opinion changed since 2018. I'm also looking into DNScrypt-proxy and ODOH.

[Joulinar]

Cloudflared can be used with Quad9 as well. It's just an application that can be used with various upstream DNS provider.

[yuukiAme]

Yes. I understood that. The pi-hole Docs did mentioned the https upstream that is using Cloudflare DNS. Of course I could change it over to Quad9. I'm just making sure I'm learning all of the other options as well.

[yuukiAme]

Update: It seems that after I switched everything to Quad9, from the DietPi-Config, to the Cloudflared, set up with Quad9 DoH as upstream DNS, Wireguard app into my PiVPN, www.pornhub.com still doesn't work over the VPN. So I installed Intra on my Android Tablet, selected Quad9 as the DNS over HTTPS server , erased the history and cache on my browser. I can access www.pornhub.com now.

Now I'm not sure what to think now. The issue can be closed I guess.

[MichaIng]

They are wrong: Cloudflare does not respect country filtering, at least not in your case. We do already know that clients get the correct IP from Cloudflare, and the connection is blocked on HTTP(S) connection level to/from the true host instead.

Not sure whether it is really OOT at superuser, but I wouldn't know a better StackExchange site in this case. I left a comment to ask about this, and add the relevant info, let's see ...

[MichaIng]

https://meta.superuser.com/questions/15219/where-to-ask-questions-about-non-dns-based-country-based-website-http-blocking

[yuukiAme]

@MichaIng @Joulinar

Good news.

I went nuts after drinking too many cups of coffee. So I did a factory reset on an Android device (Xiaomi) to reduce all variables. So the phone is fresh. Only using network number 2 so no Pi-hole and Cloudflared. Only Default DNS from ISP.

I installed 1.1.1.1 app from Google Play Store. Install VPN profile, uses the Google Chrome that came with phone. To switch to DNS-over-HTTPS. Checked with 1.1.1.1/help to confirm that it is using cloudflare DoH. I tested the five websites mentioned in the Superuser question.

They all work now. No delays. Instant content delivery.

I formed a hypothesis it could the Adguard.apk and maybe its the adguardca.crt. I uses Adguard app on all my devices. Except for my PC because everything works on it, I have uBlock Origin on Firefox to deal with ads and Pi-Hole to block the domains completely.

Background: The AdGuard app requires the user to install a CA certificate provided by AdGuard to enhanced the ad blocks and DNS filters. Otherwise, Firefox browser will throw HSTS / SSL errors. It required the same on MacOS with a custom user cert added in the Keychain. I think I found the problem.

To test out the hypothesis, I removed the CA cert from my Samsung tablet and uninstalled the Adguard.apk. Reboot the device. Install 1.1.1.1 app, install VPN profile. This device nows can open the 5 websites above as well.

Big progress.

Now that solves half the issue. Because MacOS and iPhone are still failing to open www.pornhub.com and sometimes with medium.com even without Adguard and its VPN profile. No clue why. It's not iCloud+ private relay.

Either Pornhub SSL/TLS certificate has a problem verify its identity with Adguard CA cert. Not too sure on the traffic and route from the ISP side.

Do you have any methods to debug related to SSL/TLS cert instead? Should I ask the expert over in the Unbound issues for help? He was pretty good at looking at my unbound journalctl log.

I can finally close this issue. Looks like I will have to remove all Adguard app and its CA from the trusted CA.

EDIT: I forgot. I need to test this over VPN as well. Before closing this issue.

[MichaIng]

Okay that is a very good explanation: Note that HTTPS filtering/inspection means that AdGuard decrypts, filters and in case blocks all traffic/packets between you and the website you want to access. This includes porn, online banking (aside of some hardcoded exceptions), etc. So I am quite sure that this AdGuard HTTPS filter/firewall rates some pornhub content as malicious, block it, hence replaces it with these weird 404/408 HTML documents I was wondering about. The only thing I am wondering about, is why the DietPi system was affected by this as well.

The way this is implemented, basically compromises HTTPS at its core. It is like a MITM, which HTTPS normally prevents against, but in this case you are explicitly asked to trust the AdGuard certificate which re-encrypts traffic. I mean, AdGuard software is (mostly or completely?) open source, and we have no reason to mistrust them (now). But IMO the principle to break the essential security of HTTPS in the name of security is just wrong. And I am not alone with this opinion.

We are using Cloudflare as proxy/CDN/cache/DDoS protection for HTTP(S) traffic to our website, which basically does the same. However, it does so transparently, i.e. resolving dietpi.com gives you IP addresses from the known Cloudflare Edge servers, the related TLS certificate is controlled by use, can be enabled/disabled/revoked, it adds request and response headers to tell clients what they are connecting to etc. And most importantly, DietPi is not a sensitive/privacy concerning topic, and everything you can do on our website is/will be public anyway. Bug report and survey uploads, done via SFTP, bypass Cloudflare, btw. But I would be at least concerned about an app/actor which decrypts/reads really all traffic to any website/web service 😉.

[yuukiAme]

This includes porn, online banking (aside of some hardcoded exceptions), etc. So I am quite sure that this AdGuard HTTPS filter/firewall rates some pornhub content as malicious, block it, hence replaces it with these weird 404/408 HTML documents I was wondering about.

This might explain it. If this is true, when I reinstall Adguard.apk, install the adguard-ca.crt, I turn off the dns filtering and firewall, it should allow my devices to start opening the HTTPS to the 5 websites I mentioned above and content delivery should work as intended.

But the weird part in this is why bbc.com and medium.com fails as well. Their websites shouldn't be on their DNS filtering, right? I only my ISP DNS is interested in blocking those traffic.

The way this is implemented, basically compromises HTTPS at its core. It is like a MITM, which HTTPS normally prevents against, but in this case you are explicitly asked to trust the AdGuard certificate which re-encrypts traffic. I mean, AdGuard software is (mostly or completely?) open source, and we have no reason to mistrust them (now). But IMO the principle to break the essential security of HTTPS in the name of security is just wrong. And I am not alone with this opinion.

I will email their support and create a ticket asking for clarification on their ca crt and dns filtering. Decrypting my HTTPS traffic on their end and see my query is obvious concerning. But I understand why it had to be done. How else Adguard suppose to know if that specific traffic needed to block and not delivery to client device.

But I would be at least concerned about an app/actor which decrypts/reads really all traffic to any website/web service 😉.

Viewing the traffic is just one problem. If they are storing customer traffic and still linking it to me, and my government happens to required them by law to hand it over is another problem.

I've looking for ways to encrypt traffic between client (PC) to DNS resolver (Pi-hole or Adguard Home). The internet pointed me to dnscrypt.info and there are so many implentation between the client implementation vs server implementation that I don't know which one to uses. Any suggestions?

[MichaIng]

No need to ask AdGuard about that: It is intended behaviour, to block ads and/or block traffic to malicious sites. But yes, you are right, bbc.com and medium.com should of course not be blocked, at best, in case, only ads embedded in these websites. And the DietPi system should not have been affected. So probably the AdGuard HTTPS filtering was only one part of the issue.

Filtering will be done locally on an automated level, hence your traffic is, of course, not sent to AdGuard servers where they could read it. Though, not impossible that bits of information are sent to be checked by a cloud block list or heuristic. At least in general this is how some anti virus software do/did it. Though for ads filtering such is surely overkill an not done.

So for me, the problem is not concerns or mistrust against AdGuard now, but the method of decrypting HTTPS traffic, and that way breaking end-to-end encryption as well as authentication (installing a generally trusted cert for a website that is not coming from this website), in general. Actors who are trustable now can change, be hacked, or whatever, so having such method installed and generally accepted as "security enhancement" IMO bears a general risk for the future. There may be edge cases, but not as by default enabled feature for common end user software, IMO.

I've looking for ways to encrypt traffic between client (PC) to DNS resolver (Pi-hole or Adguard Home).

Downstream DoH or DoT with AdGuard Home and/or WireGuard/VPN with Pi-hole/AGH as you were already doing.

[yuukiAme]

I have since reinstall Adguard.apk from their Adguard website. Installed their CA.crt again to my Xiaomi.

I did a bit more research. In Adguard app Settings > Filtering > Network > HTTPS Filtering > HTTPS-Filtered websites > turn OFF Filter websites with EV certificates.

This is the real reason why 3 out of the 5 websites mentioned above broke. Not sure why DietPi itself broke either. I guess I can close this issue now.

Thank you for your time and knowledge @MichaIng and @Joulinar . This has been a very productive week and a ton of learning experience for me.