connect-proxy example isn't working with DNS Over HTTPS

NghiaTranUIT commented 3 years ago

Expected behavior

Connect-Proxy example should work with DNS Over HTTPS -> Can access the Internet as usual

Actual behavior

Could not connect to certain domains

2020-12-08T10:18:44+0700 error com.apple.nio-connect-proxy.ConnectHandler : remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:51759) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) channel=ObjectIdentifier(0x000000010840c7a0) Connect failed: NIOConnectionError(host: "in.appcenter.ms", port: 443, dnsAError: Optional(NIO.SocketAddressError.unknown(host: "in.appcenter.ms", port: 443)), dnsAAAAError: Optional(NIO.SocketAddressError.unknown(host: "in.appcenter.ms", port: 443)), connectionErrors: [])

2020-12-08T10:18:44+0700 error com.apple.nio-connect-proxy.ConnectHandler : channel=ObjectIdentifier(0x0000000100c0b770) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:51760) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) Connect failed: NIOConnectionError(host: "google.com", port: 443, dnsAError: Optional(NIO.SocketAddressError.unknown(host: "google.com", port: 443)), dnsAAAAError: Optional(NIO.SocketAddressError.unknown(host: "google.com", port: 443)), connectionErrors: [])

2020-12-08T10:18:44+0700 error com.apple.nio-connect-proxy.ConnectHandler : remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:51855) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) channel=ObjectIdentifier(0x000000010800a3a0) Connect failed: connectTimeout(NIO.TimeAmount(nanoseconds: 10000000000))

It happens 50-50 all the time. Sometimes I could connect, but sometimes I couldn't and get a bunch of errors.
If I delete the DNS Profile -> 100% it's working

Steps to reproduce

Download DNS Profile at https://github.com/paulmillr/encrypted-dns
Install cloudflare-https.mobileconfig on Big Sur macOS
Go to System Preferrrence -> Profile and install it again
Verify that you can open any websites from Safari and Google Chrome as usual`
Set HTTP/HTTPS Proxy from Wifi -> Advanced -> Proxies to 9090
Start connect-proxy at port 9090
Quit Safari or Google Chrome if it's opening. Try to access some well-known websites, such as google.com, stackoverflow.com, ...

version/commit hashes from all involved dependencies

The lasted commit on the main branch (https://github.com/apple/swift-nio-examples/commit/09044ff88e506377c51ade0b0112d3054d0021f6)

Swift & OS version (output of `swift --version && uname -a`)

Apple Swift version 5.3.1 (swiftlang-1200.0.41 clang-1200.0.32.8)
Target: x86_64-apple-darwin20.1.0
Darwin Nghias-MacBook-Pro.local 20.1.0 Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64 x86_64

Thank Swift NIO team for developing a powerful tool 👍

Lukasa commented 3 years ago

Thanks @NghiaTranUIT!

This is a surprising behaviour to me, I definitely don't know the macOS DNS configuration well enough to understand this issue. I'll consult with some colleagues and get back to you.

Lukasa commented 3 years ago

Ok, I haven't yet reproduced this today. I'll do some more investigation tomorrow. Please also do file Feedback Assistant reports and note their numbers here.

Lukasa commented 3 years ago

Out of curiosity @NghiaTranUIT, when you run this are you seeing DNS lookups to Cloudflare domains?

NghiaTranUIT commented 3 years ago

Yes, I can see they call CONNECT cloudflare-dns.com:443 HTTP/1.1

-12-09T18:59:58+0700 info com.apple.nio-connect-proxy.ConnectHandler : localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) channel=ObjectIdentifier(0x0000000108b04230) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:57790) CONNECT cloudflare-dns.com:443 HTTP/1.1
2020-12-09T18:59:58+0700 info com.apple.nio-connect-proxy.ConnectHandler : localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) channel=ObjectIdentifier(0x0000000108b04230) remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:57790) Connecting to cloudflare-dns.com:443
2020-12-09T18:59:59+0700 info com.apple.nio-connect-proxy.ConnectHandler : remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:57791) channel=ObjectIdentifier(0x0000000100e07210) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) CONNECT p63-caldav.icloud.com:443 HTTP/1.1
2020-12-09T18:59:59+0700 info com.apple.nio-connect-proxy.ConnectHandler : remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:57791) channel=ObjectIdentifier(0x0000000100e07210) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) Connecting to p63-caldav.icloud.com:443
2020-12-09T19:00:00+0700 info com.apple.nio-connect-proxy.ConnectHandler : remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:57792) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) channel=ObjectIdentifier(0x0000000108b06000) CONNECT cloudflare-dns.com:443 HTTP/1.1
2020-12-09T19:00:00+0700 info com.apple.nio-connect-proxy.ConnectHandler : remoteAddress=Optional([IPv4]127.0.0.1/127.0.0.1:57792) localAddress=Optional([IPv4]127.0.0.1/127.0.0.1:9090) channel=ObjectIdentifier(0x0000000108b06000) Connecting to cloudflare-dns.com:443

After a while, I'm still received a bunch of Timeout Error like the first comment. Stop and Start the project again, I'm able to visit those websites as usual. Like 50% reproducible rate.

Full log: log.txt

Lukasa commented 3 years ago

Are you seeing those actually work? That is, do those DNS lookups ever succeed?

NghiaTranUIT commented 3 years ago

I just noticed that the CloudFlare profile is unsiged. Not sure if it's related 🤔

NghiaTranUIT commented 3 years ago

Here is the success log: You can see the app call cloudflare-dns.com and there is no error log. I'm able to access google.com and another website as usual.

success_log.txt

If I start/stop the project, 50% works, 50% doesn't work for some reason 🤔

Lukasa commented 3 years ago

I think we have a recursive DNS issue. I think if the DNS lookup for Cloudflare-dns is cached then we don't have an issue, but if we try to do a new lookup we get stuck.

Lukasa commented 3 years ago

@NghiaTranUIT Can I ask you to try adding cloudflare-dns.com to the excluded hosts list for the HTTPS proxying to see if that resolves the issue?

NghiaTranUIT commented 3 years ago

I added cloudflare-dns.com and www.google.com to the exclusive list, but for some reason, I still can see cloudflare-dns.com from the console log, but not www.google.com

Result:

Still see cloudflare DNS traffic in the console log -> Look like it does not by-pass the proxy
I'm able to access www.google.com (since it's in the exclusive list)
I could not access another website, such as https://news.ycombinator.com (see the error log)

Log: log.txt

Lukasa commented 3 years ago

That’s really useful information @NghiaTranUIT, thanks!

apple / swift-nio-examples