Add an option to disable HTTP/2

[online2311]

Fastly error: unknown domain: github.com

This prompt often appears as if it is not for the reason that http2 is not supported.

[ameshkov]

Could you please show the configuration that you're using and explain how to reproduce this error?

[online2311]

docker run -d --name sniproxy --net host --restart=always ameshkov-sniproxy:v1.4 --dns-redirect-ipv4-to=100.106.241.93 --dns-redirect-ipv6-to=fd7a:115c:a1e0:ab12:4843:cd96:626a:f15d --dns-upstream=1.1.1.1

There are a lot of api.github.com requests in the logs

2023/06/23 15:22:23 [info] sniproxy: [69] start tunneling to api.github.com:443
2023/06/23 15:22:24 [info] sniproxy: [69] finished tunneling to api.github.com:443. received 4250, sent 568, elapsed: 1.029453723s, rate (bytes/sec): 4680.152097
2023/06/23 15:22:29 [info] dnsproxy: rewriting DNS for AAAA api.github.com.
2023/06/23 15:22:29 [info] dnsproxy: rewriting DNS for A api.github.com.
2023/06/23 15:22:30 [info] sniproxy: [70] start tunneling to api.github.com:443
2023/06/23 15:22:31 [info] sniproxy: [70] finished tunneling to api.github.com:443. received 4249, sent 568, elapsed: 1.552524491s, rate (bytes/sec): 3102.688575
2023/06/23 15:22:37 [info] dnsproxy: rewriting DNS for AAAA api.github.com.
2023/06/23 15:22:37 [info] dnsproxy: rewriting DNS for A api.github.com.
2023/06/23 15:22:37 [info] sniproxy: [71] start tunneling to api.github.com:443
2023/06/23 15:22:39 [info] sniproxy: [71] finished tunneling to api.github.com:443. received 4250, sent 568, elapsed: 1.481952536s, rate (bytes/sec): 3251.116269
2023/06/23 15:22:44 [info] dnsproxy: rewriting DNS for A api.github.com.
2023/06/23 15:22:44 [info] dnsproxy: rewriting DNS for AAAA api.github.com.
2023/06/23 15:22:44 [info] sniproxy: [72] start tunneling to api.github.com:443
2023/06/23 15:22:46 [info] sniproxy: [72] finished tunneling to api.github.com:443. received 4250, sent 568, elapsed: 1.469098207s, rate (bytes/sec): 3279.562916
2023/06/23 15:22:51 [info] dnsproxy: rewriting DNS for AAAA api.github.com.
2023/06/23 15:22:51 [info] dnsproxy: rewriting DNS for A api.github.com.
2023/06/23 15:22:51 [info] sniproxy: [73] start tunneling to api.github.com:443
2023/06/23 15:22:52 [info] sniproxy: [73] finished tunneling to api.github.com:443. received 4249, sent 568, elapsed: 1.284277244s, rate (bytes/sec): 3750.747763
2023/06/23 15:22:58 [info] dnsproxy: rewriting DNS for AAAA api.github.com.
2023/06/23 15:22:58 [info] dnsproxy: rewriting DNS for A api.github.com.
2023/06/23 15:22:58 [info] sniproxy: [74] start tunneling to api.github.com:443
2023/06/23 15:23:00 [info] sniproxy: [74] finished tunneling to api.github.com:443. received 4250, sent 568, elapsed: 1.502674042s, rate (bytes/sec): 3206.284174
2023/06/23 15:23:05 [info] dnsproxy: rewriting DNS for AAAA api.github.com.
2023/06/23 15:23:05 [info] dnsproxy: rewriting DNS for A api.github.com.

curl 'https://api.github.com/_private/browser/stats' \\n  -H 'authority: api.github.com' \\n  -H 'accept: */*' \\n  -H 'accept-language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6' \\n  -H 'cache-control: no-cache' \\n  -H 'content-type: text/plain;charset=UTF-8' \\n  -H 'cookie: logged_in=yes; dotcom_user=online2311; _octo=GH1.1.1284312577.1682263906; color_mode=%7B%22color_mode%22%3A%22auto%22%2C%22light_theme%22%3A%7B%22name%22%3A%22light%22%2C%22color_mode%22%3A%22light%22%7D%2C%22dark_theme%22%3A%7B%22name%22%3A%22dark%22%2C%22color_mode%22%3A%22dark%22%7D%7D; preferred_color_mode=light; tz=Asia%2FShanghai' \\n  -H 'dnt: 1' \\n  -H 'origin: https://github.com' \\n  -H 'pragma: no-cache' \\n  -H 'referer: https://github.com/ameshkov/sniproxy/issues/3' \\n  -H 'sec-ch-ua: "Not.A/Brand";v="8", "Chromium";v="114", "Microsoft Edge";v="114"' \\n  -H 'sec-ch-ua-mobile: ?0' \\n  -H 'sec-ch-ua-platform: "macOS"' \\n  -H 'sec-fetch-dest: empty' \\n  -H 'sec-fetch-mode: no-cors' \\n  -H 'sec-fetch-site: same-site' \\n  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58' \\n  --data-raw '{"stats": [{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","cls":0.021481644359990906,"ssr":false,"lazy":false,"alternate":false}],"timestamp":1687519588458,"loggedIn":true,"staff":false,"bundler":"webpack"},{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","lcp":1754.3999999985099,"ssr":false,"lazy":false,"alternate":false}],"timestamp":1687519588461,"loggedIn":true,"staff":false,"bundler":"webpack"},{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","hpc":1754.3999999985099,"soft":false,"ssr":false,"mechanism":"HARD","lazy":false,"alternate":false,"hpcFound":true}],"timestamp":1687519588462,"loggedIn":true,"staff":false,"bundler":"webpack"},{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","hpc":1754.3999999985099,"soft":false,"ssr":false,"mechanism":"HARD","lazy":false,"alternate":false,"hpcFound":false}],"timestamp":1687519588463,"loggedIn":true,"staff":false,"bundler":"webpack"}] }' \\n  --compressed
 1396  curl 'https://api.github.com/_private/browser/stats' \ \n  -H 'authority: api.github.com' \\n  -H 'accept: */*' \\n  -H 'accept-language: zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6' \\n  -H 'cache-control: no-cache' \\n  -H 'content-type: text/plain;charset=UTF-8' \\n  -H 'cookie: logged_in=yes; dotcom_user=online2311; _octo=GH1.1.1284312577.1682263906; color_mode=%7B%22color_mode%22%3A%22auto%22%2C%22light_theme%22%3A%7B%22name%22%3A%22light%22%2C%22color_mode%22%3A%22light%22%7D%2C%22dark_theme%22%3A%7B%22name%22%3A%22dark%22%2C%22color_mode%22%3A%22dark%22%7D%7D; preferred_color_mode=light; tz=Asia%2FShanghai' \\n  -H 'dnt: 1' \\n  -H 'origin: https://github.com' \\n  -H 'pragma: no-cache' \\n  -H 'referer: https://github.com/ameshkov/sniproxy/issues/3' \\n  -H 'sec-ch-ua: "Not.A/Brand";v="8", "Chromium";v="114", "Microsoft Edge";v="114"' \\n  -H 'sec-ch-ua-mobile: ?0' \\n  -H 'sec-ch-ua-platform: "macOS"' \\n  -H 'sec-fetch-dest: empty' \\n  -H 'sec-fetch-mode: no-cors' \\n  -H 'sec-fetch-site: same-site' \\n  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.58' \\n  --data-raw '{"stats": [{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","cls":0.021481644359990906,"ssr":false,"lazy":false,"alternate":false}],"timestamp":1687519588458,"loggedIn":true,"staff":false,"bundler":"webpack"},{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","lcp":1754.3999999985099,"ssr":false,"lazy":false,"alternate":false}],"timestamp":1687519588461,"loggedIn":true,"staff":false,"bundler":"webpack"},{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","hpc":1754.3999999985099,"soft":false,"ssr":false,"mechanism":"HARD","lazy":false,"alternate":false,"hpcFound":true}],"timestamp":1687519588462,"loggedIn":true,"staff":false,"bundler":"webpack"},{"webVitalTimings":[{"name":"https://github.com/ameshkov/sniproxy/issues/3","app":"rails","hpc":1754.3999999985099,"soft":false,"ssr":false,"mechanism":"HARD","lazy":false,"alternate":false,"hpcFound":false}],"timestamp":1687519588463,"loggedIn":true,"staff":false,"bundler":"webpack"}] }' \\n  --compressed -vvl

By crawling the request it was found that the request was redirected to the http/2 protocol, and then it was not accessible to keep doing this.

[online2311]

When "api.github.com" is inaccessible

Fastly error: unknown domain: github.com. Please check that this domain has been added to a service.
Details: cache-bfi-kbfi7400039-BFI

[online2311]

This issue should be a problem with the github.com CDN mechanism, but the api.github.com interface cannot be downgraded to http/1.1 to request it, and every request is redirected.

[online2311]

https://github.com/dlundquist/sniproxy/issues/178 I found out that other projects have similar problems caused by SNIPROXY, which also does not support http/2. I have transferred the corresponding domain name to public DNS resolution alone and it works fine, without using SNIPROXY for this domain.

[ameshkov]

Actually, let me reopen this issue as it's rather interesting.

We're dealing with HTTP/2 connection coalescing here: https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing/

The root cause of the issue is that every domain name is getting redirected to the same IP address and when the browser sees that several domains that use the same wildcard certificate are located on the same IP, it may try coalescing. I.e. it may try to send request to api.github.com to a connection that was originally opened for processing requests to github.com.

I guess the only thing that can be done is downgrading HTTP/2 connections. It's possible by modifying the first TLS packet ClientHello and removing h2 from the ALPN extension before sending it further.

The fix may be done here, it's not trivial though.

In addition to that it may be required to suppress type=HTTPS queries in the DNS proxy as they may also be used to indicate supported protocols.

[online2311]

Yes, add an option to disable HTTP/2. This is a better solution than supporting HTTP/2.

[conblem]

Hi Everybody

As far as I can tell, for better or worse you can't change the Client Hello atleast with TLS 1.3. Inside Server Handshake Keys Calc the Server uses the hash of the ClientHello for further encryption. In practice I also hit this limitation, I am currently writing a similar proxy in Rust. Changing or removing the ALPN causes TLS Errors.

I will try to downgrade the connection to TLS 1.3 and see if that works. Would still be nice if TLS 1.3 would work...

[conblem]

Small update:

This also won't work with TLS 1.2. The client calculates the hash of all the Handhshake messages in Client Handshake Finished, the same goes for the Server in the step afterwards. Regardless this would not have made a difference as TLS 1.3 seems to have downgrade protections in place, as described in this Article.

If I'm not totally mistaken this issue can be closed because this can't feasibly be implemented. Cheers I am interested to hear any thoughts!

[ameshkov]

Thanks for the tip, this is indeed quite an issue and it's hard to say what can be done about that.

The only thing that comes to my mind is tweaking DNS HTTPS record as ALPN can also be specified there. The problem is that HTTPS records at the moment are only used by Apple devices.