Some SSL sites bypass intercept proxy

[SmashGuy2]

Hi everyone,

When using GoProxy in transparent intercept mode, I find that Google-related sites bypass the proxy after the initial load.

Example: https://golang.org/doc/

When I query this site the first time, it shows up in the log. Then when doing a hard refresh, or even navigating to new pages that I have never loaded before from the same domain, they bypass the proxy.

Any ideas as to what may be going on?

[elazarl]

Did you try MITM all sites?

On Thu, Jul 20, 2017, 12:34 AM SmashGuy2 notifications@github.com wrote:

Hi everyone,

When using GoProxy in transparent intercept mode, I find that Google-related sites bypass the proxy after the initial load.

Example: https://golang.org/doc/

When I query this site the first time, it shows up in the log. Then when doing a hard refresh, or even navigating to new pages that I have never loaded before from the same domain, they bypass the proxy.

Any ideas as to what may be going on?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/elazarl/goproxy/issues/232, or mute the thread https://github.com/notifications/unsubscribe-auth/AAP4ooI0s5TH-4kkHuc3BDzpbKfWqQrbks5sPnZUgaJpZM4OdVAg .

[mzimmerman]

What makes you think that it isn't being proxied?

On Wed, Jul 19, 2017 at 8:47 PM Elazar Leibovich notifications@github.com wrote:

Did you try MITM all sites?

On Thu, Jul 20, 2017, 12:34 AM SmashGuy2 notifications@github.com wrote:

Hi everyone,

When using GoProxy in transparent intercept mode, I find that Google-related sites bypass the proxy after the initial load.

Example: https://golang.org/doc/

When I query this site the first time, it shows up in the log. Then when doing a hard refresh, or even navigating to new pages that I have never loaded before from the same domain, they bypass the proxy.

Any ideas as to what may be going on?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/elazarl/goproxy/issues/232, or mute the thread < https://github.com/notifications/unsubscribe-auth/AAP4ooI0s5TH-4kkHuc3BDzpbKfWqQrbks5sPnZUgaJpZM4OdVAg

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/elazarl/goproxy/issues/232#issuecomment-316564005, or mute the thread https://github.com/notifications/unsubscribe-auth/ACTSU_UHJSzy46Bp4YdX-5eDJ5pd59TEks5sPqOGgaJpZM4OdVAg .

[SmashGuy2]

Did you try MITM all sites?

Incoming requests made from Chrome to Google.com are not even being seen by the proxy. They appear to bypass it. All other https requests are in fact being proxied correctly.

Interestingly, requests to Google.com from Microsoft Edge are being proxied correctly.

Could they be using some other kind of protocol?

What makes you think that it isn't being proxied?

I'm watching the log and have put in debugging printfs to indicate incoming TLS requests. I also inject a small comment into the HTML response which I can see from within the browser source. This comment appears on virtually all websites, just not on requests made to Google websites from Chrome.

[SmashGuy2]

I don't know if this will help, but here's the output from the Network tab when I query Google.com:

Request URL:https://www.google.com/ Request Method:GET Status Code:200 Remote Address:216.58.192.164:443 Referrer Policy:no-referrer-when-downgrade

Response Headers alt-svc:quic=":443"; ma=2592000; v="39,38,37,36,35" cache-control:private, max-age=0 content-encoding:gzip content-type:text/html; charset=UTF-8 date:Thu, 20 Jul 2017 17:08:38 GMT expires:-1 server:gws set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com status:200 strict-transport-security:max-age=86400 x-frame-options:SAMEORIGIN x-xss-protection:1; mode=block

Request Headers :authority:www.google.com :method:GET :path:/ :scheme:https accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8 accept-encoding:gzip, deflate, br accept-language:en-US,en;q=0.8 cache-control:max-age=0 cookie:OTZ=3952649_76_80_104160_76_446820; HSID=A3sAVoYOI1NNIMc49; SSID=Ayw5AUmhQMxPu9gcd; APISID=Uc-Ekk5CNnAypcwr/AqxldKvivOi9z5AUz; SAPISID=c_6nd_PxebrccpwL/AtCYtStUWYz3ce0vT; SID=7gSyVXzBz4T3bpHm6x_IA7iX5uubehDqyIZlMx8d8DdKSL8Lh5r89Qg6qco8VehQPbXu6w.; S=quotestreamer=jl1d9K770O1zS6T6R0lRHgaJ3YVq0E9d; NID=108=PNkAkSwC1t5S3XiLroiBW2dPp2nZrhNaaq5mks8ArBX3lFYOWxeyQ5DjNNnpq_TU7aHeZ1JF8MtN3HGLHGnNQzIrVT7PrGInchzSw4e8PhmfBS2dZNDT48yUAVjcqwopn9WZFLUsCtwJXImAYc40w3LDfl0WKfBIhDIEtqzQkz2540YibkkVMGIiBxAWez7bn1aN0ODb4OnqLnRr1CseAc7ukWu2tf_Oaip8tJbPKZgRM2QvXCs67DJH82rzevtFQQEJop4zU0A3lw; DV=s59SNxs2SF5McBdH4fogMB8ly4wP1pVMxolFnT2lDQEAACBqIcFJ--6B2wAAANwIt5sdk6JbVQAAAA dnt:1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 x-chrome-uma-enabled:1 x-client-data:CI+2yQEIo7bJAQjEtskBCP6ZygEI+5zKAQipncoB

[elazarl]

Isn't it HSTS

On Thu, Jul 20, 2017, 8:09 PM SmashGuy2 notifications@github.com wrote:

I don't know if this will help, but here's the output from the Network tab when I query Google.com:

Request URL:https://www.google.com/ Request Method:GET Status Code:200 Remote Address:216.58.192.164:443 Referrer Policy:no-referrer-when-downgrade Response Headers alt-svc:quic=":443"; ma=2592000; v="39,38,37,36,35" cache-control:private, max-age=0 content-encoding:gzip content-type:text/html; charset=UTF-8 date:Thu, 20 Jul 2017 17:08:38 GMT expires:-1 server:gws set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=. google.com set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=. www.google.com set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain= www.google.com set-cookie:OTZ=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain= google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain= google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain= www.google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=. google.com set-cookie:DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=. www.google.com status:200 strict-transport-security:max-age=86400 x-frame-options:SAMEORIGIN x-xss-protection:1; mode=block Request Headers :authority:www.google.com :method:GET :path:/ :scheme:https

accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng, /;q=0.8 accept-encoding:gzip, deflate, br accept-language:en-US,en;q=0.8 cache-control:max-age=0 cookie:OTZ=3952649_76_80_104160_76_446820; HSID=A3sAVoYOI1NNIMc49; SSID=Ayw5AUmhQMxPu9gcd; APISID=Uc-Ekk5CNnAypcwr/AqxldKvivOi9z5AUz; SAPISID=c_6nd_PxebrccpwL/AtCYtStUWYz3ce0vT; SID=7gSyVXzBz4T3bpHm6x_IA7iX5uubehDqyIZlMx8d8DdKSL8Lh5r89Qg6qco8VehQPbXu6w.; S=quotestreamer=jl1d9K770O1zS6T6R0lRHgaJ3YVq0E9d; NID=108=PNkAkSwC1t5S3XiLroiBW2dPp2nZrhNaaq5mks8ArBX3lFYOWxeyQ5DjNNnpq_TU7aHeZ1JF8MtN3HGLHGnNQzIrVT7PrGInchzSw4e8PhmfBS2dZNDT48yUAVjcqwopn9WZFLUsCtwJXImAYc40w3LDfl0WKfBIhDIEtqzQkz2540YibkkVMGIiBxAWez7bn1aN0ODb4OnqLnRr1CseAc7ukWu2tf_Oaip8tJbPKZgRM2QvXCs67DJH82rzevtFQQEJop4zU0A3lw; DV=s59SNxs2SF5McBdH4fogMB8ly4wP1pVMxolFnT2lDQEAACBqIcFJ--6B2wAAANwIt5sdk6JbVQAAAA dnt:1 upgrade-insecure-requests:1 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 x-chrome-uma-enabled:1 x-client-data:CI+2yQEIo7bJAQjEtskBCP6ZygEI+5zKAQipncoB

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/elazarl/goproxy/issues/232#issuecomment-316769679, or mute the thread https://github.com/notifications/unsubscribe-auth/AAP4opdGVdevPxdMHy5cp-BSKk0u8lQeks5sP4nTgaJpZM4OdVAg .

[SmashGuy2]

I believe so, but that shouldn't matter, should it? Microsoft Edge retrieves the same content (with the response HSTS header) and it goes through the proxy.

Here's the same info from Edge:

Request URL: https://www.google.com/ Request Method: GET Status Code: 200 / OK

- Request Headers Accept: text/html, application/xhtml+xml, image/jxr, / Accept-Encoding: gzip, deflate, br Accept-Language: en-US, en; q=0.5 Cache-Control: no-cache Connection: Keep-Alive Cookie: DV=g5fGsdAKDtNEcBdH4fogMB9Veg8Q1tWNcLvZMSm6FQAAACCTcWJRZ09pEwAAAIhaSHDSvnvgEgAAAA; SID=tgRHzl6Hnk7Uc8CIqfYkMurmCckVLu_C5og0uBdUvaIBuI5FlB1CH2_7Rx2E0YucZtwY9A.; HSID=AIvATs9nPknNnZ7Qq; SSID=AzwSp5SUtw_FY5z01; APISID=B0f6ztLidMPHGJqf/Ax3Q7eWHsvk7xtGqP; SAPISID=ZG48cYIFZRKiJVTU/ALVXLR9LF2iYE8EMv; OGPC=65517568-3:19002264-1:879517696-3:; OGP=-65517568:-879517696:; NID=108=laeWIJ79J35rpwnIC9d8JZ1Oj7ExSKZ8rSJqzBIDVpCLbCD-EiV7SW2MFZRyTmfyh-F6vcDj-oJEdc1Kx0tyW4My0vmd_9a4tR8m506vdABwfMGkx_6vgcrYiZfAZUdnYutzQetmnrcyWM_KNnBeB1PrtaGAYJkhyddQ_D2JxJ5e0x9sgvQ3Jawl8j8NDymBil8uz2HM6ersFoKP9E4IBgBVHA Host: www.google.com User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063

- Response Headers Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,36,35" Cache-Control: private, max-age=0 Content-Type: text/html; charset=UTF-8 Date: Thu, 20 Jul 2017 17:17:35 GMT Expires: -1 Server: gws Set-Cookie: DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.google.com Set-Cookie: DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=.www.google.com Set-Cookie: DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=google.com Set-Cookie: DV=; expires=Mon, 01-Jan-1990 00:00:00 GMT; path=/; domain=www.google.com Strict-Transport-Security: max-age=86400 Transfer-Encoding: chunked X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block

[SmashGuy2]

Found the problem... Chrome is implementing the QUIC protocol and bypassing proxies by utilizing UDP as a transport protocol. Don't have a solution yet though.

https://groups.google.com/a/chromium.org/forum/#!topic/proto-quic/ksokVdwXfQ0

[SmashGuy2]

Ok, so blocking UDP per the above link works in disabling Google's QUIC algorithm. No big loss as Google seems to be the only one using it and it makes no noticeable impact on page load time.

However, Google falls back to HTTP/2 protocol, which is also getting around the proxy. So this needs to be blocked somehow.

[SmashGuy2]

Ok, the problem was with my code. Blocking UDP to ports 80 and 443 disables QUIC and forces Google sites to go through the proxy.