elazarl / goproxy

An HTTP proxy library for Go
BSD 3-Clause "New" or "Revised" License
6.06k stars 1.1k forks source link

Response headers incomplete #414

Closed ciokan closed 4 years ago

ciokan commented 4 years ago

I have this OnResponse hook and many of the queries return with incomplete headers.

ex: https://www.youtube.com/s/desktop/a386e432/jsbin/www-i18n-constants-en_US.vflset/www-i18n-constants.js

returns without a content type even though it is being sent by the server:

Screenshot 2020-11-24 at 14 51 53

Headers that are received in the OnResponse

map[Access-Control-Expose-Headers:[ETag, Cache-Control, Next-Page, Content-Length, Content-Type, Expires, Last-Modified, Pragma, Alert, Backoff, Retry-After] Age:[3038] Cache-Control:[max-age=3600] Connection:[keep-alive] Content-Security-Policy:[default-src 'none'; frame-ancestors 'none'; base-uri 'none';] Date:[Tue, 24 Nov 2020 12:52:34 GMT] Etag:["1606204709703"] Expires:[Tue, 24 Nov 2020 12:03:49 GMT] Strict-Transport-Security:[max-age=31536000] Via:[1.1 1ac3fd533bf6be1b511077f8b8e23bfd.cloudfront.net (CloudFront)] X-Amz-Cf-Id:[NFIjxF0TL3xITvd8Q_lDVMudbWckBOggWsb5cFVK9Xv5-Wkonv1lEw==] X-Amz-Cf-Pop:[FRA2-C1] X-Cache:[Hit from cloudfront] X-Content-Type-Options:[nosniff]]

At first I thought the browser may be "filling" out some missing bits but I also checked with curl:

curl -v https://www.youtube.com/s/desktop/a386e432/jsbin/www-i18n-constants-en_US.vflset/www-i18n-constants.js
*   Trying 172.217.164.142...
* TCP_NODELAY set
* Connected to www.youtube.com (172.217.164.142) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=*.google.com
*  start date: Nov  3 07:33:42 2020 GMT
*  expire date: Jan 26 07:33:42 2021 GMT
*  subjectAltName: host "www.youtube.com" matched cert's "*.youtube.com"
*  issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fad0e80c400)
> GET /s/desktop/a386e432/jsbin/www-i18n-constants-en_US.vflset/www-i18n-constants.js HTTP/2
> Host: www.youtube.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< accept-ranges: bytes
< vary: Accept-Encoding, Origin
< content-type: text/javascript
< cross-origin-resource-policy: cross-origin
< content-length: 2169
< date: Wed, 18 Nov 2020 15:50:36 GMT
< expires: Thu, 18 Nov 2021 15:50:36 GMT
< last-modified: Tue, 17 Nov 2020 20:27:57 GMT
< x-content-type-options: nosniff
< server: sffe
< x-xss-protection: 0
< cache-control: public, max-age=31536000
< age: 507790
< alt-svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
<

I have nothing in this code but what I got from the example files:

proxy := goproxy.NewProxyHttpServer()
proxy.OnResponse().DoFunc(func(resp *http.Response, ctx *goproxy.ProxyCtx) *http.Response {
    log.Info(resp.Header)
})

Any idea if I'm missing something here?