browserup / browserup-proxy

BrowserUp Proxy is a free utility to watch, test, and manipulate web application network traffic and performance.
https://browserup.com
Apache License 2.0
164 stars 41 forks source link

Content-Encoding: gzip header is not present when response filter is added #315

Open amjd opened 3 years ago

amjd commented 3 years ago

Describe the bug When you add a custom response filter using any of the methods (BrowserUpProxy#addResponseFilter, BrowserUpProxy#addFirstHttpFilterFactory, BrowserUpProxy#addLastHttpFilterFactory), Content-Encoding: gzip header does not appear in the HAR, despite the server sending it. In addition, Content-Length header also seems to get updated with the uncompressed size of the response. So it seems that the proxy decompresses the gzip response, and also removes the evidence of compression in the process.

Interestingly, this only seems to occur with gzip-compressed responses and not brotli. I believe that may be because BUP / BMP doesn't support brotli decompression yet.

To Reproduce Steps to reproduce the behavior:

  1. Add Proxy with these settings, in this manner:

    // create and start proxy
    BrowserUpProxy proxy = new BrowserUpProxyServer();
    proxy.setTrustAllServers(true);
    proxy.start();
    
    // add response filter
    proxy.addResponseFilter((response, contents, info) -> System.out.println(info.getUrl() + " " + response.headers()));
    
    // start capturing HAR
    proxy.enableHarCaptureTypes(REQUEST_HEADERS, RESPONSE_HEADERS);
    proxy.newHar();
  2. Open any website that supports gzip compression (e.g., https://www.example.com - real site) through the proxy, with "Accept-Encoding: gzip" header added to the request. I used the proxy with Selenium and Chrome browser.
  3. Save HAR and check the response headers. Content-Encoding header would be missing from the response. Similar request through curl or browser includes the header.
    // save and analyze the HAR
    Har har = proxy.getHar();

Expected behavior When a site supports gzip compression and Accept-Encoding header is sent with gzip value, server responds with Content-Encoding: gzip header and Content-Length header contains the compressed size. This information should be present in HAR as it was actually received.

Please complete the following information: