When Http Request use gzip, It will cause !!! REQUEST BODY COULD NOT BE DECODED !!!

[Haoxiqiang]

Very strange!

[Haoxiqiang]

{
    "log": {
        "version": "1.2",
        "creator": {
            "name": "HTTP Toolkit",
            "version": "Unknown"
        },
        "pages": [
            {
                "id": "Ktor/client",
                "title": "Ktor/client",
                "startedDateTime": "2022-10-08T19:31:38.139+08:00",
                "pageTimings": {}
            }
        ],
        "entries": [
            {
                "pageref": "Ktor/client",
                "startedDateTime": "2022-10-08T19:31:38.139+08:00",
                "time": 512.0809419155121,
                "request": {
                    "method": "POST",
                    "url": "xxxxx",
                    "httpVersion": "HTTP/1.1",
                    "cookies": [],
                    "headers": [
                        {
                            "name": "content-encoding",
                            "value": "gzip"
                        },
                        {
                            "name": "user-agent",
                            "value": "ktor/client insight/1.6.3-canary-kt-1.6.21"
                        },
                        {
                            "name": "accept-encoding",
                            "value": "deflate;q=1.0,gzip;q=0.9"
                        },
                        {
                            "name": "accept-charset",
                            "value": "UTF-8"
                        },
                        {
                            "name": "accept",
                            "value": "*/*"
                        },
                        {
                            "name": "content-type",
                            "value": "application/json"
                        },
                        {
                            "name": "content-length",
                            "value": "544"
                        },
                        {
                            "name": "host",
                            "value": "apm-cn.thefrodo.com"
                        },
                        {
                            "name": "connection",
                            "value": "Keep-Alive"
                        }
                    ],
                    "queryString": [],
                    "headersSize": -1,
                    "bodySize": 544,
                    "_requestBodyStatus": "discarded:not-decodable"
                },
                "response": {
                    "status": 200,
                    "statusText": "OK",
                    "httpVersion": "HTTP/1.1",
                    "cookies": [],
                    "headers": [
                        {
                            "name": "date",
                            "value": "Sat, 08 Oct 2022 11:31:38 GMT"
                        },
                        {
                            "name": "content-type",
                            "value": "text/plain; charset=utf-8"
                        },
                        {
                            "name": "content-length",
                            "value": "30"
                        },
                        {
                            "name": "connection",
                            "value": "keep-alive"
                        },
                        {
                            "name": "set-cookie",
                            "value": "INGRESSCOOKIE=1665228699.823.1325.806785; Path=/appcrash; Secure; HttpOnly"
                        },
                        {
                            "name": "strict-transport-security",
                            "value": "max-age=15724800; includeSubDomains"
                        }
                    ],
                    "content": {
                        "mimeType": "text/plain; charset=utf-8",
                        "size": 30,
                        "text": "{\"ret\":1,\"msg\":\"OK\",\"data\":{}}"
                    },
                    "redirectURL": "",
                    "headersSize": -1,
                    "bodySize": 30
                },
                "cache": {},
                "timings": {
                    "blocked": -1,
                    "dns": -1,
                    "connect": -1,
                    "ssl": -1,
                    "send": 1.831737995147705,
                    "wait": 509.80642104148865,
                    "receive": 0.4427828788757324
                }
            }
        ],
        "_tlsErrors": []
    }
}

[pimterry]

Hi @Haoxiqiang, thanks for reporting this. In general, errors like this mean the request body really is corrupted. In this case, the request headers say that the body is gzipped (so HTTP Toolkit has to un-gzip it to show the contents) but then the body does not have the correct gzip header data, so it can't be decoded.

It would be nice to show a better error here instead of the spinner, but in general there's no way for HTTP Toolkit to decode content in most of these cases, so there's not much we can do.

How are you sending this request? Any idea why the body might be corrupted like this?

[Haoxiqiang]

In fact, I checked all the requests I proxied and found if the request is using compression, it will always show loading ui, check document/console. All have similar output, whether it's br or gzip

[Haoxiqiang]

e.g. It's google play log report.

{
    "log": {
        "version": "1.2",
        "creator": {
            "name": "HTTP Toolkit",
            "version": "Unknown"
        },
        "pages": [
            {
                "id": "Com.google.android.gms/223616037 (Android .gms)",
                "title": "Com.google.android.gms/223616037 (Android .gms)",
                "startedDateTime": "2022-10-10T17:00:56.588+08:00",
                "pageTimings": {}
            }
        ],
        "entries": [
            {
                "pageref": "Com.google.android.gms/223616037 (Android .gms)",
                "startedDateTime": "2022-10-10T17:00:56.588+08:00",
                "time": 226.42526698112488,
                "request": {
                    "method": "POST",
                    "url": "https://play.googleapis.com/log/batch",
                    "httpVersion": "HTTP/2.0",
                    "cookies": [],
                    "headers": [
                        {
                            "name": ":method",
                            "value": "POST"
                        },
                        {
                            "name": ":authority",
                            "value": "play.googleapis.com"
                        },
                        {
                            "name": ":scheme",
                            "value": "https"
                        },
                        {
                            "name": ":path",
                            "value": "/log/batch"
                        },
                        {
                            "name": "content-length",
                            "value": "562"
                        },
                        {
                            "name": "content-encoding",
                            "value": "br"
                        },
                        {
                            "name": "content-type",
                            "value": "application/x-brotli"
                        },
                        {
                            "name": "cookie",
                            "value": "NID=ANONYMOUS"
                        },
                        {
                            "name": "user-agent",
                            "value": "com.google.android.gms/223616037 (Linux; U; Android 11; zh_CN_#Hans; Pixel 2 XL; Build/RP1A.201005.004.A1; Cronet/105.0.5195.35)"
                        },
                        {
                            "name": "accept-encoding",
                            "value": "gzip, deflate, br"
                        }
                    ],
                    "queryString": [],
                    "headersSize": -1,
                    "bodySize": 562,
                    "_requestBodyStatus": "discarded:not-decodable"
                },
                "response": {
                    "status": 200,
                    "statusText": "",
                    "httpVersion": "HTTP/2.0",
                    "cookies": [],
                    "headers": [
                        {
                            "name": "content-type",
                            "value": "text/plain; charset=UTF-8"
                        },
                        {
                            "name": "content-encoding",
                            "value": "gzip"
                        },
                        {
                            "name": "date",
                            "value": "Mon, 10 Oct 2022 09:00:56 GMT"
                        },
                        {
                            "name": "server",
                            "value": "Playlog"
                        },
                        {
                            "name": "cache-control",
                            "value": "private"
                        },
                        {
                            "name": "content-length",
                            "value": "102"
                        },
                        {
                            "name": "x-xss-protection",
                            "value": "0"
                        },
                        {
                            "name": "x-frame-options",
                            "value": "SAMEORIGIN"
                        },
                        {
                            "name": "alt-svc",
                            "value": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""
                        }
                    ],
                    "content": {
                        "mimeType": "text/plain; charset=UTF-8",
                        "size": 85,
                        "text": "CMCpBxpNChIKDkFORFJPSURfQkFDS1VQEAAKEQoNQkFUVEVSWV9TVEFUUxAACg8KC1NNQVJUX1NFVFVQEAAKCAoEVFJPThAAEKfogIi5uqjc0QEiAA==",
                        "encoding": "base64"
                    },
                    "redirectURL": "",
                    "headersSize": -1,
                    "bodySize": 102
                },
                "cache": {},
                "timings": {
                    "blocked": -1,
                    "dns": -1,
                    "connect": -1,
                    "ssl": -1,
                    "send": 1.3349049091339111,
                    "wait": 224.59304213523865,
                    "receive": 0.49731993675231934
                }
            }
        ],
        "_tlsErrors": []
    }
}

[pimterry]

That almost certainly means there are broken requests, where the header does not correctly match the content encoding. You can't just add a content-encoding header to compress data - you have to also compress the body to match, or the request is unusable by any valid server or proxy.

You can test a working case like so:

echo '{"test": "value"}' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip" -H"Content-Type: application/json" https://httpbin.org/anything

This compresses a JSON body with gzip, and sends it with the correct content-encoding & content-type headers. If you run this in an intercepted terminal, you will see that HTTP Toolkit can intercept and decode that with no problems. I'm very confident that request body decompression works correctly for valid requests - at the end of the day, this is almost certainly a problem with the client, not HTTP Toolkit.

In the example HAR above, the request is extra suspicious, because of this:

{
    "name": "content-encoding",
    "value": "br"
},
{
    "name": "content-type",
    "value": "application/x-brotli"
}

That's saying the the content is wrapped in Brotli encoding, and also the content inside (after you decompress the Brotli wrapper) is also Brotli content! That's almost certainly wrong, I don't think anybody should ever be double-encoding Brotli content like that.

If you're sending these requests, you should make sure you're compressing the body correctly. If somebody else is sending these requests, they're doing it wrong, and it's going to be difficult to work out what's going on here because it's basically not valid HTTP.

I will look into ways to handle that better, but at the end of the day if a client is sending invalid requests that don't follow the rules of HTTP, it's impossible to know for sure how they should be interpreted.

[Haoxiqiang]

yeah,I debug the http-encoding module,you are right. I find that some requests do not use compression but have headers added. It's my fault.