valyala / fasthttp

Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http
MIT License
21.66k stars 1.75k forks source link

Response bodies return garbled data #1547

Closed syllith closed 1 year ago

syllith commented 1 year ago

Hey there, I am executing a ton of network requests to a website using a GET request. For the most part, everything seems to work fine, however, occasionally I'll get a request that looks like this: image The response code is 200 and sometimes it happens a lot back to back, but will occasionally fix itself. Other times it's just a one off and doesn't happen again for a while. It almost looks like incorrect encoding or something? The reason I'm asking this question here is because I'm switching from the standard Go HTTP client and I've never seen this behavior with it, only with fasthttp. Is there perhaps something silly I'm doing that would be causing this? Here's the code I'm using:

req := fasthttp.AcquireRequest()
resp := fasthttp.AcquireResponse()
defer fasthttp.ReleaseRequest(req)
defer fasthttp.ReleaseResponse(resp)
req.SetRequestURI(uri)
err := fasthttp.Do(req, resp)
if err != nil {
      return
}

if resp.StatusCode() == fasthttp.StatusOK {
       fmt.Println(string(resp.Body()))
}

Thank you for any assistance.

syllith commented 1 year ago

I figured it out. Turns out it was GZIP encoded. I guess I was write about the encoding being wrong. This is what I did for any future visitors. Great package by the way, fixed my crashing when using the stock http package.

req := fasthttp.AcquireRequest()
resp := fasthttp.AcquireResponse()
defer fasthttp.ReleaseRequest(req)
defer fasthttp.ReleaseResponse(resp)
req.SetRequestURI(uri)
err := fasthttp.Do(req, resp)
if err != nil {
    time.Sleep(500 * time.Millisecond)
    continue
}

if resp.StatusCode() == fasthttp.StatusOK {
     //* Convert from GZIP
    var reader io.Reader = bytes.NewReader(resp.Body())
    switch string(resp.Header.Peek("Content-Encoding")) {
    case "gzip":
        gzipReader, err := gzip.NewReader(reader)
        defer gzipReader.Close()

        if err != nil {
            time.Sleep(500 * time.Millisecond)
            continue
        }
        reader = gzipReader
}