fasthttp client. Read body in chunks and stop downloading on condition.

[edwvee]

For example, I want to stop downloading content if it has wrong mime type. How can I do so? The documentation shows only resp.Body() which gets whole response. Below is how I use standard http library:

resp, err := client.Do(req)
if err != nil {
    return nil, err
}
body := resp.Body
defer body.Close()

checkedMime := false
readBuffer := make([]byte, 1 << 12)
bodyData := make([]byte, 0, typicalSize)
for {
    n, err := body.Read(readBuffer)
    if err != nil && err != io.EOF {
        return nil, err
    }
    bodyData = append(bodyData, readBuffer[0:n]...)
    if !checkedMime && len(bodyData) > 512{         
        mimeType := http.DetectContentType(bodyData)
        if isArchive(mimeType){
            isArchive = true
            break
        }
        if mimeType  != "text/html"{
            isUselessContent = true
            break
        }
        checkedMime = true
    }
    if len(bodyData) > tooBigContentSize {
        tooBigContent = true
        break
    }
    if err == io.EOF {
        break
    }
}
// do something with bodyData, isArchive, isUselessContent, tooBigContent
...

[erikdubbelboer]

You could try using https://pkg.go.dev/github.com/valyala/fasthttp?utm_source=godoc#Server.HeaderReceived Its ugly but you could potentially reject the request by setting MaxRequestBodySize to 1 for example.

I would also be open to a pull request that adds something to https://pkg.go.dev/github.com/valyala/fasthttp?utm_source=godoc#RequestConfig to reject the request at that point.

[edwvee]

But I have a client, not a server.

[edwvee]

Added a code using the net/http to achieve what I need as an example.

[moredure]

@erikdubbelboer, such a feature will be useful! I guess I have a patch for this, to skip downloading in case mime type in headers not expected, but not inside the actual body, but need to check.

[erikdubbelboer]

If you have something that works and you can make a pull request that would be great!

[moredure]

but it works like skip body if unwanted header met, and not allows to download body in chunks and check it for mime type