Open todd-bush opened 1 year ago
For me, this workaround helped:
import "github.com/andybalholm/brotli"
c.OnResponse(func(r *colly.Response) { ....
if contentEncodingHeader == "br" {
bodyReader := bytes.NewReader(r.Body)
brReader := brotli.NewReader(bodyReader)
decompressed, err := io.ReadAll(brReader)
if err != nil {
log.Println("Error during Brotli decompression:", err)
}
decompressedBody = string(decompressed)
}
}
I've add br support on https://github.com/truthtracer/colly, you can try it
I've run into a few sites that are compressing using Brotli over gzip.
https://chromestatus.com/feature/5420797577396224
Native support for Brotli would be a nice enhancement.
Also, if there's a workaround to handle Brotli sites, please let me know.