gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
23.2k stars 1.76k forks source link

How to set the "Host" header in the "OnRequest" Callback? #689

Open ErikOwen opened 2 years ago

ErikOwen commented 2 years ago

I have setup an OnRequest callback like so:

c.OnRequest(func(r *colly.Request) {
    for header, value := range headers {
        r.Headers.Set(header, value)
    }
})

One of these headers is the Host header, but colly will not set that specific header. How can I set the "Host" header via the OnRequest callback?

lianyiwork commented 2 years ago

I tested it for a long time.

found that in the callback function,can not set the host header.

And I try these custom_headers := map[string]string{"Host":"www.google.com"} c := colly.NewCollector( colly.Headers(custom_headers), ) It works.

Hope help you.