gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
23.26k stars 1.76k forks source link

collector.SetCookies() isn't setting Cookie header #671

Open shinebayar-g opened 2 years ago

shinebayar-g commented 2 years ago

Hi. First of all, thank you for the awesome work!

I just noticed collector.SetCookies() method isn't setting any header.

Example:

var cookies []*http.Cookie
cookies = getCookiesFromSomewhereElse()

collector := colly.NewCollector(
        colly.AllowedDomains("example.com")
)
collector.SetCookies("example.com", cookies)
collector.OnRequest(func(r *colly.Request) {
        log.Println("headers:", r.Headers)
})

prints only:

headers: &map[User-Agent:[colly - https://github.com/gocolly/colly/v2]]

No Cookie header was added. For now, I'm setting Cookie header manually.

SleepyPrince commented 2 years ago

Have you tried using r.Request.Headers instead? r.Headers is the response header instead of the request header

shinebayar-g commented 2 years ago
collector.OnRequest(func(r *colly.Request) {
        log.Println("headers:", r.Headers)
})

Function already takes a colly.Request object. Hence r.Request.Headers is not a valid code.

LennyLip commented 2 years ago

These cookies are set from storage in the http client when http client Do function is called. So it works really, but not exist in colly.Request . You can get it from the collector itself.

collector.OnRequest(func(r *colly.Request) {
        cookies := collector.Cookies("example.com")
})