search

gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
23.2k stars 1.76k forks source link

How to use colly for a second time in a code? #21

Closed ghost closed 7 years ago

ghost commented 7 years ago

Do I have to initialize a new collector? It doesn't work after this.

asciimoo commented 7 years ago

@Frodgy It should work, do you have an example code?

Creating a collector isn't that expensive, so if there is no reason to use the previously created one (e.g. preserving cookies or HTTP keep-alive), it's recommended to initialize a new one for every scraping task. Collector doesn't visit an url more than once if Collector.AllowURLRevisit attribute isn't true, perhaps this causes your problem

ghost commented 7 years ago

My fault - I didn't pay an exact attention to the hosts redirection. Thanks anyway, everything works.