search

gocolly / colly

Elegant Scraper and Crawler Framework for Golang
https://go-colly.org/
Apache License 2.0
23.2k stars 1.76k forks source link

[Question] Colly distributed scrapping ? #122

Closed yspanchal closed 6 years ago

yspanchal commented 6 years ago

Hi, According to colly documentation distributed scrapers

the best you can do is wrapping the scraper in a server. Server can be any kind of service like HTTP, TCP servers.

Can anyone provide some more information and any examples.

asciimoo commented 6 years ago

@yspanchal check out http://go-colly.org/docs/examples/scraper_server/ or https://github.com/gocolly/colly/tree/master/_examples/scraper_server

yspanchal commented 6 years ago

@asciimoo Thank you for this example. But we have to write one more client to send request to server. something like this curl -s 'http://127.0.0.1:7171/?url=http://go-colly.org/'