search

PiRSquared17 / abot

Automatically exported from code.google.com/p/abot
Apache License 2.0
0 stars 0 forks source link

Add proxy list to the crawler #97

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
I want to add a proxy list to the crawler. Is there any way I could add a list 
of proxy servers and use them effectively for each request to avoid getting 
blocked by the host server?

Original issue reported on code.google.com by alirezan...@gmail.com on 29 Apr 2013 at 9:50

GoogleCodeExporter commented 9 years ago 
This was more of a "How To" question. Transferred question and answer to the 
forum...

https://groups.google.com/forum/?fromgroups=#!topic/abot-web-crawler/aD6Q5L8zYxE

Original comment by sjdir...@gmail.com on 1 May 2013 at 5:56