Multiple domains crawl without politeness interval

mohankreddy / crawler4j

Automatically exported from code.google.com/p/crawler4j

0 stars 0 forks source link

Multiple domains crawl without politeness interval #51

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago

Is it possible to add large number of unique seed domains, and make crawler to 
choose which page to crawl is such a way, that the requests are evenly 
distributed between the domains, so that it's not neccesary to wait politeness 
interval between requests ?

Original issue reported on code.google.com by ole...@gmail.com on 29 May 2011 at 8:11

GoogleCodeExporter commented 9 years ago

[deleted comment]

GoogleCodeExporter commented 9 years ago

Is this similar to issue 69

Original comment by omkarash...@gmail.com on 10 Jun 2012 at 6:43

GoogleCodeExporter commented 9 years ago

Original comment by avrah...@gmail.com on 18 Aug 2014 at 3:07

Added labels: Type-Enhancement
Removed labels: Type-Defect

GoogleCodeExporter commented 9 years ago

Original comment by avrah...@gmail.com on 18 Aug 2014 at 3:10

Added labels: Priority-Low
Removed labels: Priority-Medium

GoogleCodeExporter commented 9 years ago

Original comment by avrah...@gmail.com on 18 Aug 2014 at 3:11

Changed state: Accepted

GoogleCodeExporter commented 9 years ago

Issue 140

Original comment by avrah...@gmail.com on 18 Aug 2014 at 3:14