What steps will reproduce the problem?
When we index a number of domains with many threads, sometimes politeness delay
is not sufficient. In such a senario, the sae domain must not be accessed by
two threads. this will givve the web server better breathing space.
What is the expected output? What do you see instead?
some function like crawler.setUniqueDomainFor Threads(true)
What version of the product are you using?
latest
Please provide any additional information below.
none
Original issue reported on code.google.com by christu...@gmail.com on 29 Mar 2012 at 4:18
Original issue reported on code.google.com by
christu...@gmail.com
on 29 Mar 2012 at 4:18