Different Domains for different threads

What steps will reproduce the problem?
When we index a number of domains with many threads, sometimes politeness delay 
is not sufficient. In such a senario, the sae domain must not be accessed by 
two threads. this will givve the web server better breathing space.

What is the expected output? What do you see instead?
some function like crawler.setUniqueDomainFor Threads(true)

What version of the product are you using?
latest

Please provide any additional information below.
none

Original issue reported on code.google.com by christu...@gmail.com on 29 Mar 2012 at 4:18

jesbin / crawler4j

Different Domains for different threads #140