added host-specific queues and now host-wide delay policies can be enforced

dakrone / itsy

A threaded web-spider written in Clojure

181 stars 30 forks source link

added host-specific queues and now host-wide delay policies can be enforced #6

Open shriphani opened 10 years ago

shriphani commented 10 years ago

I have added a default-delay-policy (wait 3 seconds before next request). And now, a ready queue is used to look @ the next set of URLs. Increments are done once the handler does what it needs to with the body

dakrone commented 10 years ago

I'm out of the country right now, but I'll take a look when I get back, thanks!