search

dakrone / itsy

A threaded web-spider written in Clojure
181 stars 30 forks source link

Delay policies #4

Open shriphani opened 10 years ago

shriphani commented 10 years ago

Hi, Right now your crawler polls sites quite aggressively. It would be appropriate to wait for a short while before sending another request to a host.

dakrone commented 10 years ago

Hi @shriphani,

Sure, I will look at adding a way to throttle sending requests. In the meantime, you can workaround this by adding throttling (like a Thread/sleep or something equivalent) to the handler for Itsy.