Open shriphani opened 10 years ago
Hi, Right now your crawler polls sites quite aggressively. It would be appropriate to wait for a short while before sending another request to a host.
Hi @shriphani,
Sure, I will look at adding a way to throttle sending requests. In the meantime, you can workaround this by adding throttling (like a Thread/sleep or something equivalent) to the handler for Itsy.
Hi, Right now your crawler polls sites quite aggressively. It would be appropriate to wait for a short while before sending another request to a host.