The interval for icrawler

hellock / icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

http://icrawler.readthedocs.io/en/latest/

MIT License

844 stars 174 forks source link

The interval for icrawler #55

Open h0nestliar opened 5 years ago

h0nestliar commented 5 years ago

First, I want to say thank you. The icrawler does have me a lot. I have read the source code, but didn't find a way to specify the interval for crawling a website. Is it possible to specify the interval for icrawler?

hellock commented 5 years ago

Currently there is no minimal interval for crawling sites. You may add some time limitation in Feeder, Parser or Downloader.