Add rate limit for crawl - Githubissues

workeffortwaste / horseman

The detailed update and issue repository for the Horseman crawler.

https://gethorseman.app/

16 stars 0 forks source link

Add rate limit for crawl #118

Closed workeffortwaste closed 1 year ago

workeffortwaste commented 1 year ago

Add rate limit for the crawl queue for stubborn sites.

workeffortwaste commented 1 year ago

Added v0.3.1