How to limit the crawling speed of a spider

BuilderIO / gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

https://www.builder.io/blog/custom-gpt

ISC License

18.15k stars 1.88k forks source link

Open ZhangzheBJUT opened 7 months ago

ZhangzheBJUT commented 7 months ago

If it crawls too fast, some websites will restrict concurrent access based on IP, and crawling too quickly can result in access being denied.

discohead commented 6 months ago

seconding this issue... I just DDoS'd some poor package maintainer's personal docs site.