search

typesense / typesense-docsearch-scraper

A fork of Algolia's awesome DocSearch Scraper, customized to index data in Typesense (an open source alternative to Algolia)
https://typesense.org/docs/guide/docsearch.html
Other
97 stars 36 forks source link

Throttling scrapy? #13

Open lanegoolsby opened 2 years ago

lanegoolsby commented 2 years ago

I noticed that Scrapy has several options for throttling the crawler. I added the following to my config.env file but they seemed to have no effect. Is it possible to use these env vars? If so, what is the syntax?

AUTOTHROTTLE_ENABLED=true CONCURRENT_REQUESTS_PER_DOMAIN=1 AUTOTHROTTLE_DEBUG=true AUTOTHROTTLE_TARGET_CONCURRENCY=1