Open davidejones opened 3 years ago
As you might imagine while our content has grown so too has the time to index. Is there anyway the concurrency settings of scrapy could be exposed? or some feature to run sections of an index in parallel be added to a roadmap?
e.g settings like this
REACTOR_THREADPOOL_MAXSIZE = 128 CONCURRENT_REQUESTS = 256 CONCURRENT_REQUESTS_PER_DOMAIN = 256 CONCURRENT_REQUESTS_PER_IP = 256
Yes, please. Is there any concurrency currently? If not, there really should be...
As you might imagine while our content has grown so too has the time to index. Is there anyway the concurrency settings of scrapy could be exposed? or some feature to run sections of an index in parallel be added to a roadmap?
e.g settings like this