Sorry for asking here, I tried to get an answer on stackoverflow, but no luck. If I get an answer, I'll update the question on the stackoverflow. Thank you!
I have a project with ~10 spiders, and I run some of them simultaneously using scrapyd. However, I have doubts about whether my CONCURRENT_REQUESTS configuration is correct.
Currently, my CONCURRENT_REQUESTS is 32, but I have seen that it is recommended that this value be much higher (>= 100). But I have a question, is the total number of simultaneous requests that all the running spiders can make or is it the number of simultaneous requests that a single spider can make?
I assume it is the number of simultaneous requests that all the spiders can make and that is why they recommend it to be as high as possible. And I see that I can regulate the number of requests that each spider will make using CONCURRENT_REQUESTS_PER_DOMAIN.
Scrapyd can manage multiple projects, each of which contains multiple spiders. CONCURRENT_REQUESTS operates per-project (i.e. for all spiders in that project).
Sorry for asking here, I tried to get an answer on stackoverflow, but no luck. If I get an answer, I'll update the question on the stackoverflow. Thank you!
I have a project with ~10 spiders, and I run some of them simultaneously using scrapyd. However, I have doubts about whether my
CONCURRENT_REQUESTS
configuration is correct.Currently, my
CONCURRENT_REQUESTS
is32
, but I have seen that it is recommended that this value be much higher (>= 100). But I have a question, is the total number of simultaneous requests that all the running spiders can make or is it the number of simultaneous requests that a single spider can make?I assume it is the number of simultaneous requests that all the spiders can make and that is why they recommend it to be as high as possible. And I see that I can regulate the number of requests that each spider will make using
CONCURRENT_REQUESTS_PER_DOMAIN
.