Misc changes to speedup scraping

calgo-lab / green-db

The monorepo that powers the GreenDB.

https://calgo-lab.github.io/green-db/

22 stars 2 forks source link

Closed se-jaeger closed 2 years ago

se-jaeger commented 2 years ago

I recognized that the scraping took very long. This PR bundles some speedup ideas:

decrease waiting time (for the same domain)
make sure we start different spiders because scrapy can parallelize more if it has to scrape different domains.