How to do concurrent scraping?

clemfromspace / scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

Do What The F*ck You Want To Public License

919 stars 348 forks source link

How to do concurrent scraping? #95

Open vbuterin2 opened 3 years ago

vbuterin2 commented 3 years ago

I currently use single thread scraper to crawl google.com. But I have tons of search terms. scrapy-selenium only open 1 browser, so I could only search one term at one time. Should I use remote browser or other solutions?

Flushot commented 3 years ago

@vbuterin2 I think your question is a duplicate of https://github.com/clemfromspace/scrapy-selenium/issues/76

cs246810 commented 3 years ago

You could run two scrapy crawl spider_name processes and set CONCURRENT_REQUESTS = 1.