search

Crinibus / scraper

Web scraper for scraping, tracking and visualizing prices of products on various websites.
MIT License
176 stars 30 forks source link

Optimize scraping with threads #192

Closed Crinibus closed 1 year ago

Crinibus commented 2 years ago

Update so that scraping at different domains can happen at the same time, but scraping a each domain is sequentially

E.g. if there is four products to be scraped from two different domains, then one product from each domain is scraped at the same time, then after the request_delay the next product from each domain is being scraped at the same time.