search

gaspa93 / googlemaps-scraper

Google Maps reviews scraping
GNU General Public License v3.0
358 stars 131 forks source link

Parallelism #42

Closed SudhanshuBlaze closed 1 year ago

SudhanshuBlaze commented 1 year ago

Can we make __scroll and __expand_reviews work parallely in the get_reviews function to improve performance?

gaspa93 commented 1 year ago

Hi @SudhanshuBlaze,

How can you parallelise scrolling function? It is "linear" by definition: you have to scroll down the page to reach the oldest reviews. By consequence, expanding reviews is only possible when they are loaded in the page after scrolling.

In any case, keep in mind that scrolling triggers an API call to Google: if these calls are too frequent (e.g.: not similar to a real "human" scrolling), server does not respond anymore and blocks you for a while. This is done because there are also official API provided by Google that you can pay for to obtain data faster.

Bye, Mattia