search

gosom / google-maps-scraper

scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
MIT License
852 stars 115 forks source link

Scraper stops after ~3-5 minutes #26

Open yasiralamriki opened 8 months ago

TEST11R commented 8 months ago

bro in windows it doesnt even start, can u help ?

gosom commented 8 months ago

@riku8405 What do you mean it stops after 3-5 minutes ?

This could be normal behaviour .

Please provide more information

yasiralamriki commented 8 months ago

@riku8405 What do you mean it stops after 3-5 minutes ?

This could be normal behaviour .

Please provide more information

Theres no mroe completed jobs it gets stuck

yasiralamriki commented 8 months ago

I tried using IP rotation with 2 machines, so rotating through 2 IPs and this still happens, I dont know if its an issue with the scraper or if I need more please let me know

gosom commented 8 months ago

Do you get any results?

Do you believe that google blocks the requests ?

yasiralamriki commented 8 months ago

I get results, but they stop after about 3-5 minutes, probably but not sure.

gosom commented 7 months ago

can you please provide the input ? In particular I am interested in how many queries your input file contains.

Neurovert commented 7 months ago

@gosom I think it's about the fact that when using your app, "You've reached the end of the list" is shown in the places pane in Google maps - usually it shows no more than just 20 places. I've tried it many times, always same result - but only with query 'accomodation, Czech Republic'. so maybe I have to query each village separately? When I'm browsing maps with the same query on my normal browser, with Google account logged in, it loads the places endlessly when scrolling.

EDIT: I tried going more granular with the query accomodation <city/town/village>, Czech Republic. Of couse now I find much more, but still usually only upto 20 hits.

I'll retry with a logged in Google account.

darealdemayo commented 7 months ago

When i run it with -c 8 or anything higher, it will produce the same error after about 20 minutes for me. when running with -c 1 it can run for hours without a problem -c 20 will cause timeouts after less than one minute

I also noticed that when I start the scraper, it takes less than one second per job and it slowly increases to almost 6 seconds per job. the speed at which it increases seems to depend on the concurrency settings. and all that while top shows ~50-80% cpu and less than 50% ram usage. speedtest shows ping <20ms to google and 100+mbit up/down.

thus, I would guess that what we are seeing here is some sort of rate limitation.

how can we make sure that the browsers run by the scraper are logged into a google account?