serpapi / public-roadmap

Public Roadmap for SerpApi, LLC (https://serpapi.com)
57 stars 6 forks source link

[Google Local Services API] Scrape bookings nearby #2001

Closed NateSkiles closed 1 month ago

NateSkiles commented 2 months ago

A user requested we add the "x bookings nearby" text from Google Local Services.

image

Playground | Google | Documentation

Intercom

ocvit commented 2 months ago

Playground and Google links do not return any results. Any other search that can be used as an example?

NateSkiles commented 2 months ago

@ocvit I'm seeing this in the Google link still:

image

Additionally, I'm still seeing results for the query in the playground link:

image

I'm not sure if the link provided in the response for Google services should work, but running the query I get the following returned:

"google_local_services_url": "https://www.google.com/_/AdsHomeservicesConsumerUi/data/batchexecute?hl=en"

But navigating to that link returns:

image

I thought this might be expected due to how we scrape the Google Local Services page, but please let me know if not and I can create a separate issue for that.

ocvit commented 2 months ago

I found that Local Services API is very sensitive to IP from which you access the page.

Results for U.S.-based data_cid:

Screenshot 2024-10-04 at 17 51 52

Results for Canada-based data_cid:

Screenshot 2024-10-04 at 17 56 34

We do not match data_cid to proxy location, so we can get empty results pretty easily:

Screenshot 2024-10-04 at 17 29 15

Talking about X bookings nearby - I couldn't generate a page with this field even once:

Screenshot 2024-10-04 at 16 54 28

There are too many variables here. Basically, with current approach we can't guarantee that data we fetch is valid, i.e. it's not just hidden by incorrect proxy that we use. X bookings nearby is a whole another layer of randomness.

I'm curious is there a way to override country-by-IP detection with some additional search param.

NateSkiles commented 2 months ago

@ocvit Thanks for looking into this! I will add the freezer tag on this for now, as this appears to be much more complicated than I initially assumed.

ocvit commented 1 month ago

Feature has been released.

bookings_nearby field is fetched now + it is correctly rendered in Playground preview.

Screenshot 2024-10-10 at 12 46 38