search

serp-spider / search-engine-google

:spider: Google client for SERPS
https://serp-spider.github.io
Other
168 stars 61 forks source link

SERP API Integration? #84

Open hartator opened 7 years ago

hartator commented 7 years ago

Do you know if we can integrate with another backend like ours SerpAPI to get the Google Results?

We have solved the issues related to proxies and captchas, that will be great to integrate seemly with tools like SERPS. I can work on a fork if you want.

gsouf commented 7 years ago

Hi @hartator

Thank for your interest in the library. It's for sure possible to integrate with your backend, there are several ways to do that. As far as I can imagine for now:

We have to take care about how the library will behave because of the urls being different from original google urls. Do your serpApi provide the final google url that was fetched? That might solve some problems.

I think it's better to do this within another package (let's say search-engine-google-serpAPI) that I would be happy to host within the organization.

hartator commented 7 years ago

Hi @gsouf!

Thank for your interest in the library. It's for sure possible to integrate with your backend, there are several ways to do that. As far as I can imagine for now:

  • deeply by providing a custom implementation for google url interface
  • in surface by providing a client that fetches data from your urls and uses the internal parser for the output

Thank you for the awesome work you have been putting in this. I guess whatever is simpler. I suppose we can use our JSON output to hash and be fine, or directly the raw html should be exactly the same as Google's.

We have to take care about how the library will behave because of the urls being different from original google urls. Do your serpApi provide the final google url that was fetched? That might solve some problems.

Not right now as we provide only raw html at the moment, but we can add this in the JSON output. We've tried to do a 1:1 mapping of each request params, so params already look similar. Does it help? For example: https://serpapi.com/search?q=Coffee&hl=fr&gl=fr&num=100 -> https://www.google.com/search?q=Coffee&oq=Coffee&hl=fr&gl=fr&num=100&sourceid=chrome&ie=UTF-8

I think it's better to do this within another package (let's say search-engine-google-serpAPI) that I would be happy to host within the organization.

Sure, whatever is the most convenient for you guys.