search

scholarly-python-package / scholarly

Retrieve author and publication information from Google Scholar in a friendly, Pythonic way without having to worry about CAPTCHAs!
https://scholarly.readthedocs.io/
The Unlicense
1.29k stars 292 forks source link

Enable using SerpAPI to fetch results #425

Open arunkannawadi opened 2 years ago

arunkannawadi commented 2 years ago

With ScraperAPI limiting only 40 searches per month with its free plan, SerpAPI offers 100 searches per month and has good documentation of various Google Scholar APIs. https://serpapi.com/

We should create a SerpAPI class that be used to setup ProxyGenerator in scholarly.

arunkannawadi commented 2 years ago

This is trickier than I thought because SerpAPI offers different endpoints for different types of queries (unlike ScraperAPI which has a single endpoint). Furthermore, they don't offer setting up a proxy, as we do with all current services. This will require getting pages from SerpAPI endpoints instead of getting from GS via the proxy.