[Enhancement] Support for Google Search API keys

laramies / theHarvester

E-mails, subdomains and names Harvester - OSINT

http://www.edge-security.com/

11.44k stars 2.02k forks source link

[Enhancement] Support for Google Search API keys #529

Open advanced4 opened 4 years ago

advanced4 commented 4 years ago

Feature Request Support using an API key for Google search, which would include Google dependent sources like Linkedin, Twitter, & Trello. Google & Linkedin are particularly great sources of information, but Google is very quick to rate limit requests.

Describe the feature request or bug or other Add a new source, "google_api" which supports using a Google Search API/CSE key

NotoriousRebel commented 4 years ago

Google's rate limiting is a bit much which is why we plan on adding socks5 proxy capabilities eventually. Once that is added do you think pursuing the Google API key would still be worth it?

advanced4 commented 4 years ago

I'm not sure - it would certainly help, but I tried earlier today with regular HTTP proxies and they are all already rate-limited by Google, so thus far proxies aren't helpful.

Google API includes (I think) 100 queries per day for free, which would go a long way. A similar project https://github.com/sundowndev/PhoneInfoga/blob/v1.11/lib/googlesearch.py#L99 implemented it this way. I've only glossed over theHarvester's code base, but hopefully this is a light lift to integrate.