search

GhostManager / Ghostwriter

The SpecterOps project management and reporting engine
https://ghostwriter.wiki
BSD 3-Clause "New" or "Revised" License
1.23k stars 174 forks source link

Idea: Google SafeBrowsing Lookup #85

Open ad0nis opened 3 years ago

ad0nis commented 3 years ago

Would be great to have Google SafeBrowsing as an additional source of domain health checking.

https://transparencyreport.google.com/safe-browsing/search?hl=en

Or API: https://developers.google.com/safe-browsing/v4/get-started

chrismaddalena commented 3 years ago

Yes! Coincidentally, I was just looking at it this week to start sketching out just such an addition. I did not know about the API, so that's exciting.

Going forward, the health check-ups may need to be focused on Google Safe Browsing, VirusTotal, and other aggregators. The individual sites are less and less reliable, and some have started banning IP addresses (permanently or for very extended periods) if your requests suggest you aren't just occasionally checking a handful of domain names. (So it goes with anything that requires web scraping.) We might keep the scraper functions around as something you can decide to enable at your own risk, but they'll be difficult to maintain (and use) in the future.

fsacer commented 1 year ago

Agreed, this would be super useful, seems like an easy add with a single GET request or?

chrismaddalena commented 1 year ago

I looked at the Google Safe Browsing API in 2020 and determined it might be risky. There are two APIs, the "Lookup API" and the "Update API."

https://developers.google.com/safe-browsing/v4

The Lookup API is a simple POST request. That could be implemented with little difficulty, but there is a privacy / OPSEC concern, as noted by Google under the "Disadvantages" section of that API:

Privacy: URLs are not hashed, so the server knows which URLs you look up.

To protect the domain names, you can use the Update API to look up hashed values. The Update API uses hashed values and requires a local database. There is a Go package that makes this easier, but it's non-trivial to get this all integrated into something like Ghostwriter.

There may be no issue with using the Lookup API, but it is interesting Google calls attention to the privacy concern in their documentation. That suggests they may use the data (what you look up) elsewhere.

github-actions[bot] commented 9 hours ago

This issue has been labeled as stale because it has been open for 30 days with no activity.