search

nccgroup / ScoutSuite

Multi-Cloud Security Auditing Tool
GNU General Public License v2.0
6.75k stars 1.07k forks source link

GCP scanning should use the Cloud Asset API #940

Open timretout opened 3 years ago

timretout commented 3 years ago

Is your feature request related to a problem? Please describe.

When scanning organisations with a moderately large number of projects, I often receive 429 status responses from the Service Usage API - it is rate-limited to 240 requests per minute by default. I can see that ScoutSuite is making a large volume of requests to this API - presumably num projects * num services requests.

It is not possible to increase the project quota above 240 requests per minute without filling out a form to lift the hard limit.

Describe the solution you'd like

The quota limit increase request form mentions Cloud Asset Inventory as more suitable for scanning large numbers of projects:

Analysis of many projects simultaneously is normally accomplished through Cloud Asset Inventory, which can search or export all project metadata across an organization. Service Usage assets in Cloud Asset Inventory include the enabled services for each project.

The Cloud Asset API allows searches of resources across multiple projects, and has a default quota of 600 searches per minute.

Describe alternatives you've considered

An alternative is to fix ScoutSuite to handle and retry 429 responses with exponential backoff. The form continues:

Traffic that is not user facing, such as offline and batch scripts, should be resilient to quota shortfalls and retry quota errors rather than fail.

However, because the requests would still be rate-limited, this means completing the scan would take longer than desired.

(Additionally, I believe using this API would simplify granting permissions to permit scanning on GCP, and resolve a separate issue where IAM roles are not scanned on projects unless the IAM API is enabled.)

timretout commented 3 years ago

While I've opened #942 to fix the immediate problem, I suspect the Cloud Asset API might still be a better choice for scaling to larger organizations - e.g. with >240 projects hitting Service Usage will probably fail even after the caching is fixed.

x4v13r64 commented 3 years ago

The Cloud Asset API allows searches of resources across multiple projects, and has a default quota of 600 searches per minute.

Gotcha, we'll look into it.

An alternative is to fix ScoutSuite to handle and retry 429 responses with exponential backoff.

This should already be the case, I guess those specific requests aren't being handled adequately (this is hard to test as our dev environment isn't that large).

Additionally, I believe using this API would simplify granting permissions to permit scanning on GCP,

Can you expand on this?

and resolve a separate issue where IAM roles are not scanned on projects unless the IAM API is enabled.

Huh, if this is the case could be worth opening a separate issue?

x4v13r64 commented 3 years ago

I've been looking into this and I'm not convinced by the Cloud Asset Inventory option, since that API needs to be manually enabled before querying, which in many scenarios (e.g. for auditors) may not be an option.