search

nccgroup / ScoutSuite

Multi-Cloud Security Auditing Tool
GNU General Public License v2.0
6.75k stars 1.07k forks source link

Feature request: automatic rate-limiting #1669

Open rdegraaf-ncc3 opened 2 months ago

rdegraaf-ncc3 commented 2 months ago

Is your feature request related to a problem? Please describe.

Sometimes ScoutSuite sends too many requests to AWS and they start rate-limiting us:

2024-09-19 11:21:59 LT12115 scout[1334] ERROR emr.py L22: Failed to describe EMR cluster: An error occurred 
(ThrottlingException) when calling the DescribeCluster operation (reached max retries: 4): Rate exceeded

It appears that ScoutSuite reacts with a generic failure handler and retries the request immediately, which just makes the problem worse.

Describe the solution you'd like

ScoutSuite should detect rate-limit errors and automatically adjust its request rate accordingly.

Describe alternatives you've considered

ScoutSuite does have command-line options to limit request rates: https://github.com/nccgroup/ScoutSuite/wiki/Handling-Rate-Limiting. However, using them requires the user to cancel the scan and re-run it, making some guess at appropriate request rates. Since ScoutSuite reports success even if there were errors such as this, it's easy for users to miss the errors, think that they had a complete scan, and miss important issues.

x64-latacora commented 2 months ago

Incremental backoff was implemented but it looks like this https://github.com/nccgroup/ScoutSuite/commit/3fe5cde7f86f2b905c21712f54aeef2415b320d6 change turned it off?