search

nccgroup / ScoutSuite

Multi-Cloud Security Auditing Tool
GNU General Public License v2.0
6.62k stars 1.05k forks source link

Improve AWS Rate Limiting Support #666

Open x4v13r64 opened 4 years ago

x4v13r64 commented 4 years ago

Is your feature request related to a problem? Please describe.

Investigate the possibility of proactive handling of AWS rate limiting.

https://github.com/jantman/awslimitchecker could be leveraged to ensure we never hit AWS quotas (but backoff before doing so). Apparently the Service Quotas service (mentioned here https://awslimitchecker.readthedocs.io/en/latest/getting_started.html#service-quotas-service) could also be of use. I'm not sure how much awslimitchecker is using that service in addition to Trusted Advisor.

Describe alternatives you've considered

This has been worked on as part of the following:

x4v13r64 commented 4 years ago

Check:

ramimac commented 4 years ago

Summary of previous conversation re:ratelimiting:

awslimitchecker works on a per-service, per-quota level. Didn't seem feasible to implement: as we'd add significant overhead checking limits, and would likely require refactoring to implement for each service.

https://github.com/museadmin/botorate looks like a great possible solution, but wasn't accepted into upstream boto (https://github.com/boto/botocore/pull/1812) and we don't want to be reliant on a boto fork.

If Boto or AWS implement support for this natively, I think we :100: will adopt.

As a stopgap, the current backoff and configurable max-rate appear sufficient.

However, we would like to add some guidance on rate limiting, especially in large environments.

  1. Provide guidance on reasonable values for max-rate
  2. Consider implementing a tool to divine the ideal max-rate for a given environment:
    1. Grab a list of services in use (and potentially scale of their usage)
    2. Grab limits for those services
    3. Suggest a max-rate

The latter may be overkill, as we've only observed issues with EC2 rate-limits.

x4v13r64 commented 4 years ago

Added the https://github.com/nccgroup/ScoutSuite/wiki/Handling-Rate-Limiting wiki page.

I'll leave this issue open but move it off the stack.

ramimac commented 4 years ago

Recently encountered this for the first time with an account that has ~750 EC2 instances. TODO: look at ways of optimizing EC2 code to limit necessary requests...

x4v13r64 commented 4 years ago

TODO: look at ways of optimizing EC2 code to limit necessary requests...

There may be a few cases where this can be optimized, but it also depends heavily on what's going on within the account itself.

ramimac commented 4 years ago

my gut feel is that a major issue is the potential rabbit hole of snapshots. It might be nice to provide some way to disable collecting snapshots in large accounts as a stop-gap solution, and see if that improves most cases

ramimac commented 4 years ago

alternatively or additionally, allow a way to set a max amount of snapshots to check

x4v13r64 commented 4 years ago

my gut feel is that a major issue is the potential rabbit hole of snapshots.

I think that is correct. We could add some filtering to only include snapshots from the last x months? I don't want to go down a rabbit hole of options though...

ramimac commented 4 years ago

Yeah, in general something to avoid. But this specific case is encountered frequently enough, and is a major barrier for large environments. Allowing a flag to set max number (seems easier to implement and more reliable than time) would be valuable