search

duo-labs / cloudmapper

CloudMapper helps you analyze your Amazon Web Services (AWS) environments.
BSD 3-Clause "New" or "Revised" License
5.96k stars 800 forks source link

Region filter #695

Closed ghost closed 4 years ago

ghost commented 4 years ago

This feature adds an optional --regions argument (comma-separated string) to the collect command. If specified, only the region(s) given (plus the default region) will be collected.

Example Usage:

python cloudmapper.py collect --regions us-west-2

This would result in running collect on both us-west-2 and us-east-1 (in the case of my account).

TESTS: I am not able to run the package tests, even within pipenv shell:

(cloudmapper)  ~/cloudmapper   region-filter  make test
pipenv run -- bash tests/scripts/unit_tests.sh
/Users/tom/.local/share/virtualenvs/cloudmapper-GF7Lg4zO/bin/python: No module named nose
make: *** [test] Error 1

Also, I have a very simple AWS account (currently only running a few instances in us-west-2). I discovered running the report is unsuccessful if I don't include us-east-1, so I added the default region if the argument is specified.

I would need assistance testing this on more complex setups, but this feature avoids a lot of calls for describing AWS resources in regions I don't use.

CLAassistant commented 4 years ago

CLA assistant check
All committers have signed the CLA.

ghost commented 4 years ago

Well, my bad on the tests... needed to run pipenv install --dev

Nonetheless, I get the same 2 failures with and without my changes above: FAIL: test_build_data_structure (test_prepare.TestPrepare) FAIL: test_get_public_nodes (test_public.TestPublic)

0xdabbad00 commented 4 years ago

Sorry about the tests. That is fixed now.

This duplicates #496, which I had not yet merge (... for over a year). This is simpler by using a a single regions flag, so I've decided to go with these changes. In order to merge this, I wanted to make slight edits, so I created a new PR (#698), so you'll be in the git history, but I didn't want to merge this directly without those edits.

ozbillwang commented 4 years ago

something wrong with this change

I set the region --regions ap-southeast-2, i can see the colleciton still go with two regions:

  Making call for account-data/./xxx/ap-southeast-2/ec2-describe-security-groups.json
  Making call for account-data/./xxx/us-east-1/ec2-describe-security-groups.json
ozbillwang commented 4 years ago

@tgrushka @0xdabbad00 cced

0xdabbad00 commented 4 years ago

This is working as intended. The region filter adds the default region (ie. us-east-1) in this line: https://github.com/duo-labs/cloudmapper/pull/695/files#diff-f6fe91e3358619090b886027361d2b1eR225

This is needed because global services (ex. iam) require the data be collected from the default region. CloudMapper would break if us-east-1 was not collected from.

However, this filter could be improved by only collecting the global services from us-east-1, and not, for example collecting security groups as shown in your example. Happy to receive a PR @ozbillwang if you want to improve the filtering for that.