duo-labs / cloudmapper

CloudMapper helps you analyze your Amazon Web Services (AWS) environments.
BSD 3-Clause "New" or "Revised" License
5.99k stars 809 forks source link

Collect breaks collecting ec2-describe-vpcs in ap-northeast-3 #751

Open ldomb opened 4 years ago

ldomb commented 4 years ago

Please mention the following:

Runs through most of the regions and then breaks below.

Traceback (most recent call last): File "cloudmapper.py", line 72, in main() File "cloudmapper.py", line 66, in main commands[command].run(arguments) File "/Users/ldo/development/cloudmapper/commands/collect.py", line 599, in run collect(args) File "/Users/ldo/development/cloudmapper/commands/collect.py", line 429, in collect with open(describe_vpcs_file, "r") as f2: FileNotFoundError: [Errno 2] No such file or directory: 'account-data/./account1/ap-northeast-3/ec2-describe-vpcs.json'

ldomb commented 4 years ago

happens also when running on ubuntu not just OSX

0xdabbad00 commented 4 years ago

ap-northeast-3 is Osaka which you need to request access to. When I run aws --region ap-northeast-3 ec2 describe-vpcs on the command-line for my accounts, I get:

An error occurred (OptInRequired) when calling the DescribeVpcs operation: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.

I assume something similar showed up when you ran collect? Are VPCs supported in Osaka? I believe that region only has one AZ, so maybe there are fewer services available there? CloudMapper does have checks to avoid running calls in regions where that service does not exist, but maybe Osaka left those out?

As I do not have experience with the Osaka region you'll have to do some debugging on your own. If you're ok with ignoring the data in that region for the purposes of auditing and other needs, you can bypass this error by deleting the ap-northeast-3 directory, and modifying your account-data/ACCOUNT/describe-regions.json file to omit that region.

ldomb commented 4 years ago

Not sure why this happens as

aws --region ap-northeast-3 ec2 describe-vpcs provide the below for me.

{ "Vpcs": [ { "VpcId": "vpc-7674fe1f", "InstanceTenancy": "default", "CidrBlockAssociationSet": [ { "AssociationId": "vpc-cidr-assoc-74f3921d", "CidrBlock": "172.31.0.0/16", "CidrBlockState": { "State": "associated" } } ], "State": "available", "DhcpOptionsId": "dopt-a241dccb", "OwnerId": "xxxxxx", "CidrBlock": "172.31.0.0/16", "IsDefault": true } ] }

Happy to do some debugging here but will probably just delete the osaka dir to have it run through successfully for now. I assume that this will cause a challenge as well if I want to run cloudmapper in fargate?

mjaggard commented 3 years ago

Can someone confirm what's needed here? When I get this, has the collect command finished everything else or does this failure prevent it from completing? Is there something I can put into describe-regions.json to avoid this? As far as I know, we're not using ap-northeast-3 at all.

0xdabbad00 commented 3 years ago

ap-northeast-3 is now a normal region. Previously, it had been "special", where you needed to request access. The boto SDK used by CloudMapper needs to be updated so it understands this is now a normal region.