Collect breaks collecting ec2-describe-vpcs in ap-northeast-3

[ldomb]

Please mention the following:

• What command was run? python cloudmapper.py collect --config config.json --account account1
• Are you working out of a virtualenv environment, Docker, or something else? VirtualEnv

Runs through most of the regions and then breaks below.

Traceback (most recent call last): File "cloudmapper.py", line 72, in main() File "cloudmapper.py", line 66, in main commands[command].run(arguments) File "/Users/ldo/development/cloudmapper/commands/collect.py", line 599, in run collect(args) File "/Users/ldo/development/cloudmapper/commands/collect.py", line 429, in collect with open(describe_vpcs_file, "r") as f2: FileNotFoundError: [Errno 2] No such file or directory: 'account-data/./account1/ap-northeast-3/ec2-describe-vpcs.json'

[ldomb]

happens also when running on ubuntu not just OSX

[0xdabbad00]

ap-northeast-3 is Osaka which you need to request access to. When I run aws --region ap-northeast-3 ec2 describe-vpcs on the command-line for my accounts, I get:

An error occurred (OptInRequired) when calling the DescribeVpcs operation: You are not subscribed to this service. Please go to http://aws.amazon.com to subscribe.

I assume something similar showed up when you ran collect? Are VPCs supported in Osaka? I believe that region only has one AZ, so maybe there are fewer services available there? CloudMapper does have checks to avoid running calls in regions where that service does not exist, but maybe Osaka left those out?

As I do not have experience with the Osaka region you'll have to do some debugging on your own. If you're ok with ignoring the data in that region for the purposes of auditing and other needs, you can bypass this error by deleting the ap-northeast-3 directory, and modifying your account-data/ACCOUNT/describe-regions.json file to omit that region.

[ldomb]

Not sure why this happens as

aws --region ap-northeast-3 ec2 describe-vpcs provide the below for me.

{ "Vpcs": [ { "VpcId": "vpc-7674fe1f", "InstanceTenancy": "default", "CidrBlockAssociationSet": [ { "AssociationId": "vpc-cidr-assoc-74f3921d", "CidrBlock": "172.31.0.0/16", "CidrBlockState": { "State": "associated" } } ], "State": "available", "DhcpOptionsId": "dopt-a241dccb", "OwnerId": "xxxxxx", "CidrBlock": "172.31.0.0/16", "IsDefault": true } ] }

Happy to do some debugging here but will probably just delete the osaka dir to have it run through successfully for now. I assume that this will cause a challenge as well if I want to run cloudmapper in fargate?

[mjaggard]

Can someone confirm what's needed here? When I get this, has the collect command finished everything else or does this failure prevent it from completing? Is there something I can put into describe-regions.json to avoid this? As far as I know, we're not using ap-northeast-3 at all.

[0xdabbad00]

ap-northeast-3 is now a normal region. Previously, it had been "special", where you needed to request access. The boto SDK used by CloudMapper needs to be updated so it understands this is now a normal region.