Error when syncing multiple accounts: Unhandled exception during sync stage 'aws'

[jbar462]

Title: Error when syncing multiple accounts: Unhandled exception during sync stage 'aws'

Description: When I try to sync the multiple accounts that I have setup, using 'AWS_CONFIG_FILE=/root/.aws/config cartography --neo4j-uri bolt://localhost:7687 --aws-sync-all-profiles' cartography works for a little bit then errors and stops running. Seems to crash at the same stage of the sync each time, when "INFO:cartography.intel.aws.apigateway:Syncing AWS APIGateway Rest APIs for region 'us-east-1' in account 'xxxxxxxxxx'." Will paste full log of error below. Also something that caught my eye in the error log was that the final line shows "json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"

To Reproduce: run AWS_CONFIG_FILE=/root/.aws/config cartography --neo4j-uri bolt://localhost:7687 --aws-sync-all-profiles

Logs: INFO:cartography.intel.aws.apigateway:Syncing AWS APIGateway Rest APIs for region 'us-east-1' in account 'XXXXXXXXXX'. ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) Traceback (most recent call last): File "/usr/local/bin/cartography", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

Please complete the following information::

• Cartography release version 0.34.1
• Python version: 3.7.9

Additional context: Unsure if related but i get some warning messages = "WARNING:neo4j:FIXME: should always disconnect before connect", is this common/how do i fix it?

[achantavy]

Also something that caught my eye in the error log was that the final line shows "json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"

This sounds like it should not be happening. @kedarghule have you seen this before?

Unsure if related but i get some warning messages = "WARNING:neo4j:FIXME: should always disconnect before connect", is this common/how do i fix it?

That is unrelated - that log message is emitted from the neo4j python driver hahaha.

[achantavy]

Oh also, are you only seeing this happen for apigateways in one account? That is, does this code work in your other accounts that have apigateways?

[jbar462]

I used "AWS_PROFILE= cartography --neo4j-uri bolt://localhost:7687", error was still occuring at APIGateway section. But this time different error message of "botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation: Too Many Requests". Below is the full log, thanks for the help! INFO:cartography.intel.aws.apigateway:Syncing AWS APIGateway Rest APIs for region 'us-east-1' in account 'XXXXXXXXXX'. ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation: Too Many Requests Traceback (most recent call last): File "/usr/local/bin/cartography", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation: Too Many Requests

[achantavy]

Talked about this in #559 and https://github.com/lyft/cartography/pull/603#issuecomment-816972301: I think this can be helped by setting the AWS_MAX_ATTEMPTS environment variable. As discussed in #603, we need to update the docs on how to set this.

As a side note, glad that I'm not the only one having occasional trouble with this particular sync.

[jbar462]

I tried setting the AWS_MAX_ATTEMPTS to low values (less than 10) on two different accounts, one that I called using the '--aws-sync-all-profiles' and another where i selected a specific profile.

command: AWS_CONFIG_FILE=/root/.aws/config AWS_MAX_ATTEMPTS=5 cartography --neo4j-uri bolt://localhost:7687 --aws-sync-all-profiles error: INFO:cartography.intel.aws.apigateway:Syncing AWS APIGateway Rest APIs for region 'us-east-1' in account 'xxxxxxxxxxxxx'. ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) Traceback (most recent call last): File "/usr/local/bin/cartography", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

command: AWS_PROFILE= AWS_MAX_ATTEMPTS=3 cartography --neo4j-uri bolt://localhost:7687 error: INFO:cartography.intel.aws.apigateway:Syncing AWS APIGateway Rest APIs for region 'us-east-1' in account 'xxxxxxxxxxxxxx'. ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation: Too Many Requests Traceback (most recent call last): File "/usr/local/bin/cartography", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation: Too Many Requests

I'm running into the same errors on both that are listed in above comments... anything else i can do to try and get this working? Thanks for the help thus far.

[mpurusottamc]

@jbar462 Can you try using AWS_RETRY_MODE=standard as an environment variable for handling the request errors? Legacy mode does not handle TooManyRequestsException. Whereas "standard" mode takes care of this.

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#standard-retry-mode

[jbar462]

Unfortunately still getting errors :( Any other steps you think i could take to try and troubleshoot this and understand why i'm getting these errors? Thanks to everyone for the help on this issue so far! attempt 1:

• command: AWS_PROFILE= AWS_RETRY_MODE=standard cartography --neo4j-uri bolt://localhost:7687
• error (at same spot as all other accounts, syncing aws apigateway): ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) Traceback (most recent call last): File "/usr/local/bin/cartography", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

attempt 2 (tried a different account):

• command: AWS_PROFILE= AWS_RETRY_MODE=standard cartography --neo4j-uri bolt://localhost:7687
• error (occured in same spot as attempt 1, syncing aws apigateway): ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation (reached max retries: 2): Too Many Requests Traceback (most recent call last): File "/usr/local/bin/cartography", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation (reached max retries: 2): Too Many Requests

attempt 3:

• command: AWS_CONFIG_FILE=/root/.aws/config AWS_RETRY_MODE=standard cartography --neo4j-uri bolt://localhost:7687 --aws-sync-all-profiles
• error (occurred at same spot as other attempts but threw the jsondecodeerror that is usually thrown when attempting to use --aws-sync-all-profiles) ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1) Traceback (most recent call last): File "/usr/local/bin/cartography", line 8, in sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 369, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 359, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 68, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 332, in parse_policy policy = Policy(json.loads(policy)) File "/usr/local/lib/python3.7/json/init.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/lib/python3.7/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

[ryohare]

I ran into this same issue and took a look at it. The issue is a json.loads is tossing an uncaught exception at https://github.com/lyft/cartography/blob/de1b19439c96915ca68818d67d4e72bb3c2b97fd/cartography/intel/aws/apigateway.py#L332.

To work around, I added a try/except block around there and was able to get the app to run the rest of the day.

The root cause is the policy passed in is doubly escaped JSON. See below for an example.

Policy = {\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"execute-api:Invoke\",\"Resource\":\"arn:aws:execute-api:us-east-1:deadbeef:2stva8ras3\/*\/*\/*\"}]}

I plan to send a PR in the next few days to handle the exception and to escape the Policy string.

[jbar462]

Id love to see that PR, and once its up ill give it a shot for sure. Thanks for taking the time to add onto this thread!

[ryohare]

https://github.com/lyft/cartography/pull/610

[jbar462]

So i was able to test with the new changes made in #611 , and first i tried doing a multiple account sync. The specific command i ran was AWS_CONFIG_FILE=/root/.aws/config AWS_RETRY_MODE=standard cartography --neo4j-uri <neo4j-db> --aws-sync-all-profiles. Unfortunately when attempting to sync the AWS APIGateway's it hit another unhandled exception. Error message: ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 333, in parse_policy policy = policy.replace("\", "") AttributeError: 'NoneType' object has no attribute 'replace' Traceback (most recent call last): File "/usr/local/bin/cartography", line 11, in load_entry_point('cartography==0.35.0', 'console_scripts', 'cartography')() File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 333, in parse_policy policy = policy.replace("\", "") AttributeError: 'NoneType' object has no attribute 'replace'

I am also currently running a test to see if I can get things to behave by just syncing a single account. I will post on here the results of that test run. Still somewhat lost on why I am running into this issue, when it seems like nobody else really is. Could it just be the size of account that i am trying to sync? What do you guys think my issues could be?

[jbar462]

I think i hit the same error at the APIGateway's again. Command = AWS_PROFILE=aws-profile-1 AWS_RETRY_MODE=standard cartography --neo4j-uri bolt://localhost:7687 Error msg: ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 333, in parse_policy policy = policy.replace("\", "") AttributeError: 'NoneType' object has no attribute 'replace' Traceback (most recent call last): File "/usr/local/bin/cartography", line 11, in load_entry_point('cartography==0.35.0', 'console_scripts', 'cartography')() File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 297, in load_rest_api_details parsed_policy = parse_policy(api_id, policy) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 333, in parse_policy policy = policy.replace("\", "") AttributeError: 'NoneType' object has no attribute 'replace'

[ryohare]

Yeah, that error is on the fix I introduced. I'll issue a new PR to fix it up. Unclear why we would be calling this when there is a None policy. Will put a check for none fix highest in the stack and have a unit test for a None policy parsing.

[ryohare]

https://github.com/lyft/cartography/pull/614

[jbar462]

Unfortunately still not working with the updates made in #614. Running into the TooManyRequestsException, thought that setting AWS_RETRY_MODE=standard would prevent that from being an unhandled exception but let me know if you guys notice anything.

ATTEMPT 1: Command: AWS_CONFIG_FILE=/root/.aws/config AWS_RETRY_MODE=standard cartography --neo4j-uri bolt://localhost:7687 --aws-sync-all-profiles

Error msg: ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation (reached max retries: 2): Too Many Requests Traceback (most recent call last): File "/usr/local/bin/cartography", line 11, in load_entry_point('cartography==0.36.0', 'console_scripts', 'cartography')() File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation (reached max retries: 2): Too Many Requests

ATTEMPT 2: Command: AWS_PROFILE= AWS_RETRY_MODE=standard cartography --neo4j-uri bolt://localhost:7687

Error msg: ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation (reached max retries: 2): Too Many Requests Traceback (most recent call last): File "/usr/local/bin/cartography", line 11, in load_entry_point('cartography==0.36.0', 'console_scripts', 'cartography')() File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 201, in start_aws_ingestion requested_syncs, File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 139, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 376, in sync sync_apigateway_rest_apis(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, *kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 366, in sync_apigateway_rest_apis load_rest_api_details(neo4j_session, stages_certificate_resources, current_aws_account_id, aws_update_tag) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 296, in load_rest_api_details for api_id, stage, certificate, resource, policy in stages_certificate_resources: File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 46, in get_rest_api_details resources = get_rest_api_resources(api, client) File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 64, in timed return method(*args, kwargs) File "/usr/local/lib/python3.7/site-packages/cartography/intel/aws/apigateway.py", line 93, in get_rest_api_resources for page in response_iterator: File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 255, in iter response = self._make_request(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/paginate.py", line 332, in _make_request return self._method(current_kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the GetResources operation (reached max retries: 2): Too Many Requests

[achantavy]

@jbar462 Thanks for the details and for bearing with us. In the meantime while we figure this out, it might make sense to temporarily disable the apigateway sync if this is blocking you from syncing other datatypes.

You can use --aws-requested-syncs; see https://github.com/lyft/cartography/blob/44aa4b4566da696d20d4ecbfa15b490bdfe6581e/cartography/cli.py#L169 and https://github.com/lyft/cartography/blob/44aa4b4566da696d20d4ecbfa15b490bdfe6581e/cartography/intel/aws/resources.py#L32.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[ryan-lane]

Not stale

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[joshfaust]

not stale

ERROR:cartography.sync:Unhandled exception during sync stage 'aws'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/cartography/sync.py", line 74, in run
    stage_func(neo4j_session, config)
  File "/usr/local/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 202, in start_aws_ingestion
    _sync_multiple_accounts(
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 139, in _sync_multiple_accounts
    _sync_one_account(
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 47, in _sync_one_account
    regions = _autodiscover_account_regions(boto3_session, current_aws_account_id)
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 84, in _autodiscover_account_regions
    regions = ec2.get_ec2_regions(boto3_session)
  File "/usr/local/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/ec2/__init__.py", line 14, in get_ec2_regions
    result = client.describe_regions()
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeRegions operation: You are not authorized to perform this operation.
Traceback (most recent call last):
  File "/usr/local/bin/cartography", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/cartography/cli.py", line 464, in main
    return CLI(default_sync, prog='cartography').main(argv)
  File "/usr/local/lib/python3.9/site-packages/cartography/cli.py", line 444, in main
    return cartography.sync.run_with_config(self.sync, config)
  File "/usr/local/lib/python3.9/site-packages/cartography/sync.py", line 151, in run_with_config
    return sync.run(neo4j_driver, config)
  File "/usr/local/lib/python3.9/site-packages/cartography/sync.py", line 74, in run
    stage_func(neo4j_session, config)
  File "/usr/local/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 202, in start_aws_ingestion
    _sync_multiple_accounts(
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 139, in _sync_multiple_accounts
    _sync_one_account(
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 47, in _sync_one_account
    regions = _autodiscover_account_regions(boto3_session, current_aws_account_id)
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 84, in _autodiscover_account_regions
    regions = ec2.get_ec2_regions(boto3_session)
  File "/usr/local/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/cartography/intel/aws/ec2/__init__.py", line 14, in get_ec2_regions
    result = client.describe_regions()
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeRegions operation: You are not authorized to perform this operation.

[marco-lancini]

@cribdragg3r : looking at:

botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeRegions operation: You are not authorized to perform this operation.

Does the principal you are using to authenticate against AWS have the ec2:DescribeRegions permission? From docs: point 3 of the Multiple AWS Account Setup section