Error syncing AWS Kms key data

[csykora-flexion]

Title: Error Syncing AWS Kms key data

Description:

When I sync AWS account data, the sync process fails when syncing Kms keys.

To Reproduce:

I can recreate by just re-running the AWS sync, it fails every time on the Kms keys.

Logs:

File "/lyft/cartography/cartography/intel/aws/kms.py", line 363, in sync
    sync_kms_keys(neo4j_session, boto3_session, region, current_aws_account_id, update_tag)
  File "/lyft/cartography/cartography/util.py", line 64, in timed
    return method(*args, **kwargs)
  File "/lyft/cartography/cartography/intel/aws/kms.py", line 353, in sync_kms_keys
    load_kms_key_details(neo4j_session, policy_alias_grants_data, region, current_aws_account_id, aws_update_tag)
  File "/lyft/cartography/cartography/util.py", line 64, in timed
    return method(*args, **kwargs)
  File "/lyft/cartography/cartography/intel/aws/kms.py", line 231, in load_kms_key_details
    _load_kms_key_grants(neo4j_session, grants, update_tag)
  File "/lyft/cartography/cartography/util.py", line 64, in timed
    return method(*args, **kwargs)
  File "/lyft/cartography/cartography/intel/aws/kms.py", line 162, in _load_kms_key_grants
    neo4j_session.run(
  File "/cartography/lib/python3.8/site-packages/neo4j/__init__.py", line 503, in run
    self._connection.fetch()
  File "/cartography/lib/python3.8/site-packages/neobolt/direct.py", line 419, in fetch
    return self._fetch()
  File "/cartography/lib/python3.8/site-packages/neobolt/direct.py", line 461, in _fetch
    response.on_failure(summary_metadata or {})
  File "/cartography/lib/python3.8/site-packages/neobolt/direct.py", line 755, in on_failure
    raise CypherError.hydrate(**metadata)
neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: `Zone offset not in valid range: -18:00 to +18:00`

Please complete the following information::

• Cartography release version or commit hash: 8867f64e48116a79d803607906a9da0794f89294
• Python version: 3.83.
• OS (feel free to omit this if you don't think it's relevant to your issue): OS Mac 11.4

Additional context:

Also get the error with "apigateway" sync

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[achantavy]

Very interesting, haven't seen this before; what Neo4j database version are you using and which neo4j python driver are you using? We run 3.5.x and neo4j==1.7.{latest} respectively.

@kedarghule are you able to take a look?

[csykora-flexion]

python --version Python 3.8.3

pip list neo4j 1.7.6 neobolt 1.7.17 neotime 1.7.4

Neo4j Server version: 3.5.28 (community)

[csykora-flexion]

Looking at the CreationDate and LocationDate format in the aliases list: 'CreationDate': datetime.datetime(2021, 2, 5, 11, 56, 59, 757000, tzinfo=tzlocal()), 'LastUpdatedDate': datetime.datetime(2021, 2, 5, 11, 56, 59, 757000, tzinfo=tzlocal())}

Not exactly sure why it's adding the "tzinfo=tzlocal()" into that data structure?

Anyway, if I convert that datetime format (seems this code is also doing the conversion for key_list: https://github.com/lyft/cartography/blob/master/cartography/intel/aws/kms.py#L324) This corrects the issue.

[billthepayne]

I've been able to reliably recreate this issue on an Amazon Linux 2 instance by changing the timezone to anything other than UTC (e.g. EST). Once changed, this error pops up frequently using Cartography v. 0.39.0.

Reverting the TZ back to UTC immediately resolves the issue and Cartography runs to completion.

@csykora-flexion - if you're willing to share, what TZ is your host set to? If you can, does changing it to UTC fix the issue?

So there's less ambiguity, these are the steps I took on my host regarding TZ change:

Modified the file '/etc/sysconfig/clock' to contain:

• ZONE="UTC"
• UTC=true

Linked the appropriate zoneinfo file to /etc/localtime:

• sudo ln -sf /usr/share/zoneinfo/UTC /etc/localtime

**A reboot was not required to see results from the Cartography run.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[arthurnowak]

I got this same error: INFO:cartography.graph.statement:Completed None statement #None INFO:cartography.graph.job:Finished job aws_kms_details ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 207, in start_aws_ingestion requested_syncs, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 145, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 363, in sync sync_kms_keys(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, *kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 353, in sync_kms_keys load_kms_key_details(neo4j_session, policy_alias_grants_data, region, current_aws_account_id, aws_update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 231, in load_kms_key_details _load_kms_key_grants(neo4j_session, grants, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 165, in _load_kms_key_grants UpdateTag=update_tag, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neo4j/init.py", line 503, in run self._connection.fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 419, in fetch return self._fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 461, in _fetch response.on_failure(summary_metadata or {}) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 755, in on_failure raise CypherError.hydrate(metadata) neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: Zone offset not in valid range: -18:00 to +18:00 Traceback (most recent call last): File ".pyenv/versions/3.7.8/bin/cartography", line 8, in sys.exit(main()) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 207, in start_aws_ingestion requested_syncs, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 145, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 363, in sync sync_kms_keys(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, *kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 353, in sync_kms_keys load_kms_key_details(neo4j_session, policy_alias_grants_data, region, current_aws_account_id, aws_update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 231, in load_kms_key_details _load_kms_key_grants(neo4j_session, grants, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 165, in _load_kms_key_grants UpdateTag=update_tag, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neo4j/init.py", line 503, in run self._connection.fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 419, in fetch return self._fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 461, in _fetch response.on_failure(summary_metadata or {}) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 755, in on_failure raise CypherError.hydrate(metadata) neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: Zone offset not in valid range: -18:00 to +18:00

neo4j 1.7.6 neobolt 1.7.17 neotime 1.7.4 neo4j-community-3.5.28-unix

[arthurnowak]

cartography-0.44.0

[francilioaraujo]

Got the same error on ECR loading. The last log lines are:

INFO:cartography.intel.aws.ecr:Getting ECR repositories for region 'us-east-1'.                                                                                                  
INFO:cartography.intel.aws.ecr:Loading XX ECR repositories for region us-east-1 into graph.                                                                                      
ERROR:cartography.sync:Unhandled exception during sync stage 'aws'                                                                                                               
Traceback (most recent call last):                                                                                                                                               
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/sync.py", line 73, in run                                                   
    stage_func(neo4j_session, config)                                                                                                                                            
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed                                                 
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 202, in start_aws_ingestion
    _sync_multiple_accounts(                
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 139, in _sync_multiple_accounts
    _sync_one_account(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 57, in _sync_one_account
    RESOURCE_FUNCTIONS[func_name](**sync_args)                                          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 149, in sync
    load_ecr_repositories(neo4j_session, repositories, region, current_aws_account_id, update_tag)
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 62, in load_ecr_repositories
    neo4j_session.run(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/neo4j/__init__.py", line 503, in run
    self._connection.fetch()                
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 419, in fetch
    return self._fetch()                    
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 461, in _fetch
    response.on_failure(summary_metadata or {})                                         
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 755, in on_failure
    raise CypherError.hydrate(**metadata)                                               
neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: `Zone offset not in valid range: -18:00 to +18:00
Traceback (most recent call last):                                    
  File "/home/<REDACTED>/bin/cartography", line 8, in <module>                                          
    sys.exit(main())                        
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/cli.py", line 441, in main             
    return CLI(default_sync, prog='cartography').main(argv)                                                                                                                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/cli.py", line 421, in main             
    return cartography.sync.run_with_config(self.sync, config)                                                                                                                   
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/sync.py", line 150, in run_with_config 
    return sync.run(neo4j_driver, config)                                                                                                                                          File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/sync.py", line 73, in run              
    stage_func(neo4j_session, config)                                                                                                                                            
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed            
    return method(*args, **kwargs)                                    
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 202, in start_aws_ingestion           
    _sync_multiple_accounts(                                          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 139, in _sync_multiple_accounts      
    _sync_one_account(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 57, in _sync_one_account
    RESOURCE_FUNCTIONS[func_name](**sync_args)                                          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 149, in sync
    load_ecr_repositories(neo4j_session, repositories, region, current_aws_account_id, update_tag)
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 62, in load_ecr_repositories
    neo4j_session.run(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/neo4j/__init__.py", line 503, in run
    self._connection.fetch()                
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 419, in fetch
    return self._fetch()                    
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 461, in _fetch
    response.on_failure(summary_metadata or {})                                         
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 755, in on_failure
    raise CypherError.hydrate(**metadata)                                               
neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: `Zone offset not in valid range: -18:00 to +18:00`

My /etc/localtime is America/Fortaleza. Cartography version is 0.45.0, neo4j running in container with tag 3.5

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[chandanchowdhury]

Is this still an issue? I have KMS keys with creation date in EDT but not facing this issue.