search

cartography-cncf / cartography

Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.
https://cartography-cncf.github.io/cartography/
Apache License 2.0
3.01k stars 342 forks source link

Error syncing AWS Kms key data #649

Open csykora-flexion opened 3 years ago

csykora-flexion commented 3 years ago

Title: Error Syncing AWS Kms key data

Description:

When I sync AWS account data, the sync process fails when syncing Kms keys.

To Reproduce:

I can recreate by just re-running the AWS sync, it fails every time on the Kms keys.

Logs:

File "/lyft/cartography/cartography/intel/aws/kms.py", line 363, in sync
    sync_kms_keys(neo4j_session, boto3_session, region, current_aws_account_id, update_tag)
  File "/lyft/cartography/cartography/util.py", line 64, in timed
    return method(*args, **kwargs)
  File "/lyft/cartography/cartography/intel/aws/kms.py", line 353, in sync_kms_keys
    load_kms_key_details(neo4j_session, policy_alias_grants_data, region, current_aws_account_id, aws_update_tag)
  File "/lyft/cartography/cartography/util.py", line 64, in timed
    return method(*args, **kwargs)
  File "/lyft/cartography/cartography/intel/aws/kms.py", line 231, in load_kms_key_details
    _load_kms_key_grants(neo4j_session, grants, update_tag)
  File "/lyft/cartography/cartography/util.py", line 64, in timed
    return method(*args, **kwargs)
  File "/lyft/cartography/cartography/intel/aws/kms.py", line 162, in _load_kms_key_grants
    neo4j_session.run(
  File "/cartography/lib/python3.8/site-packages/neo4j/__init__.py", line 503, in run
    self._connection.fetch()
  File "/cartography/lib/python3.8/site-packages/neobolt/direct.py", line 419, in fetch
    return self._fetch()
  File "/cartography/lib/python3.8/site-packages/neobolt/direct.py", line 461, in _fetch
    response.on_failure(summary_metadata or {})
  File "/cartography/lib/python3.8/site-packages/neobolt/direct.py", line 755, in on_failure
    raise CypherError.hydrate(**metadata)
neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: `Zone offset not in valid range: -18:00 to +18:00`

Please complete the following information::

  • Cartography release version or commit hash: 8867f64e48116a79d803607906a9da0794f89294
  • Python version: 3.83.
  • OS (feel free to omit this if you don't think it's relevant to your issue): OS Mac 11.4

Additional context:

Also get the error with "apigateway" sync

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

achantavy commented 3 years ago

Very interesting, haven't seen this before; what Neo4j database version are you using and which neo4j python driver are you using? We run 3.5.x and neo4j==1.7.{latest} respectively.

@kedarghule are you able to take a look?

csykora-flexion commented 3 years ago

python --version Python 3.8.3

pip list neo4j 1.7.6 neobolt 1.7.17 neotime 1.7.4

Neo4j Server version: 3.5.28 (community)

csykora-flexion commented 3 years ago

Looking at the CreationDate and LocationDate format in the aliases list: 'CreationDate': datetime.datetime(2021, 2, 5, 11, 56, 59, 757000, tzinfo=tzlocal()), 'LastUpdatedDate': datetime.datetime(2021, 2, 5, 11, 56, 59, 757000, tzinfo=tzlocal())}

Not exactly sure why it's adding the "tzinfo=tzlocal()" into that data structure?

Anyway, if I convert that datetime format (seems this code is also doing the conversion for key_list: https://github.com/lyft/cartography/blob/master/cartography/intel/aws/kms.py#L324) This corrects the issue.

billthepayne commented 3 years ago

I've been able to reliably recreate this issue on an Amazon Linux 2 instance by changing the timezone to anything other than UTC (e.g. EST). Once changed, this error pops up frequently using Cartography v. 0.39.0.

Reverting the TZ back to UTC immediately resolves the issue and Cartography runs to completion.

@csykora-flexion - if you're willing to share, what TZ is your host set to? If you can, does changing it to UTC fix the issue?

So there's less ambiguity, these are the steps I took on my host regarding TZ change:

Modified the file '/etc/sysconfig/clock' to contain:

Linked the appropriate zoneinfo file to /etc/localtime:

**A reboot was not required to see results from the Cartography run.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

arthurnowak commented 3 years ago

I got this same error: INFO:cartography.graph.statement:Completed None statement #None INFO:cartography.graph.job:Finished job aws_kms_details ERROR:cartography.sync:Unhandled exception during sync stage 'aws' Traceback (most recent call last): File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 207, in start_aws_ingestion requested_syncs, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 145, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 363, in sync sync_kms_keys(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, *kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 353, in sync_kms_keys load_kms_key_details(neo4j_session, policy_alias_grants_data, region, current_aws_account_id, aws_update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 231, in load_kms_key_details _load_kms_key_grants(neo4j_session, grants, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 165, in _load_kms_key_grants UpdateTag=update_tag, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neo4j/init.py", line 503, in run self._connection.fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 419, in fetch return self._fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 461, in _fetch response.on_failure(summary_metadata or {}) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 755, in on_failure raise CypherError.hydrate(metadata) neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: Zone offset not in valid range: -18:00 to +18:00 Traceback (most recent call last): File ".pyenv/versions/3.7.8/bin/cartography", line 8, in sys.exit(main()) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/cli.py", line 441, in main return CLI(default_sync, prog='cartography').main(argv) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/cli.py", line 421, in main return cartography.sync.run_with_config(self.sync, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/sync.py", line 150, in run_with_config return sync.run(neo4j_driver, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/sync.py", line 73, in run stage_func(neo4j_session, config) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 207, in start_aws_ingestion requested_syncs, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 145, in _sync_multiple_accounts aws_requested_syncs=aws_requested_syncs, # Could be replaced later with per-account requested syncs File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/init.py", line 57, in _sync_one_account RESOURCE_FUNCTIONS[func_name](sync_args) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 363, in sync sync_kms_keys(neo4j_session, boto3_session, region, current_aws_account_id, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, *kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 353, in sync_kms_keys load_kms_key_details(neo4j_session, policy_alias_grants_data, region, current_aws_account_id, aws_update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 231, in load_kms_key_details _load_kms_key_grants(neo4j_session, grants, update_tag) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/util.py", line 73, in timed return method(*args, kwargs) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/cartography/intel/aws/kms.py", line 165, in _load_kms_key_grants UpdateTag=update_tag, File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neo4j/init.py", line 503, in run self._connection.fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 419, in fetch return self._fetch() File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 461, in _fetch response.on_failure(summary_metadata or {}) File ".pyenv/versions/3.7.8/lib/python3.7/site-packages/neobolt/direct.py", line 755, in on_failure raise CypherError.hydrate(metadata) neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: Zone offset not in valid range: -18:00 to +18:00

neo4j 1.7.6 neobolt 1.7.17 neotime 1.7.4 neo4j-community-3.5.28-unix

arthurnowak commented 3 years ago

cartography-0.44.0

francilioaraujo commented 3 years ago

Got the same error on ECR loading. The last log lines are:

INFO:cartography.intel.aws.ecr:Getting ECR repositories for region 'us-east-1'.                                                                                                  
INFO:cartography.intel.aws.ecr:Loading XX ECR repositories for region us-east-1 into graph.                                                                                      
ERROR:cartography.sync:Unhandled exception during sync stage 'aws'                                                                                                               
Traceback (most recent call last):                                                                                                                                               
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/sync.py", line 73, in run                                                   
    stage_func(neo4j_session, config)                                                                                                                                            
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed                                                 
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 202, in start_aws_ingestion
    _sync_multiple_accounts(                
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 139, in _sync_multiple_accounts
    _sync_one_account(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 57, in _sync_one_account
    RESOURCE_FUNCTIONS[func_name](**sync_args)                                          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 149, in sync
    load_ecr_repositories(neo4j_session, repositories, region, current_aws_account_id, update_tag)
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 62, in load_ecr_repositories
    neo4j_session.run(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/neo4j/__init__.py", line 503, in run
    self._connection.fetch()                
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 419, in fetch
    return self._fetch()                    
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 461, in _fetch
    response.on_failure(summary_metadata or {})                                         
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 755, in on_failure
    raise CypherError.hydrate(**metadata)                                               
neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: `Zone offset not in valid range: -18:00 to +18:00
Traceback (most recent call last):                                    
  File "/home/<REDACTED>/bin/cartography", line 8, in <module>                                          
    sys.exit(main())                        
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/cli.py", line 441, in main             
    return CLI(default_sync, prog='cartography').main(argv)                                                                                                                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/cli.py", line 421, in main             
    return cartography.sync.run_with_config(self.sync, config)                                                                                                                   
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/sync.py", line 150, in run_with_config 
    return sync.run(neo4j_driver, config)                                                                                                                                          File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/sync.py", line 73, in run              
    stage_func(neo4j_session, config)                                                                                                                                            
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed            
    return method(*args, **kwargs)                                    
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 202, in start_aws_ingestion           
    _sync_multiple_accounts(                                          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 139, in _sync_multiple_accounts      
    _sync_one_account(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/__init__.py", line 57, in _sync_one_account
    RESOURCE_FUNCTIONS[func_name](**sync_args)                                          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 149, in sync
    load_ecr_repositories(neo4j_session, repositories, region, current_aws_account_id, update_tag)
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/util.py", line 73, in timed
    return method(*args, **kwargs)          
  File "/home/<REDACTED>/lib/python3.9/site-packages/cartography/intel/aws/ecr.py", line 62, in load_ecr_repositories
    neo4j_session.run(                      
  File "/home/<REDACTED>/lib/python3.9/site-packages/neo4j/__init__.py", line 503, in run
    self._connection.fetch()                
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 419, in fetch
    return self._fetch()                    
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 461, in _fetch
    response.on_failure(summary_metadata or {})                                         
  File "/home/<REDACTED>/lib/python3.9/site-packages/neobolt/direct.py", line 755, in on_failure
    raise CypherError.hydrate(**metadata)                                               
neobolt.exceptions.CypherTypeError: Unable to construct OffsetDateTime value: `Zone offset not in valid range: -18:00 to +18:00`

My /etc/localtime is America/Fortaleza. Cartography version is 0.45.0, neo4j running in container with tag 3.5

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

chandanchowdhury commented 2 months ago

Is this still an issue? I have KMS keys with creation date in EDT but not facing this issue.