GCP access token expires for services used less frequently

[soumyadipDe]

Title: GCP access token expires for services used less frequently

Description:

Describe your idea. We are running cartography for our big GCP environment. The issue we faced is that GCP token is expiring for services that are used less frequently. The issue was as below

• Cartography uses GCP python library to fetch credentials
• Credetials are service specific i.e. different for compute/storage/dns/container/servicecatalog
• If a specific credential is not used for 1hr, the token gets revoked. If it is in use, it gets refreshed.
• Now resources like GKE or GCS buckets are few and far between projects.
• So interval between two GKE/GCS API call might be more than 1hr for which credentials gets stale and errors out.

To Reproduce:

As said, same service call between projects with gap of more than an hour will cause this. Logs: If applicable, copy and paste your console log with the failing stack trace.

ERROR:cartography.sync:Unhandled exception during sync stage 'gcp'
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 111, in run
stage_func(neo4j_session, config)
File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
return method(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 252, in start_gcp_ingestion
_sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 210, in _sync_multiple_projects
_sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 181, in _sync_single_project
storage.sync_gcp_buckets(neo4j_session, resources.storage, project_id, gcp_update_tag, common_job_parameters)
File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
return method(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 247, in sync_gcp_buckets
storage_res = get_gcp_buckets(storage, project_id)
File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
return method(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 32, in get_gcp_buckets
res = req.execute()
File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
headers=self.headers,
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 222, in _retry_request
raise exception
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
redirections, connection_type)
File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1725, in request
conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1444, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1396, in _conn_request
response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2570)
Traceback (most recent call last):
File "/usr/local/bin/cartography", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 724, in main
sys.exit(CLI(prog='cartography').main(argv))
File "/usr/local/lib/python3.7/site-packages/cartography/cli.py", line 705, in main
return cartography.sync.run_with_config(self.sync, config)
File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 189, in run_with_config
return sync.run(neo4j_driver, config)
File "/usr/local/lib/python3.7/site-packages/cartography/sync.py", line 111, in run
stage_func(neo4j_session, config)
File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
return method(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 252, in start_gcp_ingestion
_sync_multiple_projects(neo4j_session, resources, projects, config.update_tag, common_job_parameters)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 210, in _sync_multiple_projects
_sync_single_project(neo4j_session, resources, project_id, gcp_update_tag, common_job_parameters)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/__init__.py", line 181, in _sync_single_project
storage.sync_gcp_buckets(neo4j_session, resources.storage, project_id, gcp_update_tag, common_job_parameters)
File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
return method(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 247, in sync_gcp_buckets
storage_res = get_gcp_buckets(storage, project_id)
File "/usr/local/lib/python3.7/site-packages/cartography/util.py", line 197, in timed
return method(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/cartography/intel/gcp/storage.py", line 32, in get_gcp_buckets
res = req.execute()
File "/usr/local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
headers=self.headers,
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 222, in _retry_request
raise exception
File "/usr/local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
redirections, connection_type)
File "/usr/local/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
connection_type=connection_type)
File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1725, in request
conn, authority, uri, request_uri, method, body, headers, redirections, cachekey,
File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1444, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python3.7/site-packages/httplib2/__init__.py", line 1396, in _conn_request
response = conn.getresponse()
File "/usr/local/lib/python3.7/http/client.py", line 1373, in getresponse
response.begin()
File "/usr/local/lib/python3.7/http/client.py", line 319, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.7/http/client.py", line 280, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.7/ssl.py", line 1071, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.7/ssl.py", line 929, in read
return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2570)

Please complete the following information::

• Cartography release version - 0.91
• Python version: 3.10
• OS: debian

[soumyadipDe]

Added suggested PR which worked for my case

[chandanchowdhury]

Hey Saumyadip, thank you for the bug report.

Are you sure the issue is with token expiration? Asking cause I would expect HTTP 401 (Unauthorized) when token has expired instead of SSLError with KRB5_S_TKT_NYV.

[soumyadipDe]

Yes I did get 401 in another stackthread but did not save if unfortunately. Only when I got 401 I understood the issue and could fix it.

[chandanchowdhury]

Note: Might be related to #245