search

skyplane-project / skyplane

🔥 Blazing fast bulk data transfers between any cloud 🔥
https://skyplane.org
Apache License 2.0
1k stars 58 forks source link

[bug] GCP ` Invalid JWT Signature` for service account #807

Open sarahwooders opened 1 year ago

sarahwooders commented 1 year ago

Describe the bug I suddenly started getting a token error for GCP, despite no changes to my enviornment or code. I was able to resolve the issue by removing ~/.skyplane and re-running skyplane init, which probably triggered creation of new service credentials for GCP. However skyplane should be able to catch when tokens need to be refreshed and do this automatically.

Transfer client log

Traceback (most recent call last):
  File "/Users/sarahwooders/repos/skyplane-fork/skyplane/examples/test_pipeline.py", line 22, in <module>
    pipeline.start(progress=True)
  File "/Users/sarahwooders/repos/skyplane-fork/skyplane/skyplane/api/pipeline.py", line 86, in start
    topo = planner.plan(self.jobs_to_dispatch)
  File "/Users/sarahwooders/repos/skyplane-fork/skyplane/skyplane/planner/planner.py", line 98, in plan
    src_region_tag = jobs[0].src_iface.region_tag()
  File "/Users/sarahwooders/repos/skyplane-fork/skyplane/skyplane/obj_store/gcs_interface.py", line 75, in region_tag
    return "gcp:" + self.gcp_region
  File "/Users/sarahwooders/repos/skyplane-fork/skyplane/skyplane/obj_store/gcs_interface.py", line 57, in gcp_region
    bucket = self._gcs_client.lookup_bucket(self.bucket_name)
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/storage/client.py", line 818, in lookup_bucket
    return self.get_bucket(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/storage/client.py", line 772, in get_bucket
    bucket.reload(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/storage/bucket.py", line 1086, in reload
    super(Bucket, self).reload(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/storage/_helpers.py", line 246, in reload
    api_response = client._get_resource(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/storage/client.py", line 377, in _get_resource
    return self._connection.api_request(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/storage/_http.py", line 72, in api_request
    return call()
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 482, in api_request
    response = self._make_request(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 341, in _make_request
    return self._do_request(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/cloud/_http/__init__.py", line 379, in _do_request
    return self.http.request(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/auth/transport/requests.py", line 545, in request
    self.credentials.before_request(auth_request, method, url, request_headers)
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/auth/credentials.py", line 135, in before_request
    self.refresh(request)
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/oauth2/service_account.py", line 429, in refresh
    access_token, expiry, _ = _client.jwt_grant(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/oauth2/_client.py", line 303, in jwt_grant
    response_data = _token_endpoint_request(
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/oauth2/_client.py", line 274, in _token_endpoint_request
    _handle_error_response(response_data, retryable_error)
  File "/Users/sarahwooders/Library/Caches/pypoetry/virtualenvs/skyplane-zpxRN4pT-py3.9/lib/python3.9/site-packages/google/oauth2/_client.py", line 73, in _handle_error_response
    raise exceptions.RefreshError(
google.auth.exceptions.RefreshError: ('invalid_grant: Invalid JWT Signature.', {'error': 'invalid_grant', 'error_description': 'Invalid JWT Signature.'})