skyplane-project / skyplane

🔥 Blazing fast bulk data transfers between any cloud 🔥
https://skyplane.org
Apache License 2.0
999 stars 58 forks source link

[SKY-272] [bug] Error with service account key deletion for GCP #922

Open sarahwooders opened 10 months ago

sarahwooders commented 10 months ago

Describe the bug Service account key created by skyplane need to be deleted (since no more than 10 keys are allowed by GCP), however this causes an error during init.

To Reproduce Run skyplane init --reinit-gcp.

Transfer client log In the log output from Skyplane, please upload the debug log from the CLI. You can find the path to the file in the log output:

╰─ skyplane init --reinit-gcp                                                                                                   ─╯
 _____ _   ____   _______ _       ___   _   _  _____
/  ___| | / /\ \ / / ___ \ |     / _ \ | \ | ||  ___|
\ `--.| |/ /  \ V /| |_/ / |    / /_\ \|  \| || |__
 `--. \    \   \ / |  __/| |    |  _  || . ` ||  __|
/\__/ / |\  \  | | | |   | |____| | | || |\  || |___
\____/\_| \_/  \_/ \_|   \_____/\_| |_/\_| \_/\____/

13:23:36 [DEBUG] Found existing configuration file at /Users/sarahwooders/.skyplane/config, loading

Configuring GCP:
    Do you want to configure GCP support in Skyplane? [Y/n]: Y
    GCP credentials will be re-initialized
/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/google/auth/_default.py:76: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. See the following page for troubleshooting: https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds.
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
    GCP credentials found in GCP CLI
    Do you want to enable GCP support in Skyplane? [Y/n]: Y
    Enter the GCP project ID [skyplane-broadcast]:
    Using GCP service account skyplane-manual
13:23:45 [WARN]  Service account skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com has too many keys. Deleting stale keys
to create new key.
    Error saving GCP region config
    Failed to delete key projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179?alt=json returned "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.". Details: "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.">
Traceback (most recent call last):
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 170, in get_service_account_key
    service.projects().serviceAccounts().keys().delete(name=key["name"]).execute()
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179?alt=json returned "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.". Details: "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/cli/cli_init.py", line 365, in load_gcp_config
    auth.save_region_config()
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/utils/imports.py", line 33, in wrapped
    return fn(*modules_imported, *args, **kwargs)
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 36, in save_region_config
    service_account_credentials_file = self.service_account_credentials  # force creation of file
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 93, in service_account_credentials
    self._service_credentials_file = self.get_service_account_key(self._service_account_email)
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 173, in get_service_account_key
    raise ValueError(f"Failed to delete key {key['name']}: {e}")
ValueError: Failed to delete key projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179?alt=json returned "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.". Details: "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.">

    Disabling Google Cloud support

Environment info (please complete the following information):

SKY-272