search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.53k stars 1.59k forks source link

[backend] Standalone deployment on GKE version 1.19 with workload identity fails v2 compatible mode due to lack of permissions #6089

Closed neuromage closed 4 months ago

neuromage commented 3 years ago

Environment

Steps to reproduce

  1. Create a new GKE cluster (version 1.19.9-gke.1900) a. When creating the cluster, under Security options, enable Workload Identity b. Ensure you have a service account for using workload identity (e.g. workload-identity@<PROJECT-ID>.iam.gserviceaccount.com) c. Ensure this account is bound to KSA [kubeflow/pipeline-runner]. Example:
gcloud iam service-accounts add-iam-policy-binding --role roles/iam.workloadIdentityUser --member "serviceAccount:<PROJECT-ID>.svc.id.goog[kubeflow/pipeline-runner]" workload-identity@<PROJECT-ID>.iam.gserviceaccount.com
  1. Deploy KFP using option 2 as described here
  2. Run a v2-compatible pipeline with pipeline_root set to a GCS directory in which the workload identity service account above has write permissions.

Expected result

Pipeline completes successfully.

What happened instead

Pipeline failed, with launcher complaining about lacking permissions.

Fix

Enable GKE metadata server on the default node-pool using these instructions, e.g.

gcloud container node-pools update default-pool --cluster=<CLUSTER-NAME> --region=<REGION> --workload-metadata=GKE_METADATA

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

neuromage commented 3 years ago

/cc @Bobgy

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

rimolive commented 4 months ago

Closing this issue. No activity for more than a year.

/close

google-oss-prow[bot] commented 4 months ago

@rimolive: Closing this issue.

In response to [this](https://github.com/kubeflow/pipelines/issues/6089#issuecomment-2016834554): >Closing this issue. No activity for more than a year. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.