feat(gcp): allow ability to impersonate a service account

benpops89 commented 1 week ago

What changes are being made and why?

This change allows GCP tasks to be run by impersonating another service account, reasons for this could include

Enhanced Security with Least Privilege Access
Delegated and Scoped Access Control
Temporary Access with Auditing
Simplifying Resource Access in Multi-Project Environments

Thinking about how to structure this, we could change serviceAccount from a String to a HashMap and have key and impersonate keys

serviceAccount:
    key: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
    impersonate: target-sa@email.com

If you do not want to impersonate a service account then do not include the impersonate key.

How the changes have been QAed?

id: gcp_gcs_list
namespace: company.team

tasks:
  - id: list
    type: io.kestra.plugin.gcp.gcs.List
    serviceAccount:
        key: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
        impersonate: target-sa@email.com
    from: "gs://my_bucket/dir/"

I'm not sure of the best way to go about testing this - if guidance can be provided I'm happy to take it on.

Setup Instructions

In order to make this work you will need two service accounts setup and allow the main service account to impersonate the other, documentation on this can be found here. Below is an example gcloud command

gcloud iam service-accounts add-iam-policy-binding TARGET_SERVICE_ACCOUNT_EMAIL \
    --role="roles/iam.serviceAccountTokenCreator" \
    --member="serviceAccount:SOURCE_SERVICE_ACCOUNT_EMAIL" \
    --project=YOUR_PROJECT_ID

A good example may be to give one service account no GCS ObjectViewer role but give it to the other one, then the above task would be a good test, as running without impersonate should not work but providing it should then make the task work

I have never touched Java before so this is completely new to me, I think the formatting I'm using may be different to what is being used in the repo as well 🤦.

There will be lots of other places to change references to service account, including any examples, but wanted to get feedback on the concept before tackling that

closes #452

benpops89 commented 1 week ago

@loicmathieu - thanks for the feedback.

I think having a separate property for the impersonated account works well. I'll rework the PR to reflect this.

I've also worked out how to build the develop docker image with local plugins so will be able to test the changes in more detail

benpops89 commented 1 week ago

@loicmathieu - I have made the requested changes, creating a separate property for impersonated service account.

I have tested by adding the plugin to latest develop docker image and running the following flow, which runs successfully. The service account does not have storage permissions but the impersonated account does

id: impersonate
namespace: test

tasks:
  - id: create_bucket
    type: io.kestra.plugin.gcp.gcs.CreateBucket
    projectId: "poppy-demo"
    serviceAccount: "{{ read('demo-sa.json') }}"
    impersonatedServiceAccount: "storage-account@demo.iam.gserviceaccount.com"
    name: "kestra-test-bucket-creation"

In order to setup the impersonated account you need to do the following

Enable the IAM Service Account Credentials API
On the service account to be impersonated grant access to the base service account giving it the Service Account Token Creator permission. This allows the base service account to create short live tokens for the impersonated account.

If you can provide any details as to why the tests are failing I should be able to look into them

kestra-io / plugin-gcp