search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.17k stars 1.4k forks source link

[Helm] Allow users to specify service account config for K8sRunLauncher #3445

Open johannkm opened 3 years ago

johannkm commented 3 years ago

Could consider just allowing arbitrary config, although the current autofill is nice

schrockn commented 3 years ago

Consider using Permissive with defined fields. This allows you to use the config schema for known fields and "passthrough" additional fields

natekupp commented 3 years ago

hey @johannkm can you add more context on this issue?

johannkm commented 3 years ago

This could be a blocker for users wishing to launch runs in a separate namespace while using RBAC- currently we automatically set the service account for the run launcher in https://github.com/dagster-io/dagster/blob/master/helm/dagster/templates/configmap-instance.yaml#L85. If you're launching to a different namespace, it may have a different service account.

johannkm commented 3 years ago

This isn't urgent, AFAIK no users have requested it.

grcevski commented 3 years ago

I work for cloud based accounting company, this particular feature is why dagster isn’t considered :(

dougcallaway commented 8 months ago

Is this the correct mechanism to integrate AWS IAM Roles for Service Accounts (IRSA)? I'm trying to avoid managing static AWS credentials (this is highly discouraged by AWS now) as described in the User Guide.

IRSA requires the K8s job pods' service account be annotated with the IAM role ARN. Is the service account referenced in this issue only responsible for launching K8s jobs, or will the same service account be attached to the job pod (requiring AWS access)?

dougcallaway commented 7 months ago

Is this the correct mechanism to integrate AWS IAM Roles for Service Accounts (IRSA)? I'm trying to avoid managing static AWS credentials (this is highly discouraged by AWS now) as described in the User Guide.

IRSA requires the K8s job pods' service account be annotated with the IAM role ARN. Is the service account referenced in this issue only responsible for launching K8s jobs, or will the same service account be attached to the job pod (requiring AWS access)?

I found the correct way to implement IRSA using the Helm chart values. This isn't documented or exposed in values.yaml, but will be passed anyway to the dagster-user-deployments subchart:

dagster-user-deployments:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: <role for pod>