AWS Secret for Kubeflow

[strawberrypie]

Kubeflow Pipelines allow certain components to be able to access AWS resources using a secret that is defined in K8s. We need to create this secret for Kubeflow deployments. Below is a example that we previously used.

---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    aws.arn: arn:aws:iam::<ACCOUNT_ID_HERE>:user/<USER_HERE>
  name: aws-secret
  namespace: kubeflow
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <KEY_HERE>
  AWS_SECRET_ACCESS_KEY: <ACCESS_KEY_HERE>

[akastav]

We use this method https://aws.amazon.com/ru/about-aws/whats-new/2019/09/amazon-eks-adds-support-to-assign-iam-permissions-to-kubernetes-service-accounts/ . May be do it like this? https://github.com/provectus/swiss-army-kube/blob/master/modules/system/main.tf#L204 because it is not safe to store data in configs

[strawberrypie]

Sure! It is possible to select a service account which would be to run the pipeline (at least in the latest KFP, 1.0).

[RustamGimadiev]

Service Accounts are namespace-specific resources, so when you create your own Kubeflow namespace you create an additional service account default without any IRSA annotations, which used by default for pipeline runs. The one way to managing AWS access for those roles is modifying oidc_fully_qualified_subjects to match with the newly created namespace, for example, you created a new Kubeflow namespace testing with the default service account and you have an IAM role for use in pipelines.

module iam_assumable_role_admin {
  source                        = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version                       = "~> v2.6.0"
  create_role                   = true
  role_name                     = "${var.cluster_name}_pipeline_runner"
  provider_url                  = replace(data.aws_eks_cluster.this.identity.0.oidc.0.issuer, "https://", "")
  role_policy_arns              = [aws_iam_policy.this.arn]
  oidc_fully_qualified_subjects = ["system:serviceaccount:some-other-namespace:*","system:serviceaccount:testing:*"]
}