[Blueprints, 02-at-scale] Replace Instance Profile by Pod Identities

Configuration

Pod Identities Tested successfully by

1.- Removing Instance Profile configuration

data "aws_iam_policy_document" "managed_ng_assume_role_policy" {
  statement {
    sid = "EKSWorkerAssumeRole"

    actions = [
      "sts:AssumeRole",
    ]
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "managed_ng" {
  name                  = local.cbci_iam_role
  description           = "EKS Managed Node group IAM Role"
  assume_role_policy    = data.aws_iam_policy_document.managed_ng_assume_role_policy.json
  path                  = "/"
  force_detach_policies = true
  # Mandatory for EKS Managed Node Group
  managed_policy_arns = [
    "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  ]
  # Additional Permissions for for EKS Managed Node Group per https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html
  inline_policy {
    name = "${local.name}-iam_inline_policy"
    policy = jsonencode(
      {
        "Version" : "2012-10-17",
        #https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/cloudbees-cache-step#_s3_configuration
        "Statement" : [
          {
            "Sid" : "cbciS3BucketputGetDelete",
            "Effect" : "Allow",
            "Action" : [
              "s3:PutObject",
              "s3:GetObject",
              "s3:DeleteObject"
            ],
            "Resource" : "${local.cbci_s3_location}/*"
          },
          {
            "Sid" : "cbciS3BucketList",
            "Effect" : "Allow",
            "Action" : "s3:ListBucket",
            "Resource" : module.cbci_s3_bucket.s3_bucket_arn
            "Condition" : {
              "StringLike" : {
                "s3:prefix" : "cbci/*"
              }
            }
          },
        ]
      }
    )
  }
  tags = var.tags
}

resource "aws_iam_instance_profile" "managed_ng" {
  name = local.cbci_instance_profile
  role = aws_iam_role.managed_ng.name
  path = "/"

  lifecycle {
    create_before_destroy = true
  }

  tags = var.tags
}

2.- Enable Pod Identity Agent Addon

module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"
  #vEKSBpAddonsTFMod#
  version = "1.15.1"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  oidc_provider_arn = module.eks.oidc_provider_arn
  cluster_version   = module.eks.cluster_version

  eks_addons = {
    ...
    eks-pod-identity-agent = {}
  }
}

3.- Adding Pod Identity

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["pods.eks.amazonaws.com"]
    }

    actions = [
      "sts:AssumeRole",
      "sts:TagSession"
    ]
  }
}

resource "aws_iam_role" "s3" {
  name               = "eks-pod-identity-s3-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
  inline_policy {
    name = "${local.name}-iam_inline_policy"
    policy = jsonencode(
      {
        "Version" : "2012-10-17",
        #https://docs.cloudbees.com/docs/cloudbees-ci/latest/pipelines/cloudbees-cache-step#_s3_configuration
        "Statement" : [
          {
            "Sid" : "cbciS3BucketputGetDelete",
            "Effect" : "Allow",
            "Action" : [
              "s3:PutObject",
              "s3:GetObject",
              "s3:DeleteObject"
            ],
            "Resource" : "${local.cbci_s3_location}/*"
          },
          {
            "Sid" : "cbciS3BucketList",
            "Effect" : "Allow",
            "Action" : "s3:ListBucket",
            "Resource" : module.cbci_s3_bucket.s3_bucket_arn
            "Condition" : {
              "StringLike" : {
                "s3:prefix" : "cbci/*"
              }
            }
          },
        ]
      }
    )
  }
}

resource "aws_eks_pod_identity_association" "cjoc" {
  cluster_name    = module.eks.cluster_name
  namespace       = module.eks_blueprints_addon_cbci.cbci_namespace
  service_account = "cjoc"
  role_arn        = aws_iam_role.s3.arn
}

resource "aws_eks_pod_identity_association" "controllers" {
   cluster_name    = module.eks.cluster_name
   namespace       = module.eks_blueprints_addon_cbci.cbci_namespace
   service_account = "jenkins"
   role_arn        = aws_iam_role.s3.arn
 }

4.- Uploading manually the following version of the plugins (thanks @Vlatombe). The aws-java-sdk related plugins required an updated. It is expected to by in CB CAP by the next release (Until then it is not possible to configure Pod Identity for CBCI)

Proposal design: Include the configuration of the Identity Pods within the Terraform Module, not in the blueprints

Points to validate

1.-Check EKS Cluster > Access > Pod Identity associations

2.- Pod Manifest (OC or Controllers) includes Env and Vol configuration as explained in https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html

kubectl get pod team-b-0 -n cbci -o yaml | grep -i aws
        -DMASTER_ENDPOINT="https://team-b.crl.aws.ps.beescloud.com/" -DMASTER_WEBSOCKET="false"
        -Dcom.cloudbees.networking.hostname="crl.aws.ps.beescloud.com" -Dcom.cloudbees.networking.port=443
    - name: AWS_STS_REGIONAL_ENDPOINTS
    - name: AWS_DEFAULT_REGION
    - name: AWS_REGION
    - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
    - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
      value: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
    - mountPath: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount
          audience: pods.eks.amazonaws.com

Issue : Within the same identity configuration than Controllers, OC does not fetch credentials from AWS

Blueprint impact ==> OC Backup requires additional credentials
Observations:
- Pod Identity Agent logs are not consistent. Sometimes I see activity tracked on them, other times not.
  - When Credentials Env and Volumen are added to the manifest sometimes you can see on logs like {"client-addr":"10.0.48.212:53208","cluster-name":"cbci-bp02-carlos4-eks","fetched_role_arn":"arn:aws:sts::324005994172:assumed-role/eks-pod-identity-s3-role/eks-cbci-bp02--team-a-0-d4a5e71c-ce5c-4bfc-b379-7a31a6324275","fetched_role_id":"AROAUW4CDD26P2OQG6K4E:eks-cbci-bp02--team-a-0-d4a5e71c-ce5c-4bfc-b379-7a31a6324275","level":"info","msg":"Successfully fetched credentials from EKS Auth","request_time_ms":121,"time":"2024-07-04T17:10:04Z"}
  - I don't see any pointers to the issue with the OC. Once I saw The token included in the request has no service account role association for it then I tried to reproduce the same message by leaving all night the OC Identity Pod and that message was not presented in logs.
- There are not options in the helm charts to increase log verbosity https://github.com/aws/eks-pod-identity-agent/blob/main/charts/eks-pod-identity-agent/values.yaml.
- Adding an additional pod (see below example) using the cjoc service account fetches credentials and volumen configuration correctly. This probe narrows the problem within the Operation Center Kubernetes Object configuration

resource "kubectl_manifest" "test_association" {

  yaml_body = <<YAML
apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
  namespace: cbci
spec:
  serviceAccount: cjoc
  containers:
    - name: aws-cli
      image: amazon/aws-cli:latest
      command: ["sleep", "infinity"]
YAML
}

cloudbees / terraform-aws-cloudbees-ci-eks-addon

[Blueprints, 02-at-scale] Replace Instance Profile by Pod Identities #56