[aws-eks] Restricting access to Amazon EC2 instance profile credentials and use IRSA in VPC CNI plugin

[stefanolczak]

After 'IAM Roles for service accounts' was introduced in EKS there is a official way to restrict access to EC2 instance profile ( https://docs.aws.amazon.com/eks/latest/userguide/restrict-ec2-credential-access.html ) and use IRSA in VPC plugin ( https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-cni-walkthrough.html ) instead of adding AWS managed policy AmazonEKS_CNI_Policy to every role used by EC2 worker nodes.

The problem is that CDK doesn't support it and there is a need to implement some hacks:

1. Service account for aws-node can be updated by just creating new service account from CDK which updates the one created by EKS by adding annotation with IAM role.
2. User data for every EC2 has to be updated by manually modifying launch configuration for ASG generated by CDK.
3. Managed policy AmazonEKS_CNI_Policy has to be removed from EC2 by manually modifying CfnRole.managed_policy_arns because CDK adds it always by default: https://github.com/aws/aws-cdk/blob/e51921d1a81ba9d89e21567291b5f7b215726ca7/packages/%40aws-cdk/aws-eks/lib/cluster.ts#L1165

Please consider adding native support for it in CDK.

---

This is a :rocket: Feature Request

[3oris]

We use this work-around, which also addresses the issue that you need to be able to maintain updates of the plugin by yourselves.

    // START make aws-vpc-cni Helm manageable
    const patches = [];
    for (let kind of ['DaemonSet', 'ClusterRole', 'ClusterRoleBinding']) {
      const patch = new eks.KubernetesPatch(this, 'PatchAwsNode' + kind,{
        cluster: cluster,
        resourceName: kind + '/aws-node',
        resourceNamespace: 'kube-system',
        applyPatch: {
          metadata: {
            annotations: {
              'meta.helm.sh/release-name': 'aws-vpc-cni',
              'meta.helm.sh/release-namespace': 'kube-system',
            },
            labels: {
              'app.kubernetes.io/managed-by': 'Helm'
            }
          }
        },
        restorePatch: {},
        patchType: eks.PatchType.STRATEGIC
      });
      patches.push(patch); // collect patches to set dependencies below
    }
    // END make aws-vpc-cni Helm manageable

    // Create Service Account and IAM Role to let it manage ENIs (AmazonEKS_CNI_Policy)...
    const awsNodeSa = cluster.addServiceAccount('aws-node', {
      name: 'aws-node',
      namespace: 'kube-system'
    });
    awsNodeSa.role.addManagedPolicy(
      iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonEKS_CNI_Policy')
    );
    // the service account must only be destroyed after destruction of all eks worker nodes
    (cluster.defaultNodegroup as eks.Nodegroup).node.addDependency(awsNodeSa);

    // ... then apply Helm Chart for aws-vpc-cni plugin
    const awsNodeChart = cluster.addHelmChart('ChartAwsVpcCni', {
      chart: 'aws-vpc-cni',
      release: 'aws-vpc-cni',
      repository: 'https://aws.github.io/eks-charts',
      version: '1.1.0',
      namespace: 'kube-system',
      values: {
        image: {
          region: 'eu-central-1',
          tag: 'v1.7.5'
        },
        originalMatchLabels: true,
        serviceAccount: {
          create: false,
          name: awsNodeSa.serviceAccountName
        },
        crd: { create: false },
      }
    });
    awsNodeChart.node.addDependency(awsNodeSa);
    for (let patch of patches) {
      awsNodeChart.node.addDependency(patch)
    }
    // thou shalt not remove the plugin on teardown
    ((awsNodeChart.node.findChild('Resource') as cdk.CustomResource)
        .node.defaultChild as cdk.CfnCustomResource)
        .applyRemovalPolicy(cdk.RemovalPolicy.RETAIN, { applyToUpdateReplacePolicy: true });
  }

[3oris]

We since updated our setups to use AWS managed add-ons, as can be seen here: https://github.com/superluminar-io/super-eks/blob/main/src/constructs/eks-managed-addon.ts