aws-quickstart / cdk-eks-blueprints

AWS Quick Start Team
Apache License 2.0
446 stars 198 forks source link

Karpenter Addon: Can't attach custom instanceProfile to Karpenter nodes #1061

Open idchrisamzn opened 1 month ago

idchrisamzn commented 1 month ago

Describe the bug

When attaching a custom instanceProfile to Ec2NodeClassSpec, Karpenter attempts to create the EC2 instances but gets an IAM access denied as it doesn't have iam:PassRole permissions on the custom IAM role in the custom instance profile. The instance profile gets added to default-ec2nodeclass just fine.

It looks like the ec2 IAM role which gets passed to Karpenter's own IAM role is fixed as the one which gets created as part of the addon: https://github.com/aws-quickstart/cdk-eks-blueprints/blob/60256b02db630fe5b422b5e9bf47ded951efe976/lib/addons/karpenter/index.ts#L397 - therefore it could never support a custom instanceProfile

Expected Behavior

Karpenter doesn't create its own EC2 Instance Profile when instanceProfile is specified in the Ec2NodeClassSpec. Instead the custom profile is used to setup Karpenter's own IAM role

Current Behavior

Karpenter attempts to use the custom instance profile but fails to launch due ec2 instances due to the lack of iam:PassRole permissions on the custom IAM role.

Reproduction Steps

Define a custom role as such:

const karpenterRole: iam.RoleProps = {
  assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
  description: 'Role for nodes',
  managedPolicies: [
      { managedPolicyArn: `arn:${Aws.PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly` },
      { managedPolicyArn: `arn:${Aws.PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy` },
      { managedPolicyArn: `arn:${Aws.PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore` }
  ] 
};

Use the IAM role properties to create the IAM role and instance profile in the Ec2NodeClassSpec

function returnDefaultEc2NodeClassSpec(clusterName: string): Ec2NodeClassSpec {
  return {
    amiFamily: "Bottlerocket",
    subnetSelectorTerms: [{ tags: { "Name": `${clusterName}-vpc/${clusterName}-vpc/PrivateSubnet*` }}],
    securityGroupSelectorTerms: [{ tags: { "aws:eks:cluster-name": `${clusterName}-ct` }}],
    instanceStorePolicy: 'RAID0',
    tags: { 'name': `${clusterName}-apps`, },
    instanceProfile: blueprints.getResource(context => {
      return new iam.InstanceProfile(context.scope, 'KarpenterInstanceProfile', {
        role: new iam.Role(context.scope, 'KarpenterInstanceRole', karpenterRole)
      });
    }).instanceProfileName,
  };
};

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.147.3 (build 32f0fdb)

EKS Blueprints Version

1.15.1

Node.js Version

v20.15.1

Environment details (OS name and version, etc.)

Ubuntu 22.04.4 LTS

Other information

No response

idchrisamzn commented 1 month ago

Related to #893