Update Getting Started Guide to Use EKS Access Entry API

jonathan-innis commented 8 months ago

Description

How can the docs be improved?

EKS just released the Access Entry API which allows you to manage your access control management for users to the cluster through IAM roles through a separate API service rather than through the ConfigMap that currently exists on the cluster today: https://aws.amazon.com/blogs/containers/a-deep-dive-into-simplified-amazon-eks-access-management-controls/.

We should update our Getting Started docs to include how to stand-up a cluster with this new mechanism since this is the latest and greatest way to do IAM role to Kubernetes user access management on EKS

Even better than supplying the API commands for this in the Getting Started Guide would be eksctl just supporting this natively. I've opened a feature request against the repo to get this new access management in as part of cluster stand-up, similar to iamIdentityMappings today: https://github.com/eksctl-io/eksctl/issues/7425

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

johnjeffers commented 7 months ago

I opened a support case with AWS about this. You're not allowed to specify system:* groups when creating an access entry. I was told it's not currently possible to migrate karpenter to the EKS API because it requires system:bootstrappers and system:nodes, so I would have to keep using the aws-auth configmap.

I was referred to this doc and specifically this part of it:

If your cluster's platform version is earlier than the version listed in the Prerequisites section, then you must use this option. If your cluster's platform version is at or later than the platform version listed in the Prerequisites section for your cluster's Kubernetes version, and you've added entries to the ConfigMap, then we recommend that you migrate those entries to access entries. You can't migrate entries that Amazon EKS added to the ConfigMap however, such as entries for IAM roles used with managed node groups or Fargate profiles.

I'd appreciate knowing if there's a workaround for this as I'd love to switch off the configmap and use EKS API access only.

pdeva commented 7 months ago

any plans to support eks access entries. its the recommended way to use eks now and we really want to do away with aws-auth configmap

jonathan-innis commented 7 months ago

I was told it's not currently possible to migrate karpenter to the EKS API

We'll have to validate this bit. I haven't heard this up to this point, but I also don't have deep knowledge of the new Access Entry API yet.

bryantbiggs commented 7 months ago

for those landing here, here is a quick reference:

Terraform

resource "aws_eks_access_entry" "this" {
  cluster_name  = <CLUSTER_NAME>
  principal_arn = <NODE_IAM_ROLE_ARN>
  type          = "EC2_LINUX" # or "EC2_WINDOWS" - Linux covers Bottlerocket as well

  tags = { 
    env = "dev" 
  }
}

In v20.0 of the EKS Terraform module, this is handled automatically for users (other than the entry type if using Windows, which users will need to specify the "EC2_WINDOWS" type)

CloudFormation

Type: AWS::EKS::AccessEntry
Properties:
  ClusterName: <CLUSTER_NAME>
  PrincipalArn: <NODE_IAM_ROLE_ARN>
  Type: "EC2_LINUX" # or "EC2_WINDOWS" - Linux covers Bottlerocket as well
  Tags: 
    - Key: "env"
     Value: "dev"

dmegyesi commented 7 months ago

I'd appreciate knowing if there's a workaround for this as I'd love to switch off the configmap and use EKS API access only.

You need to use the special EC2_LINUX type (in the CLI, Cloudformation, UI, etc.), that will create the correct mapping to the system: groups. I confirm it works with Karpenter node roles if you selected this type.

dnmgns commented 7 months ago

I'd appreciate knowing if there's a workaround for this as I'd love to switch off the configmap and use EKS API access only.

You need to use the special EC2_LINUX type (in the CLI, Cloudformation, UI, etc.), that will create the correct mapping to the system: groups. I confirm it works with Karpenter node roles if you selected this type.

But that type only adds the system:nodes group, I guess the system:bootstrappersgroup mentioned in the Karpenter documentation is not needed then?

(reference: https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/

iamIdentityMappings:
- arn: "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}"
  username: system:node:{{EC2PrivateDNSName}}
  groups:
  - system:bootstrappers
  - system:nodes

aaroniscode commented 5 months ago

FYI for anyone wanting to use Karpenter with EKS Access Entries. system:bootstrappers is not required. I tested a cluster in EKS API only (API) mode.

Also, for those interested, here is why this works:

system:bootstrappers is used for 2 main functions:
1. Obtaining a bootstrap token as part of TLS bootstrap. This token allows the node to request a certificate from the API server that serves as it’s authentication. Amazon EKS does not allow certificate based auth and nodes authenticate using IAM so this is not required.
2. Requesting a kubelet serving certificate. EKS clusters have a ClusterRoleBinding called eks:node-bootstrapper that allows system:nodes permission to request this kubelet serving certificate.

dnmgns commented 5 months ago

FYI for anyone wanting to use Karpenter with EKS Access Entries. system:bootstrappers is not required. I tested a cluster in EKS API only (API) mode.

Also, for those interested, here is why this works:

* `system:bootstrappers` is used for 2 main functions:

  1. **Obtaining a bootstrap token as part of TLS bootstrap.** This token allows the node to request a certificate from the API server that serves as it’s authentication. Amazon EKS does not allow certificate based auth and nodes authenticate using IAM so this is not required.
  2. **Requesting a kubelet serving certificate.** EKS clusters have a ClusterRoleBinding called `eks:node-bootstrapper` that allows `system:nodes` permission to request this kubelet serving certificate.

There's this issue which makes it problematic to fully rely on EKS access entries though: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2912

It doesn't change the fact that system:bootstrappers is not required - but I thought it would be good to mention it for others arriving here.

Nuru commented 5 months ago

Reading between the lines, it looks to me like if you want Karpenter to manage only Linux nodes, you need to register its IAM Role as an EC2_LINUX worker node. However, if you want it to also be able to manage Windows nodes, then you need to register it as an EC2_WINDOWS worker node.

sagarvkd commented 3 months ago

I can pick this issue up for updating the docs.

itssimon commented 3 months ago

Removing the aws-auth entry with the system:bootstrappers group and relying on the EC2_LINUX access entry only breaks Karpenter for me. New nodes are unable to join the cluster until I add the aws-auth entry back in.

vettom commented 3 months ago

Thanks for all comments, I am able to get Karpenter working with API Auth. Here is full example code with terraform https://github.com/vettom/aws-eks-terraform/tree/main/EKS-Cluster-karpenter Works perfectly fine with aws_eks_access_entry

s-sy-y commented 1 month ago

Hey! Will this work only with a certain version of Karpenter? It doesn't seem to work for me, wondering if I need to go through the upgrade process for this. Issue is that new nodes are launched, but never join the cluster after migrating to access entries.

EKS 1.30 Karpenter v0.32.2 Access entry

Principal ARN - Same role that is used for node class instance profile
Type - EC2 Linux
Username - system:node:{{EC2PrivateDNSName}}
Group name - system:nodes
Access policies - none

edit: Turned out that fresh install had faulty clusterEndpoint config in chart values, entries worked after fixing that.

tlb-tarek-sater commented 1 week ago

If I want Karpenter to manage both Linux and Windows nodes, should I set the access entry type to EC2_WINDOWS?

bryantbiggs commented 1 week ago

@tlb-tarek-sater you would need two roles - one mapped to EC2_WINDOWS access entry type, and another mapped to EC2_LINUX. This would also mean you need two Karpenter nodepools for the two respective roles

ArieLevs commented 4 days ago

what would be the best practice for implementing this? since aws-auth map is deprecated does this issue relevant https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2683 ?

as without the ability to add extra iam:PassRole with the new node role to the controller, i had to to it "manually" i mean all is OK and working, just wanted to check if they is a "nicer" way to implement this using the officail module, and not self made policies

bryantbiggs commented 4 days ago

since aws-auth map is deprecated does this issue relevant https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2683 ?

just being able to set iam:PassRole would only be one small change while still requiring a few other changes to allow that role to be usable by nodes created by Karpenter. If you want to add this permission you can use the generic var.iam_policy_statements to add it. But that role will also need an access entry, and potentially an instance profile (depending on whether or not you let Karpenter create that for you)

aws / karpenter-provider-aws