Open jonathan-innis opened 8 months ago
I opened a support case with AWS about this. You're not allowed to specify system:*
groups when creating an access entry. I was told it's not currently possible to migrate karpenter to the EKS API because it requires system:bootstrappers
and system:nodes
, so I would have to keep using the aws-auth
configmap.
I was referred to this doc and specifically this part of it:
If your cluster's platform version is earlier than the version listed in the Prerequisites section, then you must use this option. If your cluster's platform version is at or later than the platform version listed in the Prerequisites section for your cluster's Kubernetes version, and you've added entries to the ConfigMap, then we recommend that you migrate those entries to access entries. You can't migrate entries that Amazon EKS added to the ConfigMap however, such as entries for IAM roles used with managed node groups or Fargate profiles.
I'd appreciate knowing if there's a workaround for this as I'd love to switch off the configmap and use EKS API access only.
any plans to support eks access entries. its the recommended way to use eks now and we really want to do away with aws-auth configmap
I was told it's not currently possible to migrate karpenter to the EKS API
We'll have to validate this bit. I haven't heard this up to this point, but I also don't have deep knowledge of the new Access Entry API yet.
for those landing here, here is a quick reference:
resource "aws_eks_access_entry" "this" {
cluster_name = <CLUSTER_NAME>
principal_arn = <NODE_IAM_ROLE_ARN>
type = "EC2_LINUX" # or "EC2_WINDOWS" - Linux covers Bottlerocket as well
tags = {
env = "dev"
}
}
In v20.0 of the EKS Terraform module, this is handled automatically for users (other than the entry type if using Windows, which users will need to specify the "EC2_WINDOWS"
type)
Type: AWS::EKS::AccessEntry
Properties:
ClusterName: <CLUSTER_NAME>
PrincipalArn: <NODE_IAM_ROLE_ARN>
Type: "EC2_LINUX" # or "EC2_WINDOWS" - Linux covers Bottlerocket as well
Tags:
- Key: "env"
Value: "dev"
I'd appreciate knowing if there's a workaround for this as I'd love to switch off the configmap and use EKS API access only.
You need to use the special EC2_LINUX
type (in the CLI, Cloudformation, UI, etc.), that will create the correct mapping to the system:
groups. I confirm it works with Karpenter node roles if you selected this type.
I'd appreciate knowing if there's a workaround for this as I'd love to switch off the configmap and use EKS API access only.
You need to use the special
EC2_LINUX
type (in the CLI, Cloudformation, UI, etc.), that will create the correct mapping to thesystem:
groups. I confirm it works with Karpenter node roles if you selected this type.
But that type only adds the system:nodes
group, I guess the system:bootstrappers
group mentioned in the Karpenter documentation is not needed then?
(reference: https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/
iamIdentityMappings:
- arn: "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}"
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
FYI for anyone wanting to use Karpenter with EKS Access Entries. system:bootstrappers
is not required. I tested a cluster in EKS API only (API) mode.
Also, for those interested, here is why this works:
system:bootstrappers
is used for 2 main functions:
eks:node-bootstrapper
that allows system:nodes
permission to request this kubelet serving certificate.FYI for anyone wanting to use Karpenter with EKS Access Entries.
system:bootstrappers
is not required. I tested a cluster in EKS API only (API) mode.Also, for those interested, here is why this works:
* `system:bootstrappers` is used for 2 main functions: 1. **Obtaining a bootstrap token as part of TLS bootstrap.** This token allows the node to request a certificate from the API server that serves as it’s authentication. Amazon EKS does not allow certificate based auth and nodes authenticate using IAM so this is not required. 2. **Requesting a kubelet serving certificate.** EKS clusters have a ClusterRoleBinding called `eks:node-bootstrapper` that allows `system:nodes` permission to request this kubelet serving certificate.
There's this issue which makes it problematic to fully rely on EKS access entries though: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2912
It doesn't change the fact that system:bootstrappers
is not required - but I thought it would be good to mention it for others arriving here.
Reading between the lines, it looks to me like if you want Karpenter to manage only Linux nodes, you need to register its IAM Role as an EC2_LINUX
worker node. However, if you want it to also be able to manage Windows nodes, then you need to register it as an EC2_WINDOWS
worker node.
I can pick this issue up for updating the docs.
Removing the aws-auth
entry with the system:bootstrappers
group and relying on the EC2_LINUX
access entry only breaks Karpenter for me. New nodes are unable to join the cluster until I add the aws-auth
entry back in.
Thanks for all comments, I am able to get Karpenter working with API Auth.
Here is full example code with terraform https://github.com/vettom/aws-eks-terraform/tree/main/EKS-Cluster-karpenter
Works perfectly fine with aws_eks_access_entry
Hey! Will this work only with a certain version of Karpenter? It doesn't seem to work for me, wondering if I need to go through the upgrade process for this. Issue is that new nodes are launched, but never join the cluster after migrating to access entries.
EKS 1.30 Karpenter v0.32.2 Access entry
Access policies - none
edit: Turned out that fresh install had faulty clusterEndpoint config in chart values, entries worked after fixing that.
If I want Karpenter to manage both Linux and Windows nodes, should I set the access entry type to EC2_WINDOWS
?
@tlb-tarek-sater you would need two roles - one mapped to EC2_WINDOWS
access entry type, and another mapped to EC2_LINUX
. This would also mean you need two Karpenter nodepools for the two respective roles
what would be the best practice for implementing this? since aws-auth map is deprecated does this issue relevant https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2683 ?
as without the ability to add extra iam:PassRole
with the new node role to the controller, i had to to it "manually"
i mean all is OK and working, just wanted to check if they is a "nicer" way to implement this using the officail module, and not self made policies
since aws-auth map is deprecated does this issue relevant https://github.com/terraform-aws-modules/terraform-aws-eks/issues/2683 ?
just being able to set iam:PassRole
would only be one small change while still requiring a few other changes to allow that role to be usable by nodes created by Karpenter. If you want to add this permission you can use the generic var.iam_policy_statements
to add it. But that role will also need an access entry, and potentially an instance profile (depending on whether or not you let Karpenter create that for you)
Description
How can the docs be improved?
EKS just released the Access Entry API which allows you to manage your access control management for users to the cluster through IAM roles through a separate API service rather than through the ConfigMap that currently exists on the cluster today: https://aws.amazon.com/blogs/containers/a-deep-dive-into-simplified-amazon-eks-access-management-controls/.
We should update our Getting Started docs to include how to stand-up a cluster with this new mechanism since this is the latest and greatest way to do IAM role to Kubernetes user access management on EKS
Even better than supplying the API commands for this in the Getting Started Guide would be
eksctl
just supporting this natively. I've opened a feature request against the repo to get this new access management in as part of cluster stand-up, similar toiamIdentityMappings
today: https://github.com/eksctl-io/eksctl/issues/7425