[EKS] [request]: Manage IAM identity cluster access with EKS API

ayosec commented 5 years ago

Tell us about your request

CloudFormation resources to register IAM roles in the aws-auth ConfigMap.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

A Kubernetes cluster managed by EKS is able to authenticate users with IAM roles. This is very useful to grant access to Lambda functions. However, as described in the documentation, every IAM role has to be registered manually in a ConfigMap with the name aws-auth.

For every IAM role we add to the CloudFormation stack, we have to add an entry like this:

mapRoles: |
  - rolearn: "arn:aws:iam::11223344:role/stack-FooBarFunction-AABBCCDD"
    username: lambdafoo
    groups:
      - system:masters
  - ...

This process is a bit tedious, and it is hard to automate.

It will be much better if those IAM roles can be registered directly in the CloudFormation template. For example, with something like this:

LambdaKubeUser:
  Type: AWS::EKS::MapRoles::Entry
  Properties:
    Cluster: !Ref EKSCluster
    RoleArn: !GetAtt FunctionRole.Arn
    UserName: lambdafoo
    Groups:
      - system:masters

Thus, CloudFormation will add and remove entries in the ConfigMap as necessary, with no extra manual steps.

Another AWS::EKS::MapUsers::Entry can be used to register IAM users in mapUsers.

With this addition, we can automate the extra step to register the IAM role of the worker nodes when a new EKS instance is created:

NodeInstanceKubeUser:
  Type: AWS::EKS::MapRoles::Entry
  Properties:
    Cluster: !Ref EKSCluster
    RoleArn: !GetAtt NodeInstanceRole.Arn
    UserName: system:node:{{EC2PrivateDNSName}}
    Groups:
      - system:bootstrappers
      - system:nodes

matthewcummings commented 2 years ago

@andre-lx your best best is to switch to Terraform or Pulumi and use the Kubernetes provider. CloudFormation only natively works with AWS resources i.e. not Kubernetes.

vennemp commented 2 years ago

https://aws.amazon.com/blogs/containers/bootstrapping-clusters-with-eks-blueprints/

Somewhat of a step in the right direction.

vbala81 commented 2 years ago

@vennemp Thanks for sharing the link. This is still only for CDK and Terraform options. I am also trying to automate the eks authorization similar to others in this group and I can only use CFT due to project constraints. I will also raise this up with my TAM in AWS to see if they can expedite this request

andre-lx commented 2 years ago

Not the optimal solution, but, from our understanding the only way of achieving this in CloudFormation, currently, is using the AWSQS::EKS::Cluster

https://github.com/aws-quickstart/quickstart-amazon-eks-cluster-resource-provider

The configuration is identical to the official EKS Cluster, and it's working properly.

Unfortunately, this means that we need to add an extra step setting the Execution IAM Role and activating the extension.

MarcusRosen-Rio commented 2 years ago

Hey guys.

I'm searching everywhere and can't find the answer, right now, how can we accomplish this?

So, I'm creating a private cluster through CloudFormation and want to give access to a role used by an ECS Task. The security groups are configured and now the ECS task can access the cluster (network), but I'm receiving the "error: You must be logged in to the server (Unauthorized)" as expected.

There are any chance of setting this role to the "aws-auth" configmap in order to make this connection possible?

Thanks

You can use the custom AWS QuickStart Cloudformation type, if you want to do this completely through CloudFormation. We use it and it works ok but it's a workaround.. https://github.com/aws-quickstart/quickstart-amazon-eks-cluster-resource-provider/blob/main/README.md

aavileli commented 2 years ago

AWS needs to make an API for configuring aws-auth to configure access to cluster to make provisioning tools like terraform handle it correctly when re-applying config changes

musayev-io commented 2 years ago

+1 -- this should be natively supported, not a jerry-rig through custom resources.

abebars commented 2 years ago

+1 on this. I am running into a use case where I need to control the IAM roles dynamically for a K8s operator. While it's doable to turn off the AuthMap in Terraform and manage the config map programmatically, it will be hugely benificial and more efficient to handle through AWS EKS API instead. Any idea how far is this feature from being released?

sharma-abhishek-it commented 2 years ago

+1

igorGevaerd commented 2 years ago

This really would help automate things up, removing the manual changes!

MartinEmrich commented 2 years ago

The discussion so far seems to mostly revolve around IaC and configuration, where numerous workarounds or 3rd-party-plugins seem to be sprouting everywhere.

But the current implementation seems to be actually quite fragile, even dangerous:

The User (exactly: The IAM ARN) who initially created the cluster always has full privileges on the cluster. This is hidden somewhere deep inside AWS, I read many times that it cannot be viewed or changed. Even a small typo in the aws-auth ConfigMap makes it ineffective. (The fact that mapUsers and mapRoles are inline multiline strings does not help here).

So there are easy scenarios where you can lock yourself out of your own EKS cluster:

Maybe the IAM user who created it moved on to another team, and can no longer help. (in that case, you cannot even lock it out easily). Or the cluster is so old, that no one even knows who created it in the first place, so you cannot recreate the very IAM user who did it. Or you moved from IAM users to AWS IAM Identity Center (SSO), which is best practice anyways.... And doing AWS SSO with EKS (PermissionSets etc.) and the aws-auth ConfigMap is IMHO kludgy as well....

If you now damage your aws-auth ConfigMap accidentally, there's no way back in. Furthermore, all the node group roles referenced no longer work either, leaving the cluster in a broken state if e.g. Cluster Autoscaler is scaling node groups.

IMHO the only way is to get rid of that "initial creator" concept entirely, and provide a validated (meaning bullet-proof) alternative to the aws-auth ConfigMap, held outside of the actual Kubernetes cluster.

What's your take on this?

matthewcummings commented 2 years ago

@MartinEmrich not only is it dangerous, it's also counter-intuitive. The default cluster admin permission are unique in the AWS ecosystem. I would bet that most/all of the people following this issue wasted hours debugging their cluster configurations, the first time they encountered this behavior, I definitely did.

mikestef9 commented 2 years ago

@MartinEmrich appreciate the feedback and concern. What you are describing is exactly what we are building. An EKS API replacement for the aws-auth config map. With this API, you will also be able to control whether or not the cluster creator has admin privileges.

blakepettersson commented 2 years ago

Is there a timeline (months, years, ...) for this feature?

ayosec commented 2 years ago

@mikestef9 Thanks for the update!

Can you share any estimates of when it will be available?

Will be possible to use that API entirely from CloudFormation?

jking916 commented 2 years ago

@MartinEmrich You can use an IAM role to create the cluster, which is a much better approach than using an IAM user. Any admin that can assume the role can still manage the cluster if the aws-auth configmap gets misconfigured.

An API to replace the brittle aws-auth configmap would be much more useful though as others have mentioned. This would make deploying EKS much simpler from an IaC perspective.

mikestef9 commented 2 years ago

Can't give anything too specific in this forum - but this is in active development. Will move to coming soon once I feel it aligns the roadmap categories defined here https://github.com/aws/containers-roadmap#faqs And yes, this will be an AWS API - and it will include CloudFormation support.

MartinEmrich commented 2 years ago

@jaredk-beta thanks, that's probably what we will do at least for new clusters until the proper solution is available.

m1mohamad commented 1 year ago

I made a mistake and updated my cluster with the following configmap while I was trying to test why nodes previously arent getting READY , so I lost access to the cluster, then I used the same role below to access the cluster again but I lost my system:masters permissions ! I can't get do kubectl get configmaps , is there a way to regain access ?

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: 'arn:aws:iam::xxxxxxx:role/testyy-NodeInstanceRole-1FQVVVZPS0TDP'
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

aservedio-swi commented 1 year ago

I made a mistake and updated my cluster with the following configmap while I was trying to test why nodes previously arent getting READY , so I lost access to the cluster, then I used the same role below to access the cluster again but I lost my system:masters permissions ! I can't get do kubectl get configmaps , is there a way to regain access ?
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: 'arn:aws:iam::xxxxxxx:role/testyy-NodeInstanceRole-1FQVVVZPS0TDP'
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

@m1mohamad - the user that you used to create the cluster does not appear in this configmap and should still have access. If you are able to use credentials from them, I think you should be able to fix your configmap.

LeoSpyke commented 1 year ago

How is it possible that a crucial feature like this it is still in "Open" state after 4 years?

mikestef9 commented 1 year ago

This is a major new feature, so we are being careful with testing and rollout, but this is coming soon. This feature will address the following open issues #554 #922 #1209 #1278 #573 #795 #822 #727 #926.

At launch, it will not make the integration with SSO any simpler #474, but we will share details on a follow on enhancement that will directly integrate with AWS IAM Identity Center Permission sets.

yindia commented 1 year ago

@mikestef9 Do you have any timeline for release?

ClaytonOlleyNutrien commented 1 year ago

I'm in dire need of this functionality as well. Managing EKS via Terraform has been quite a nightmare for me with various issues with aws-auth cropping up constantly with no apparent long term fixes.

MartinEmrich commented 1 year ago

@mikestef9 I understand that doing this "right" (no breaking changes, no possibilities to shoot in the own foot, no security holes) is a big challenge and it takes time.

But until the Big Bang: Could you have the AWS team at least implement a simple API that allows viewing and replacing the currently hidden IAM principal (that created the cluster in the first place and always has full admin access regardless of the aws-auth ConfigMap)?

I bet that would be comparatively simple, and would save a lot of bacon (or veggie equivalents) around here...

Thanks so much!

mikestef9 commented 1 year ago

You are right that request isn't directly related to this feature, but we are coupling support for that request together in the same launch and set of deployments, so at this point there isn't any way to decouple them.

I know how many of you are waiting on this feature, so I'll share a sneak preview of the user guide for this particular request to control cluster creator admin permissions.

The following command will create a cluster without any IAM identity having admin access to the cluster.

aws eks create-cluster \
  --name my-cluster \
  --role-arn arn:aws:iam::012345678910:role/myClusterRole \
  --resources-vpc-config subnetIds=subnet-6782e71e,subnet-e7e761ac
  --no-bootstrap-cluster-creator-admin-permissions

During or after cluster creation, you can create a standard access entry with whatever desired IAM identity as a cluster admin.

For existing clusters, after being updated to an EKS platform version that supports access entry and opting in to access entry APIs, the original cluster creator IAM identity will now be returned as an existing access entry. You can choose to delete this access entry if desired.

aws eks list-access-entries --cluster-name my-existing-cluster 
{
  "accessEntries": [ 
     "arn:aws:iam::012345678910:role/EKSClusterCreatorCICDRole",
   ]
}

aws eks delete-access-entry --cluster-name my-existing-cluster \
  --principal-arn arn:aws:iam::012345678910:role/EKSClusterCreatorCICDRole

If you don’t set the admin bootstrap parameter (or explicitly set it to True), EKS will automatically create a cluster admin access entry on your behalf. The bootstrap parameter can be set independent of cluster authentication mode (meaning you can set this parameter even with authentication mode set to CONFIG_MAP). However, if you have your cluster authentication mode set to CONFIG_MAP, the cluster creator Kubernetes admin role will effectively remain hidden, because the access entry APIs are disabled.

Note: The value that you set for bootstrapClusterCreatorAdminPermissions is not returned in the EKS DescribeCluster API. Further changes to access control post cluster creation will always be performed with access entry APIs.

dvins commented 1 year ago

This is exactly the API we are building that I referenced in a previous comment. We'll have a handful of pre-baked polices (Admin, Editor, Viewer) that you can attach through the API at a cluster wide or specific namespace level, without ever having to add anything in RBAC. You can still of course use RBAC if you need to define more fine-grained permissions per IAM entity.

We're approaching two and a half years since this comment. Is there a timeframe as for when this is going to finally be available? The recent comments that the API support is tightly coupled with a whole bunch of other capabilities is disheartening given how long this issue has been outstanding.

dvins commented 1 year ago

This is a major new feature, so we are being careful with testing and rollout, but this is coming soon. This feature will address the following open issues #554 #922 #1209 #1278 #573 #795 #822 #727 #926.

At launch, it will not make the integration with SSO any simpler #474, but we will share details on a follow on enhancement that will directly integrate with AWS IAM Identity Center Permission sets.

Also, regarding SSO, how closely (or not) will this support follow?

ceresdarkmatter commented 1 year ago

This is exactly the API we are building that I referenced in a previous comment. We'll have a handful of pre-baked polices (Admin, Editor, Viewer) that you can attach through the API at a cluster wide or specific namespace level, without ever having to add anything in RBAC. You can still of course use RBAC if you need to define more fine-grained permissions per IAM entity.

We're approaching two and a half years since this comment. Is there a timeframe as for when this is going to finally be available? The recent comments that the API support is tightly coupled with a whole bunch of other capabilities is disheartening given how long this issue has been outstanding.

Knowing the history of this garbage fire of a company - probably this will never see the light of day

Let's see what is the new excuse for this situation from these excuse of drones... ah no, what was the accepted word for them? Ah yes, "employees" I think

bryantbiggs commented 1 year ago

@ceresdarkmatter lets please be civilized and avoid negative/derogatory comments

ceresdarkmatter commented 1 year ago

@ceresdarkmatter lets please be civilized and avoid negative/derogatory comments

Bryant, I am trying but see the results of a multy-year civilised conversation. I mean - you have to get the attention about something in a way. I am loosing my patience with updates every few months that just move the goal posts. Those only indicate that probably besides the dog and pony show AWS puts in marketing materials they don't really care about the actual needs of paying users - and this situation is not the first time that I have seen this exact scenario unfolding with them. I am seriously asking what are professional ways to solve this systemic problem besides moving to another cloud provider

edit: this issue has been opened just ~4.5 years ago. not days, weeks or months

jking916 commented 1 year ago

@ceresdarkmatter As a paying customer, if you don't like what AWS is doing then take your business to another cloud provider that you feel cares more about their customers. But insulting the AWS employees in here because they don't share your viewpoint isn't appropriate or helpful.

ceresdarkmatter commented 1 year ago

@ceresdarkmatter As a paying customer, if you don't like what AWS is doing then take your business to another cloud provider that you feel cares more about their customers. But insulting the AWS employees in here because they don't share your viewpoint isn't appropriate or helpful.

@jking916 Those AWS employees are in charge of getting this issue fixed, do you realise that? It is 100% on them to get their shit together and make it happen. Them not doing that are an insult to everyone that explicitly requested this since before the Pandemic. I am not cutting them any slack and the fact that absolutely no one is trying to keep them accountable is absolutely mindblowing.

Having said that I see that I have no one to discuss with and you guys are part of the problem. This is the last reply I am going to post... unbelievable

gazal-k commented 1 year ago

I think I can empathize with AWS on this one. Technology teams working on popular product platforms would like to work on a whole bunch of things, but the reality is that it is impossible to prioritise all of them. The teams around EKS have been consistently improving EKS & related systems with new features & enhancements. Would I love to see this improvement to EKS <> IAM? Absolutely, but we've been very happy with EKS since launch (back in 2019, I believe).

nnsense commented 1 year ago

The way @ceresdarkmatter is phrasing his message isn't the most calm, OK, but.. have you ever found yourself with a prod cluster locked up because of this issue? with tens devs unable to access the cluster and you eventually having to fix it manually because terraform keeps showing "Unauthorized"? It happened to me, many times over these years, bear in mind this issue has been opened in 2019 but I've had this problem before that. Sure we need to be polite, but I can understand the frustration, and I hope the feature fixing these problems will be published as soon as possible.

gazal-k commented 1 year ago

We create and manage all infra using CI / CD. The cluster is created using the pipeline's IAM Role. That way, we can never lock ourselves out. If you don't have such a pipeline, you can still use a Role to create clusters (for now) to prevent locking yourself out. The locking out part wasn't our primary motivator for this mechanism, just that we wanted to ensure there was some rigour around how we managed infra.

vennemp commented 1 year ago

@gazal-k is right roles are the best for creation and you can even give them multiple trust relationships - though I know some corporate policies do not permit this. One thing I tested, if you create a cluster with an IAM user and that user gets deleted - if you re-create an iam user with the same name, you should still be able to access the cluster with the new IAM user since the principal is still the same.

RothAndrew commented 10 months ago

@mikestef9 any update?

ab77 commented 10 months ago

Going to leave this here, in case it helps someone. We use the following workaround using clunky CFN custom resources. Implementation detail, broadly:

push the Lambda image from public https://gallery.ecr.aws/k4v3r5y6/cfn-generic-provider to your ECR private repo
deploy custom resource Lambda into your account/region
add a custom resource to your EKS stack(s)

... (e.g.) custom resource function (one per account/region)

  Function:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: custom-resource
      Code:
        ImageUri: !Ref EcrRepositoryUri
      PackageType: Image
      # both architectures are in the manifest, pick one
      Architectures:
        - !Ref EcrRepositoryImageArch
      Role: !GetAtt Role.Arn
      Timeout: 300
      Environment:
        Variables:
          VERBOSE: 1

... (e.g.) custom resource usage example

  PatchConfigMapAwsAuth:
    Type: 'Custom::PatchConfigMap'
    Version: 1.0
    Properties:
      # assume cluster creation role to perform AWS/EKS operations (e.g. GitHub OIDC role used by CI)
      RoleArn: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/github-oidc"
      # reference to the Lambda created in the first step
      ServiceToken: !Sub 'arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:custom-resource'
      AgentService: eks
      AgentType: custom
      AgentCreateMethod: update_aws_auth_configmap
      AgentCreateArgs:
        ClusterName: !Sub "${ClusterName}"
        AgentRegion: !Sub "${AWS::Region}"
        RoleArn: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/admin"
        Username: admin
        Groups:
          - admin
          - cluster-admin
          - system:masters
      AgentUpdateMethod: update_aws_auth_configmap
      AgentUpdateArgs:
        ClusterName: !Sub "${ClusterName}"
        AgentRegion: !Sub "${AWS::Region}"
        RoleArn: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/admin"
        Username: admin
        Groups:
          - admin
          - cluster-admin
          - system:masters
      AgentDeleteMethod: delete_aws_auth_configmap
      AgentDeleteArgs:
        ClusterName: !Sub "${ClusterName}"
        AgentRegion: !Sub "${AWS::Region}"
        RoleArn: !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/admin"

The example custom resource will call this code and patch aws-auth config map.

The code may expose your AWS session keys/k8s tokens in the CloudWatch Lambda logs.

aws-auth ConfigMap must already exist (i.e after the first nodegroup is created).

jagu-sayan commented 10 months ago

Or if you use CDK : https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources.AwsCustomResource.html

ab77 commented 10 months ago

Or if you use CDK : https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.custom_resources.AwsCustomResource.html

Yeah, that's an option if the resources are created with CDK. We currently don't need/use OO IAC, so we still write plain CFN templates YML. Using a custom resource seems like a reasonable approach, which is basically what aws-cdk does under the hood anyway.

rrsela commented 9 months ago

Isn't this request fulfilled with the latest API change regarding AccessEntry? See this blog post for details..

bryantbiggs commented 9 months ago

Yes!

seifrajhi commented 9 months ago

great job team !!!

mikestef9 commented 9 months ago

EKS now supports a simplified method for managing IAM authentication and authorization.

mikestef9 commented 9 months ago

More details on the feature

Enable Access Management APIs

EKS access entry APIs require you to opt in before use. This can be done during cluster creation or updated on existing clusters. You must first ensure your cluster is on a supported platform version.

Supported platform versions

Kubernetes version	EKS Platform version
1.29+	any
1.28	eks.6
1.27	eks.10
1.26	eks.11
1.25	eks.12
1.24	eks.15
1.23	eks.17

aws eks create-cluster \
   --name my-cluster \
   --role-arn arn:aws:iam::012345678910:role/myClusterRole \
   --resources-vpc-config subnetIds=subnet-6782e71e,subnet-e7e761ac
   --access-config authenticationMode=API

Existing cluster

aws eks update-cluster-config \
   --name my-cluster \
   --access-config authenticationMode=API

EKS supports the following three values for authenticationMode:

CONFIG_MAP: The cluster will source authenticated IAM principals only from the aws-auth ConﬁgMap (this currently the default value for newly created clusters).
API_AND_CONFIG_MAP: The cluster will source authenticated IAM principals from both EKS access entry APIs and the aws-auth ConﬁgMap, with the EKS access entry APIs taking priority.
API: The cluster will source authenticated IAM principals only from EKS access entry APIs.

In a to be determined future Kubernetes version, EKS will stop supporting the aws-auth ConfigMap as an authentication source, and you will be blocked from upgrading if you haven't switched your cluster authenticationMode to opt in to access entry APIs. Users are strongly encouraged to migrate to access entry APIs as soon as possible.

Note: The default selected option at launch for new clusters in the EKS Console is API_AND_CONFIG_MAP.

Switching authentication modes on an existing cluster is a one-way operation. You can switch from CONFIG_MAP to API_AND_CONFIG_MAP. You can then switch from API_AND_CONFIG_MAP to API. You cannot revert these operations in the opposite direction. Meaning you cannot switch back to CONFIG_MAP or API_AND_CONFIG_MAP from API. And you cannot switch back to CONFIG_MAP from API_AND_CONFIG_MAP.

eks:authenticationMode is a supported IAM condition key, so you can write IAM policies that enforce clusters be created with a certain authentication mode.

Managing/Modifying Cluster Admin

Control over which identity has cluster admin permissions when creating a cluster. The following command will create a cluster without any IAM identity having Kubernetes cluster admin access to the cluster.

aws eks create-cluster \
   --name my-cluster \
   --role-arn arn:aws:iam::012345678910:role/myClusterRole \
   --resources-vpc-config subnetIds=subnet-6782e71e,subnet-e7e761ac
   --access-config authenticationMode=API
   --no-bootstrap-cluster-creator-admin-permissions

Important : bootstrapClusterCreatorAdminPermissions can only be set to False if the cluster authenticationMode is set to either API or API_AND_CONFIG_MAP.

After cluster creation (or during, as you can create access entries while a cluster is in CREATING state), you can create a standard access entry with whatever desired IAM identity as a cluster admin.

For existing clusters with authenticationMode set to CONFIG_MAP, after being updated to an EKS platform version that supports access entry and changing authenticationMode to use access entry APIs, the original IAM identity that created the cluster will be returned as an existing access entry. You can choose to delete this access entry if desired.

aws eks list-access-entries --cluster-name my-existing-cluster

{
  "accessEntries": [
    "arn:aws:iam::012345678910:role/EKSClusterCreatorCICDRole",
  ]
}

aws eks delete-access-entry --cluster-name my-existing-cluster \
   --principal-arn arn:aws:iam::012345678910:role/EKSClusterCreatorCICDRole

GitHub issues resolved:

Add AdminRole option at cluster creation #554
Add an ability to view and update the IAM entity user or role that is automatically granted system:masters permissions in the cluster's RBAC configuration #922

If you don't set the admin bootstrap parameter (or explicitly set it to True), EKS will automatically create an access entry with Kubernetes cluster administrator permissions on your behalf.

Note: The value that you set for bootstrapClusterCreatorAdminPermissions on cluster creation is not returned in the response of subsequent EKS DescribeCluster API calls. This is because the value of that field post cluster creation may not be accurate. Further changes to access control post cluster creation will always be performed with access entry APIs. The ListAccessEntries API is the source of truth for cluster access post cluster creation.

Recover cluster admin access

Previously if you made a typo in the aws-auth ConfigMap you could lock yourself out of the cluster. Or you could lose access to cluster if you accidentally deleted cluster creator IAM identity and didn't remember the IAM identity to re-create it. Now, as long as an IAM identity has permission to create access entries, you can always recover Kubernetes API access to your cluster.

aws eks create-access-entry --cluster-name my-cluster  \
   --principal-arn arn:aws:iam::012345678910:role/Admin

 aws eks associate-access-policy --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/Admin \
   --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
   --access-scope '{"type": "cluster"}'

GitHub issues resolved:

Allow customer to rollback aws-auth configmap when update the configuration wrong (Solving the goal of the request) #1209

View Kubernetes resources from EKS Console

A common challenge is EKS Console users (who didn't create the cluster and have Kubernetes cluster admin permissions by default) struggling to figure out why they can't view Kubernetes resources in the Console. Previously, a separate IAM principal with cluster access would have to add an additional aws-auth ConfigMap entry for that Console user's IAM principal. Now, a pop up will appear for IAM principals using the EKS Console that takes them to the create access entry screen with their IAM principal pre-popluated. As the long as the Console user has IAM permissions to call EKS access entry APIs, they can give themselves Kubernetes permissions from the Console itself without needing to create aws-auth ConfigMap entries anymore.

Note: The Console user's IAM identity still first requires the permission eks:AccessKubernetesApi to view resources in the console, irrespective of if they have Kubernetes level permissions granted through access entry APIs.

GitHub issues resolved:

Manage aws-auth ConfigMap in AWS Console (access entries don't actually interact with aws-auth ConfigMap, but this solves the goal of the request) #1278

Authentication

The following examples show to authenticate IAM principals to an EKS cluster, as the preferred alternative to the aws-auth ConfigMap. If no EKS access policies are attached to the principal (as shown in authorization examples below), no Kubernetes permissions are granted, but the principal will be able to successfully authenticate to the cluster, and more fine-grained authorization can be implemented using RBAC.

Username is not a required input when creating an access entry. If not specified, the username value passed to Kubernetes for authentication decisions will map to the value returned from the IAM STS get-caller-identity API. For IAM users, this is the original IAM user ARN. For IAM roles, this is the assumed role ARN with session name, of the format arn:aws:sts::012345678910:assumed-role/<role-name-without-path>/{{SessionName}}. This value can change depending on the session name, so for use with RBAC, you will want to set Kubernetes groups on the access entry to reference in Kubernetes role bindings.

We recommend leaving custom username blank, as the generated username includes the session name, which will appear in CloudTrail logs as an added security benefit. If using a custom username with IAM roles, it is strongly encouraged to set the {{SessionName}} macro. You can specify the following macros in custom usernames {{AccessKeyID}}, {{AccountID}}, {{SessionName}}. Custom usernames can't start with system:, eks:, aws:, amazon: , or iam:.

Note: IAM principals referenced in an access entry must already exist in IAM.

Authenticate an IAM user to a cluster

aws eks create-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:user/MyUser

Note: The Kubernetes username used for further authorization RBAC in this case will be the same as the input principal ARN arn:aws:iam::012345678910:user/MyUser.

Note: If you change your IAM username, that user will no longer have access to the cluster. You will need to recreate an access entry with the updated IAM user ARN.

Note: IAM best practices recommends using roles with temporary credentials, rather than users with long-term credentials.

Authenticate an IAM role to a cluster and specify Kubernetes groups

aws eks create-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole \
   --kubernetes-groups dev test

Note: The Kubernetes username generated in this case will be arn:aws:sts::012345678910:assumed-role/MyRole/<session-name>.

Authenticate an IAM role to a cluster with a custom username

aws eks create-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole \
   --username my-username

Note: No groups are specified here, because the static username of my-username can be referenced in Kubernetes RBAC policies. However, this is not recommended, as the assumed role session name will be lost from Kubernetes audit logs.

Authenticate an IAM role to a cluster with a custom username using macros

aws eks create-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole \
   --username my-username:{{AccountID}}:{{SessionName}}
   --kubernetes-groups dev test

Note: The macros will be expanded when username reaches Kubernetes for further RBAC authorization decision.

Note: As illustrated in this example, when using the {{SessionName}} macro, a : must be part of the custom username at some point before the macro (does not have to directly precede as in example above).

Update the username and/or groups of an existing access entry

aws eks update-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole \
   --username my-other-username
   --kubernetes-groups my-other-group

Note: Be careful performing this operation if you already have existing RBAC resources that reference previous username/groups. RBAC will also have to be updated to support the new username/groups.

Note: Groups are declarative. Enter all desired groups with the update call.

Creating an access entry with a role ARN containing a path

Authentication decision for EKS access entries are matched against the IAM principal ID of an IAM identity, rather than a string match against input value of the IAM identity and returned assumed role response from sts get-caller-identity (as is the case with entries in the aws-auth ConfigMap).

When creating an EKS access entry, enter the fully qualified IAM role ARN like below (do not strip off the path like you need to do in the aws-auth ConfigMap today)

aws eks create-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/teams/tooling/devToolingOperator

Note: The username in returned access entry will not include the path: arn:aws:sts::012345678910:assumed-role/devToolingOperator/{{SessionName}}.

Authenticate an IAM principal from a separate account

From account 012345678910 run the following

aws eks create-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::567890123456:role/myOtherAccountRole

An identity assuming the myOtherAccountRole role in account 567890123456 is now able to authenticate to the my-cluster cluster in account 012345678910.

Eventual Consistency

Similar to AWS IAM behavior, EKS access entries are eventually consistent, and may take several seconds to be effective after the initial API call returns successfully. You must design your applications to account for these potential delays. We recommend that you do not include access entry create/updates in the critical, high-availability code paths of your application. Instead, make changes in a separate initialization or setup routine that you run less frequently. Also, be sure to verify that the changes have been propagated before production workflows depend on them.

GitHub issues resolved:

Manage IAM identity cluster access with EKS API #185
Federated roles containing paths don't work properly with EKS #573
Privilege Escalation using aws-auth configmap #795
Add aws-iam-authenticator as a CRD (again, this feature does not interact with the aws-auth config map, but solves the goal of the request) #550

Authorization

The Kubernetes API server supports an ordered list of authorization modules, allowing modules to approve, deny, or pass on a particular action. EKS clusters now include an additional authorizer separate from RBAC, and you can optionally attach EKS defined access policies to give Kubernetes authorization permissions to the IAM principal authenticated with an access entry. Authorization can be done on a namespace or cluster scope. You can specify up to 25 namespace selectors, with optional wildcard suffix matching, to restrict that principals access to specific namespaces.

In EKS clusters, RBAC policies are currently evaluated prior to EKS access policies, so any Kubernetes Role Based Access Control (RBAC) ClusterRoleBinding or RoleBinding that specifies an IAM principal will be evaluated for an allow authorization decision before EKS Access Policies. This may change in the future, although in practice the order today does not matter, because neither RBAC nor EKS access policies support deny actions.

EKS predefines several managed access policies that mirror the default Kubernetes user facing roles, including cluster-admin, admin, edit, and, view. To view the list of all policies you can run the following command

aws eks list-access-policies

The examples below assume you have already created the access entry to authenticate the IAM principal to the cluster.

Give read only permissions for an IAM principal to an entire cluster

aws eks associate-access-policy --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole \
   --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy
   --access-scope '{"type": "cluster"}'

Give editor permissions for an IAM principal to specific namespaces

aws eks associate-access-policy --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole2 \
   --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy
   --access-scope '{"type": "namespace", "namespaces": ["app-a", "app-b"]}'

Give admin permissions to an IAM identity to access multiple namespaces with a wildcard selector

aws eks associate-access-policy --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole3 \
   --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSAdminPolicy
   --access-scope '{"type": "namespace", "namespaces": ["dev-*"]}'

Update list of allowed namespaces for an existing access entry

aws eks associate-access-policy --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole2 \
   --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSEditPolicy
   --access-scope '{"type": "namespace", "namespaces": ["app-a", "app-c"]}'

Node Authentication and Authorization

Access entries have an optional input type field for use with authorizing worker node roles to join a cluster. The following types are supported by the access entry API:

EC2_LINUX
EC2_WINDOWS
FARGATE_LINUX
STANDARD

STANDARD is the default type and will be set as the default if not passed on access entry creation. This is the default experience for non-node role access entries. Similar to with the aws-auth ConfigMap today, there is no need to manually create access entries for IAM roles used with managed node groups and Fargate. MNG and Fargate will continue to manage node entries on your behalf, irrespective of authenticationMode. When you switch an existing cluster authenticationMode from CONFIG_MAP to API_AND_CONFIG_MAP entries in the aws-auth ConfigMap managed by MNG and Fargate will be automatically migrated to node type access entries. Once your cluster is migrated to an access entry supporting authentication mode, MNG and Fargate will only manage entries using access entry APIs, not the aws-auth ConfigMap.

Note: aws-auth ConfigMap entries not associated with MNG or Fargate are not automatically migrated when switching authentication modes. These entries must be migrated manually.

Note: In practice, you will never need to manually use the FARGATE_LINUX type when creating an access entry, because that entry would always be created on your behalf by the EKS/Fargate service. If you do pre-create the access entry, EKS/Fargate will simply skip trying to create the entry itself.

The examples below would be required if using self-managed node groups or Karpenter.

aws eks create-access-entry --cluster-name my-cluster  \
   --principal-arn arn:aws:iam::012345678910:role/MyLinuxNodeRole \
   --type EC2_LINUX

aws eks create-access-entry --cluster-name my-cluster  \
   --principal-arn arn:aws:iam::012345678910:role/MyWindowsNodeRole \
   --type EC2_WINDOWS

Note: It is required to use separate IAM roles for the EC2_LINUX, EC2_WINDOWS, and FARGATE_LINUX types.

Note: You cannot attach EKS access policies to access entries with type of EC2_LINUX, EC2_WINDOWS, or FARGATE_LINUX. EKS automatically handles the permissions required for worker nodes with those access entries to join the cluster.

Note: Cross account entries are not permitted for node type access entries.

Note: The IAM identity creating node type access entries of either EC2_LINUX or EC2_WINDOWS must have iam:PassRole permission. This permission will be validated by EKS, and a failure will be returned if not present.

GitHub issues resolved:

separate aws-auth for worker node and others #822
add parameter at cluster creation for worker IAM role #727
Managed Node Groups using role with extra path do not join cluster #926

Migration fully off the aws-auth ConfigMap

The aws-auth ConfigMap can still be used until the to be announced future Kubernetes version where it is no longer supported. But migration to access entry API is strongly encouraged.

Migrate an entry in the aws-auth ConfigMap to access entry API

For an existing cluster, change your cluster authenticationMode from CONFIG_MAP to API_AND_CONFIG_MAP. Before deleting anything in the aws-auth ConfigMap, create equivalent access entries. Specify the same username and/or groups already used the aws-auth ConfigMap if necessary. After access entry creation, then delete the corresponding entry from the ConfigMap and validate your environment. Once you have migrated everything to access entry APIs, you can switch your cluster authenticationMode to API, to turn off the aws-auth ConfigMap as a source for authenticated IAM principals.

AWS Service Integrations

The access entries feature simplifies how other AWS services can obtain Kubernetes permissions needed to perform actions against EKS clusters. In the future, AWS services with EKS integrations (including EMR, Batch, Resilience Hub) will update their Service Linked Roles to contain permissions to create access entries. When you take an action in that service, the service SLR will automatically create an access entry and associate a specific EKS access policy for that service. There is no need for users to manually create access entries and associated permissions. For example, when Resilience Hub migrates to access entry, all of the steps currently outlined in this doc page will no longer be required. You can think of these "service-linked access entries" as the Kubernetes permission equivalent of AWS service-linked-roles.

Example policy a future service might add to its SLR to give itself permissions to create a specific access entry with only specific associated access policies.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "eks:CreateAccessEntry",
      "Resource": "arn:aws:eks:*:*:cluster/*",
      "Condition": {
        "ArnEquals": {
          "eks:principalArn": "${aws:principalArn}"
        },
        "StringEquals": {
          "aws:RequestTag/aws:eks:managed": "${aws:principalArn}"
        },
        "StringEqualsIfExists": {
           "eks:username": ""
        },
        "ForAllValues:StringEqualsIfExists": {
           "eks:kubernetesGroups": [ "" ]
        }
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": "eks:DeleteAccessEntry",
      "Resource": "arn:aws:eks:*:*:access-entry/*/role/*",
      "Condition": {
        "ArnEquals": {
          "eks:principalArn": "${aws:principalArn}"
        },
        "StringEquals": {
          "aws:ResourceTag/aws:eks:managed": "${aws:principalArn}"
      }
    },
    {
      "Effect": "Allow",
      "Action": "eks:AssociateAccessPolicy",
      "Resource": "arn:aws:eks:*:*:access-entry/*/role/*",
      "Condition": {
        "ArnEquals": {
          "eks:principalArn": "${aws:principalArn}"
        },
        "StringEquals": {
          "eks:policyArn": "arn:aws:eks::aws:cluster-access-policy/AmazonEKSNodeOperatorClusterPolicy",
          "eks:accessScope": "cluster",
          "aws:ResourceTag/aws:eks:managed": "${aws:principalArn}"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": "eks:AssociateAccessPolicy",
      "Resource": "arn:aws:eks:*:*:access-entry/*/role/*",
      "Condition": {
        "ArnEquals": {
          "eks:principalArn": "${aws:principalArn}"
        },
        "StringEquals": {
          "eks:policyArn": "arn:aws:eks::aws:cluster-access-policy/AmazonEKSNodeOperatorPolicy",
          "eks:accessScope": "namespace",
          "aws:ResourceTag/aws:eks:managed": "${aws:principalArn}"
        },
        "ForAllValues:StringEquals": {
          "eks:namespaces": [
            "my-namepsace" 
          ]
        }
      }
    }
  ]
}

You can use this example as a reference to write your own IAM policies that scope down what type of access entries IAM principals can create.

IAM policy control for access entries EKS access entry APIs support condition context keys to enable fine grained control over creation/management of access entries.

EKS supports the following condition keys associated with this feature

eks:principalArn
eks:username
eks:kubernetesGroups
eks:accessScope
eks:namespaces
eks:policyArn
eks:authenticationMode

Deletion

Deleting an access entry will remove any associated access policies. There is no need to first disassociate access policies.

aws eks delete-access-entry --cluster-name my-cluster \
   --principal-arn arn:aws:iam::012345678910:role/MyRole

Deleting a cluster will also automatically delete any access entries currently associated with that cluster.

SSO/AWS IAM Identity Center

There is no direct change to improve this user experience. You still need to add each auto generated role as a separate access entry. But automating that process on your own should be become easier since everything is now an AWS API. We plan to improve this UX in the future by allowing users to directly associate IAM Identity Center permission sets to EKS clusters. Follow this GitHub issue for details: EKS authentication rolearn wildcard support #474.

ab77 commented 8 months ago

There may be a bug in the CloudFormation implementation of this feature, it seems that Acess Policies are lost on STACK_UPDATE (they are present after the initial create though). In the template, resources are defined as follows:

  AccessEntryAdmin:
    Type: AWS::EKS::AccessEntry
    Properties:
      ClusterName:
        Ref: Cluster
      PrincipalArn:
        Fn::Sub: arn:${AWS::Partition}:iam::${AWS::AccountId}:role/admin
      KubernetesGroups:
      - cluster-admin
      AccessPolicies:
      - PolicyArn:
          Fn::Sub: arn:${AWS::Partition}:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy
        AccessScope:
          Type: cluster

.. after update:

The only access policy that isn't wiped, is the one attached to the creator IAM role.

lynnnnnnluo commented 8 months ago

Hello, thanks for your feedback ! I attempted to reproduce the issue in the following way but I wasn't able to reproduce I created Access entry with the template you have provided, then I triggered a stack update by updating the stack tags. Checking EKS console, the access policies are still there.

Could you describe more on the CFN template before/after the stack update so I can try reproducing ?

ab77 commented 8 months ago

Could you describe more on the CFN template before/after the stack update so I can try reproducing ?

Hello, I'll try to reproduce over here again (this is my cluster definition for now):

  Cluster:
    Type: AWS::EKS::Cluster
    Properties:
      Name: !Sub "${AWS::StackName}"
      RoleArn: !GetAtt Role.Arn
      AccessConfig:
        AuthenticationMode: API
        BootstrapClusterCreatorAdminPermissions: true
      ResourcesVpcConfig:
        EndpointPrivateAccess: true
        EndpointPublicAccess: true
        SecurityGroupIds:
          - Fn::ImportValue: !Sub "${ParentClientStack}-ClientSecurityGroup"
        SubnetIds:
          - Fn::ImportValue: !Sub "${ParentVPCStack}-SubnetAPrivate"
          - Fn::ImportValue: !Sub "${ParentVPCStack}-SubnetBPrivate"
      Version: !Ref Version
      EncryptionConfig:
        - Provider:
            KeyArn:
              Fn::ImportValue: !Sub "${ParentKmsKeyStack}-KeyArn"
          Resources:
            - secrets
      KubernetesNetworkConfig:
        IpFamily: !Ref IpFamily

.. also I looked through Cloudtrail logs and there are DisassociateAccessPolicy events from CFN, so it's definitely triggering disassociations for some reason on updates:

    "eventTime": "2023-12-26T20:36:16Z",
    "eventSource": "eks.amazonaws.com",
    "eventName": "DisassociateAccessPolicy",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "requestParameters": {
        "policyArn": "arn%3Aaws%3Aeks%3A%3Aaws%3Acluster-access-policy%2FAmazonEKSClusterAdminPolicy",
        "name": "foo-bar-eks-1",
        "principalArn": "arn%3Aaws%3Aiam%3A%3A1234567890%3Arole%2Fadmin"
    },

lynnnnnnluo commented 8 months ago

Thanks for your reply! It turns out there is a code bug in CFN update, we will be working on the fix as soon as possible.

aws / containers-roadmap