search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.22k stars 321 forks source link

EKS IAM Roles for Service Accounts (Pods) #23

Closed pauncejones closed 5 years ago

pauncejones commented 5 years ago

Update 1/9/19:

After talking about this internally, we've been working on a proposed solution for this. Below is a writeup on what we're thinking, and we've included some example scripts so you can get a feel for how we expect this to work.

Our plan for IAM and Kubernetes integration

A recent Kubernetes feature, TokenRequestProjection, allows users of Kubernetes to mount custom projected service account tokens in their pods. A “projected service account” is a bearer token that is intended for use outside of the cluster. Conveniently, these projected service account tokens are also valid OpenID Connect (OIDC) tokens. AWS IAM has supported OIDC as a federated identity provider since 2014, which has allowed customers to use an external identity to assume an IAM role.

By combining these two features, an application running in a pod can pass the projected service account token along with a role ARN to the STS API AssumeRoleWithWebIdentity, and get back temporary role credentials! In order for this to work properly, there is some setup required to create an OIDC provider, and update an IAM role's trust policy so that the Kubernetes service account for a particular cluster is permitted to assume the role.

Some of the advantages to this approach are that any pod (including host pods) can assume a role, there is not a reliance on Kubernetes annotations for security, there are not any extra processes that need to be run on nodes, and you will be able to have nodes without any IAM permissions of their own.

In the coming months we will be building out functionality in EKS to create and manage OIDC providers for EKS clusters, as well as configuring IAM roles that can be used in an EKS cluster. We will also be adding support for this authentication mechanism in the AWS SDKs.

Totally open for comments, questions or suggestions on this -- let us know in the comments!

Micah Hausler (@micahhausler), System Development Engineer on EKS

shogomuranushi commented 5 years ago

I am using kube2iam. However, I would like to attach an IAM role to Container instead of Pod.

In our use case, the container created by the customer runs on the pod. Our container also exists in the pod. For example, the container that sends customer processing results to s3.

In that case, the customer's container uses the same IAM role as us. To avoid that, I divided the pods and grouped and deployed several pods.

Once IAM roles can be attached at the container level, we do not need to manage such pod groups. There may be few similar use cases, but I'm glad they are supported.

mikkeloscar commented 5 years ago

@shogomuranushi kube-aws-iam-controller could possible solve this issue for now for you :)

blockjon commented 5 years ago

I need this badly.

cafeliker commented 5 years ago

when this nice feature will be released?

meringu commented 5 years ago

Is it possible to bind an IAM role in a different account to the service account?

KIAM currently supports this: https://github.com/uswitch/kiam/issues/166.

awsiv commented 5 years ago

BTW, where doe we get the example scripts? :)

we've included some example scripts so you can get a feel for how we expect this to work.

alex-berger commented 5 years ago

This is really a big thing for container workloads using AWS API(s) and I am really looking forward for this feature. However, this also needs some coordination with all the AWS SDKs to make sure the SDKs add support for this into the default provider chain. Is there any (non final) doucmentation or design proposal about how the SDKs are supposed to detect and use the token? I am asking, because I need to understand how we need to modify our container workloads to being able to support this new mechnism.

alexejk commented 5 years ago

However, this also needs some coordination with all the AWS SDKs to make sure the SDKs add support for this into the default provider chain

I am not entirely sure this will be needed. If the implementation is providing an instance metadata endpoint to the Pod - then current provider chain should work just fine. This is how current solutions like kiam do this to my knowledge (either by redirecting traffic with things like iptables or mocking it in different way).

I would assume the design is setting an annotation for a Pod which role to "assume" whenever the credentials are requested, and then SDK behaves exactly the same as it does today on ECS / EC2.

micahhausler commented 5 years ago

@alex-berger This provider will be in the default credential chain, but we don't have any docs published yet. SDKs will be looking for two environment variables, AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN. For a reference, check out the Go SDK with release v1.21.0 (And retries added in v1.23.2) or in the boto3 docs. We'll be releasing a mutating admisison webhook that will allow you to easily configure your pods with the required fields with full documentation when we release the feature.

@alexejk This approach will not be using a metadata based implementation, but will be in the default SDK credential chains.

mustafaakin commented 5 years ago

Out of curiosity, as AWS_ROLE_ARN is needed, how will the pods restricted to assume only specific roles? How will the security will be ensured? Will the token only allow that ARN to be assumed?

micahhausler commented 5 years ago

@mustafaakin you can modify the role’s trust policy to scope it down to specific Kubernetes service accounts or even a namespace. We’ll have full docs with the feature launch.

glapark commented 5 years ago

I have a question on EKS IAM for Pods related to the current issue. What is being implemented for the current issue is a general solution (which allows different Pods to assume different IAM roles), but there can be a simple scenario in which the same IAM role should be assigned to all Pods running anywhere in an EKS cluster. For this simple scenario, I wonder if there exists an easy solution (other than kiam etc)?

In my application, I create a master Pod using a YAML file, and the Master Pod creates a child Pod. From my experiment, it seems that the master Pod automatically assumes the IAM role associated with its EKS node, whereas the child Pod does not. The solution I am looking for is to let the child Pod assume the IAM role associated with the node where it runs.

Any comment would be appreciated very much.

lra commented 5 years ago

there can be a simple scenario in which the same IAM role should be assigned to all Pods running anywhere in an EKS cluster. For this simple scenario, I wonder if there exists an easy solution (other than kiam etc)

All the pods running on your worker nodes already inherit the IAM role assumed by your worker nodes. Leverage it?

Also this is off-topic, we should keep this issue about IAM roles for pods.

MarcusNoble commented 5 years ago

I'm not sure it's totally off-topic. Will the proposed solution here be addative to the role applied to the worker nodes or will it completely replace it?

If it replaces it (like I believe kube2iam does) then applying IAM roles to your workers will only be applicable for pods NOT specifying their own role.

cmsd2 commented 5 years ago

i would assume that the worker nodes will need PassRole on the roles used by pods, so that the pod roles replace, same as kube2iam.

micahhausler commented 5 years ago

@glapark

can be a simple scenario in which the same IAM role should be assigned to all Pods running anywhere in an EKS cluster.

Yes, but there are 2 parts. You could scope an IAM role to be assumed by any role in a cluster by managing the role's trust policy. Then, you'll have to annotate the service accounts (or provide the env variables/projectedToken volumes) for all the pods you want to assume that same role.

@MarcusNoble

Will the proposed solution here be addative to the role applied to the worker nodes or will it completely replace it?

The pod role would completely replace the node role.

@cmsd2

i would assume that the worker nodes will need PassRole on the roles used by pods, so that the pod roles replace, same as kube2iam.

The worker node role is orthogonal to the pod role, and there is no association between the two. If you were not using ECR, and configured kubelet slightly, you could technically give the node a role with zero permissions, just so it could call sts:GetCallerIdentity for aws-iam-authenticator, and pod roles will still work.

mhausenblas commented 5 years ago

FYI: https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

wjam commented 5 years ago

After looking through the blog and code, it appears that this got announced slightly early? The step of running aws eks describe-cluster --name irptest --query cluster.identity.oidc.issuer --output text won't work as the identity is only present in a fork of the aws-sdk-go library that eksctl is using https://github.com/errordeveloper/aws-sdk-go.

@tabern Do you known when the cli, Go SDK et al will be updated to include the identity information when describing the EKS cluster?

ghost commented 5 years ago

@wjam If you run the AWS CLI commands with --debug you can see the resulting response, and the AWS Console also shows it, but I'm only seeing the field on clusters created since the "release".

micahhausler commented 5 years ago

@wjam you can watch the releases on GitHub for the SDKs to be updated over the course of the day. Here are links for a few languages

micahhausler commented 5 years ago

Closing now that AWS CLI and EKS Docs are live

micahhausler commented 5 years ago

@devkid It is only for new 1.13 and 1.14 clusters today (noted in the docs) but over the coming weeks existing 1.13 clusters will gain an OIDC provider and corresponding identity field. Non-IRSA 1.13 clusters can upgrade to 1.14 and gain an identity field

mattreyuk commented 5 years ago

first of all, congrats! This is great to have. The blog article ends with We will continue to improve IRSA, addressing common asks from the community, including (but not limited to) support for cross-account roles - is there an issue open for cross-account roles I can subscribe to? I looked but couldn't find anything relevant in this repo

micahhausler commented 5 years ago

@mattreyuk I'm not aware of an issue tracking this, but our docs will soon get an update to add cross-account instructions.

The short answer is there are two ways:

[profile second_role]
source_profile = first_role
role_arn=arn:aws:iam::222233334444:role/second-account-role
[profile first_role]
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token 
role_arn=arn:aws:iam::111122223333:role/first-account-role
tabern commented 5 years ago

Hi everyone! We're really excited to announce that IAM for Service Accounts (née IAM for Pods) is now generally available from Amazon EKS.

Overview This feature lets you assign a unique IAM role to a K8s service account that can be used by individual pods running on your Kubernetes cluster. The IAM role can control access to other containerized services, AWS resources external to the cluster such as databases and secrets, or third party services and applications running outside of AWS. You can securely run multiple different services on the same set of nodes, making it easier to optimize costs and availability for the cluster.

Our blog post by @mhausenblas and @micahhausler covers usage examples and more details about how we implemented the feature.

Getting Started Information on how to get started is covered in the EKS documentation

We're excited for you to start using this feature on your Amazon EKS clusters. As always, if you have suggestions for improving this feature, please open an issue on this project. For technical support, please open a case via your AWS account.

-Nate

MarcusNoble commented 5 years ago

This is great. Tried it out yesterday. I just wanted to point out a few gotchas I encountered to help others who are trying it out:

If anyone has wrote a kube2iam -> IAM for Service Accounts migration guide I'd be very keen to see it. 😄

micahhausler commented 5 years ago

@MarcusNoble kube2iam proxies the EC2 instance metadata, which comes after web identity credentials in the SDK credential chain. This should make for an easy migration, where IAM for Service Accounts will take precedence even if kube2iam is in place.

This approach cannot be used if you already have kube2iam running on the node, this could make migrating trickier.

I have not tried this myself, but why is this the case?

schlomo commented 5 years ago

please remember to restrict access to 169.254.169.254 from pods to protect the nodes instance IAM role.

MarcusNoble commented 5 years ago
dahu33 commented 5 years ago

Linking a few issues from different AWS/Kubernetes projects to more easily follow-up the support of this new feature: https://github.com/aws-samples/amazon-cloudwatch-container-insights/issues/5 https://github.com/kubernetes-incubator/external-dns/pull/1172 https://github.com/kubernetes/autoscaler/issues/2301 https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/1011 https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/361

groodt commented 5 years ago

Thanks so much for shipping this. It looks like it would have been a lot of work.

I've just implemented this on our end and can confirm that it's working. One thing I'd note is that it would be nice to mention oidc thumbprint. I was setting up the OIDC provider using Terraform and I had no idea how to find it until I found this document: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html

It could be exposed in the EKS AWS Console, or explicitly mentioned for every region.

groodt commented 5 years ago

Question, what is the token expiry mechanism? Are the tokens issued temporary credentials with a maximum lifetime of 12h? Does this mean that there can't be long-lived pods?

rverma-nikiai commented 5 years ago

SDK have the secrets rotation mechanism. The pods have to run supported SDK.

On Wed, 11 Sep, 2019, 5:42 PM Greg Roodt, notifications@github.com wrote:

Question, what is the token expiry mechanism? Are the tokens issued temporary credentials with a maximum lifetime of 12h? Does this mean that there can't be long-lived pods?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aws/containers-roadmap/issues/23?email_source=notifications&email_token=AJEMLIFK4KY4NY7ZKTDV3WTQJDOEVA5CNFSM4GIV7VRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6OIR5Q#issuecomment-530352374, or mute the thread https://github.com/notifications/unsubscribe-auth/AJEMLICFJC4OVA4TPMXWKDTQJDOEVANCNFSM4GIV7VRA .

micahhausler commented 5 years ago

It could be exposed in the EKS AWS Console, or explicitly mentioned for every region.

@groodt the OIDC thumbprint is an IAM requirement and not a feature of EKS. When setting up an IAM provider in the console, the root CA's thumbprint is automatically selected. This fingerprint is a hash of the serving Certificate Authority.

I opened an issue on the terraform TLS provider for this to be more easily automated.

In the mean time, you could use a local-exec that does something like this to get the root CA's thumbprint, write it to a file, and consume it with local_file

Question, what is the token expiry mechanism? Are the tokens issued temporary credentials with a maximum lifetime of 12h? Does this mean that there can't be long-lived pods?

The token lifetime is configured on the kube-apiserver using the flag --service-account-max-token-expiration and EKS sets this to 24 hours. The mounted token is automatically rotated by the kubelet at either 80% of the token's TTL or 24 hours. You can manually configure the expirationSeconds on a projected token volume to any value between 24 hours and ~5 minutes~ 10 minutes.

The AWS SDKs re-read the token from disk every time the retrieved AWS credentials expire.

groodt commented 5 years ago

@micahhausler Thanks for the response.

The mounted token is automatically rotated by the kubelet at either 80% of the token's TTL or 24 hours. You can manually configure the expirationSeconds on a projected token volume to any value between 24 hours and 5 minutes 10 minutes.

The AWS SDKs re-read the token from disk every time the retrieved AWS credentials expire.

Sounds great!

I opened an issue on the terraform TLS provider for this to be more easily automated.

Thanks! Yes, I worked out how to do it manually to put into a script, but a Terraform resource for it would be fantastic.

Thanks for you great work!

MarcusNoble commented 5 years ago

@micahhausler you mentioned that the thumbprint is automatically selected when using the console, what happens when using the CLI / SDK? Is there functionality there to have it automatically populated?

micahhausler commented 5 years ago

@MarcusNoble in the SDKs it’s up to the client to figure it out.

Here’s an example in Go of getting the root fingerprint

marcincuber commented 5 years ago

@micahhausler I have created OIDC provide for multiple EKS. I also obtained OIDC thumbprint separately for each provider. Am I correct by saying that OIDC thumbprint is always the same for EKS?

MarcusNoble commented 5 years ago

@marcincuber that is the same conclusion I came to also, and same in all regions. For the time being I have added the thumbprint as a hardcoded string in my terraform. Not sure how often / if this value changes.

marcincuber commented 5 years ago

@MarcusNoble I believe the root CA expires in like 2034 or something like that. I have hardcoded it for now as well. Thanks for confirming that you see the same behaviour.

MarcusNoble commented 5 years ago

Ha! Well... I'll set myself a reminder 😆

I guess it'd only be an issue if the root CA needs to be recalled for whatever reason. Though I have no idea how that would be handled even if you'd done it through the web UI.

dahu33 commented 5 years ago

One thing I'm finding extremly frustrating when using IAM Roles for Service Accounts is that the OIDC_PROVIDER has to be hardcorded in the cloudformation templates...

In the example policy below, the StringEquals condition requires the OIDC_PROVIDER in the key but AFAIK, cloudformation doesn't allow for parameter substitution in dictionary key...

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/OIDC_PROVIDER"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "OIDC_PROVIDER:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME"
        }
      }
    }
  ]
}

Is there any official (or workaround) solution to this issue?

kenske commented 5 years ago

@dahu33 use terraform instead, it will make your life easier in so many other ways.

savithruml commented 5 years ago

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/OIDC_PROVIDER" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "OIDC_PROVIDER:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME" } } } ] }

We ended up using Jinja2 substitution to build our templates

jqmichael commented 5 years ago

Has anyone tried using AWS CDK to write their cfn template? You will be writing Python or Golang code instead of json/yaml.

https://docs.aws.amazon.com/cdk/api/latest/docs/aws-iam-readme.html

On Tue, Sep 24, 2019 at 4:39 AM Pierre R notifications@github.com wrote:

One thing I'm finding extremly frustrating when using IAM Roles for Service Accounts is that the OIDC_PROVIDER has to be hardcorded in the cloudformation templates...

In the example policy below, the StringEquals condition requires the OIDC_PROVIDER in the key but AFAIK, cloudformation doesn't allow for parameter substitution in dictionary key...

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::AWS_ACCOUNT_ID:oidc-provider/OIDC_PROVIDER" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "OIDC_PROVIDER:sub": "system:serviceaccount:SERVICE_ACCOUNT_NAMESPACE:SERVICE_ACCOUNT_NAME" } } } ] }

Is there any official (or workaround) solution to this issue?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aws/containers-roadmap/issues/23?email_source=notifications&email_token=ADMYWQU5MLNNJONSP43LOFLQLH36LA5CNFSM4GIV7VRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7OBT6I#issuecomment-534518265, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMYWQXJZU5TEU7GBY6QNLDQLH36LANCNFSM4GIV7VRA .

max-rocket-internet commented 5 years ago

We ended up using Jinja2

🙁

Has anyone tried using AWS CDK to write their cfn template?

🙁

use terraform instead, it will make your life easier in so many other ways.

This x1000 🚀

savithruml commented 5 years ago

Has anybody tried to read the token as a non-root user? The path where the token is stored /var/run/secrets/eks.amazonaws.com/serviceaccount/token is owned by root, so not sure how this will work. Looking for suggestions.

fimbulvetr commented 5 years ago

Has anybody tried to read the token as a non-root user? The path where the token is stored /var/run/secrets/eks.amazonaws.com/serviceaccount/token is owned by root, so not sure how this will work. Looking for suggestions.

After reading the source code for k8s I found that you must have:

      securityContext:
        fsGroup: 1000 // should be the group id your process is running as in the container

on the container that will read the volume. If you don't have an fsGroup it will hardcoded to 0600. If you do have fsGroup spec'd, it will be hardcoded to 0660. This doesn't seem to be configurable.

There really should be a bug report for this as it seems like something that would come up often. If the process in your container runs as root, you wouldn't see this error but hopefully that's not happening often.

micahhausler commented 5 years ago

@savithruml @fimbulvetr You can follow along for that issue on aws/amazon-eks-pod-identity-webhook#8 and kubernetes/kubernetes#82573.

davidshin commented 5 years ago

I'm trying to get the AWS CNI Plugin to use IRSA using this guide (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-cni-walkthrough.html), but I'm having trouble getting the aws-node pods to assume role specified in the service account annotation.

The documentation here (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html) says that the minimum AWS Go SDK version is 1.23.13, but amazon-vpc-cni-k8s is using 1.21.7 in master (https://github.com/aws/amazon-vpc-cni-k8s/blob/master/go.mod).

The guide mentioned seems to suggest to use AWS CNI Plugin v1.5.3 ("If your CNI version is earlier than 1.5.3, use the following command to upgrade your CNI version to the latest version"...), but v1.5.3 is also using the AWS Go SDK v1.21.7, which is earlier than the minimum required version for IRSA support.

Is the guide wrong?

(UPDATE: turns out that my trust policy was configured incorrectly. Everything is working as expected using AWS CNI Plugin v1.5.4. But still unsure why the discrepancy between the minimum AWS GO SDK (v1.23.13) and the version used in AWS CNI Plugin v1.5.4 (GO SDK v1.21.7)