[EKS] [request]: Add ability to set oidc options

[jaxxstorm]

Tell us about your request Using IAM for authentication is not always the most desirable solution. I'd like to be able to specify server side configuration, specifically for oidc authentication

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? When interacting with EKS clusters from non Amazon resources, having to create and specify access keys and secret keys can be cumbersome. We do this a lot at $work and because we can't specify an oidc configuration for the API server, we end up having to build our own clusters. Being able to set the following API flags on the api server:

• oidc-issuer-url
• oidc-client-id
• oidc-ca-file
• oidc-username-claim
• oidc-groups-claim

Would make this possible.

Are you currently working around this issue? We are building our clusters using kubeadm :(

Additional context This could be argued that it should be made possible to specify any API server flags, but I'd like to keep the scope smaller now

[christopherhein]

Thanks for submitting this. This is not currently on our roadmap.

Can you help me understand more about you use case, would https://aws.amazon.com/blogs/opensource/integrating-ldap-ad-users-kubernetes-rbac-aws-iam-authenticator-project/ help to solve this?

It would give you the ability to use your AWS SSO to back your IAM roles which you could configure to use any LDAP/AD/SAML authentication mechanisms.

[jaxxstorm]

This is possible yes, but when using tooling like Azure Active Directory, it add an additional layer of configuration.

Currently, as a user using oidc, I can:

• Assign the user to a group in Azure AD
• Map that group to a Kubernetes RBAC role

However, under the model you're proposing, I'd have to:

• Assign a user to a group in Azure AD
• Map that user to an IAM role, presumably with role assumption
• Map the IAM role to a Kubernetes RBAC role

This additional layer adds complexity for debugging purposes. For example - if a user has a permission issue, I have to track the issue down by checking their AD group, then checking their IAM role, then RBAC. It seems unnecessary to me to not just add the API server flag.

[lstoll]

FWIW we ended up deploying an auth proxy to our clusters that authenticates users via OIDC, and then impersonates them to the API server. That allows us to manage access via our existing systems for users that may not have AWS access otherwise. The code is vaguely based on https://github.com/ericchiang/kube-oidc#kube-oidc-proxy

[christopherhein]

FWIW we ended up deploying an auth proxy to our clusters that authenticates users via OIDC, and then impersonates them to the API server. That allows us to manage access via our existing systems for users that may not have AWS access otherwise. The code is vaguely based on https://github.com/ericchiang/kube-oidc#kube-oidc-proxy

Sounds neat, is the code open source anywhere?

[jaxxstorm]

@lstoll that sounds ideal - I would love some help/pointers. I really do think this should be reconsidered by AWS, so upvotes would be awesome if people want this feature

[sermilrod]

I am also interested in this feature. The ability to configure identity providers is something I would like EKS to support. I need this because we want to map specific users to rbac rules so we have full and granular control over who has what access to kubernetes clusters without having to create an IAM user nor assume a role for it. This is particularly important in multi-tenant clusters.

[universam1]

FWIW we ended up deploying an auth proxy to our clusters that authenticates users via OIDC, and then impersonates them to the API server. That allows us to manage access via our existing systems for users that may not have AWS access otherwise.

We do the similar as well, but the OIDC integration in K8s is required still in order to map the JWT token into RBAC roles. For instance Kubernetes Dashboard needs that mapping to gain cluster level access. @christopherhein this missing support is a surprise no-go for us in the effort moving to EKS!

[christopherhein]

Thank you everyone for the feedback, really appreciate you talking about your use cases. Changing the label to under consideration 👍

[jaxxstorm]

@christopherhein would it be easier to get this considered if we filed an issue which allows us to specify API server configuration in general, rather than specifically for oidc flags?

[christopherhein]

@christopherhein would it be easier to get this considered if we filed an issue which allows us to specify API server configuration in general, rather than specifically for oidc flags?

These more specific issues really help us to prioritize what is most important. The larger "enable all API flags configurable" are much harder given there is so many controls that could introduce instability if improperly set if everything was customizable. Appreciate you putting this in and sharing the use cases.

[universam1]

To second that point of only requesting this very feature for now - while it is rather simple to configure the oidc configuration it is completely indispensable regarding alternatives! IAM auth can be considered an option for tooling access but application level that is no alternative. Is there an estimation possible @christopherhein ? Would be helpful for planning

[christopherhein]

Estimation for this feature of adding OIDC flag capability @universam1 ? or are you talking about the Pod->IAM configuration #23 ?

[pammi22]

@christopherhein is there any Estimation for this feature of adding OIDC flag capability?

[christopherhein]

@pammi22 not as of now, we removed the not-planned label in favor of the under consideration while we consider how we would enable this customization with the control plane.

[whereisaaron]

The lack of OIDC support for EKS is our single biggest issue for adoption. With stock k8s and dex we have a very clean, two-factor, federated single sign-on for clusters. These flags requested here would enable EKS clusters to have the same great user experience.

If EKS had built-in AWS Cognito support it is possible we could replace dex with Cognito for federating OIDC IDPs. But that support isn’t there currently AFAIK, and I can’t find any existing helper app that might bridge this gap in EKS. Anyone know of a project like this?

[aleks-mariusz]

I would like to echo all the sentiments..

We use AD for corporate identity management, and to access AWS console we let the infra/admin group (only) use Okta which loads the console using a dedicated IAM role that grants full access. It is unclear to me how setting up EKS with this assumed IAM role would affect having cluster-admin privs, but wanting to give end-users (sandbox'd) permissions to the kubernetes API server is very messy using EKS.

For contrast, what we currently have on-prem is AD -> federation via Keycloak -> which integrates with the k8s dashboard via the proxy -> k8s dashboard.. and this allows end-users to log in with their AD creds cleanly, which works by mapping via JWT's provided by keycloak (which contain their AD group membership), that membership can then be mapped directly via RBAC rolebindings, to what roles they have (which control what access they have).

It is such a clean and tight integration that it makes us want to consider skipping using EKS altogether and having a cloud-based kubernetes cluster by just managing the control plane ourselves. I have yet to see what using EKS gets you over just rolling your own control plane if we don't have access to provide consistency (auth-wise) with what we have on-prem. :-( Allowing us to add those 4 extra oidc related flags [ oidc-client-id / oidc-issuer-url / oidc-username-claim / oidc-groups-claim ] would have us giving EKS more serious consideration for use.

[Spazzy757]

Has there been an update to this issue? We are currently doing a POC for quite a large company to prove that they can move there on-prem clusters to EKS, however, they insist that they would need AD integration. Adding those flags could open up a much larger consumer base for EKS as it would allow migration from systems like AKS where the system is integrated into Azure Active Directory

[philwhln]

We had to cancel our PoC of EKS because of this limitation.

[ccojocar]

Any ETA for this feature? Thanks

[phanama]

Any updates? This limitation hesitates us from adopting EKS

[therealsamlin]

OIDC is the single reason why we're using kops still instead of EKS. Just saying

[ghost]

@therealsamlin take a look at https://github.com/jetstack/kube-oidc-proxy

[pvdvreede]

@yuriipolishchuk-finsight that seems like an option for the future when its production ready. The only limitation is that because it uses impersonation to work it means that users of the cluster cannot take advantage of impersonation. We use impersonation to elevate our read only privileges to cluster admin so that would be something we would loose with that project.

Nice find though! 👍

[keattang]

I created this auth proxy specifically for getting around this issue https://github.com/keattang/eks-auth-proxy/releases/new

[joshkurz]

Here is a minimal solution that allows you to use oidc and eks. https://github.com/ww-tech/aws-sts-proxy/blob/master/examples/client/usage blog post about this setup is here https://medium.com/ww-tech-blog/amazon-eks-authentication-with-google-openid-connect-oidc-via-a-proxy-api-service-ebbf7913295c

[vitaly-am]

is oidc options as eks api parameters in roapmap already there have any updates? it's really crucial and must-have option for EKS! we really need that for many of our EKS existing clusters

[olemarkus]

It would be very useful with an update here now. If I undestand https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ correctly, OIDC settings are now always set, but set to an AWS OIDC provider. And K8s does not support multiple providers https://github.com/kubernetes/kubernetes/issues/71162

[vitaly-am]

just my little update, as per many weeks of researching, now l able to run AD integration via kube-oidc-proxy. long story short: kube-oidc-proxy make you able to use oidc in EKS and now you can get 2 scenarios for AD auth in AWS ASK 1) gangway+dex+kube-oidc-proxy 2)kuberos+keycloak+kube-oidc-proxy

[omerfsen]

+1

[wolfencom]

Like many have said this was one of the biggest things that made us hesitate to use EKS here at Ticketmaster.

[posix4e]

At our job we've started to call k8s clusters which don't allow for this feature as tier 2 since we can't put any secure workloads on them.

[kdwodc]

+1 for us as well! PLEASE!

[dlippiatt]

+1 here as well.

[mo-saeed]

+1 here, it would be great to have it.

[pakoAku]

+1 would help us a lot

[vittoriocanilli]

+1 that would help us in our project too!

[gtaylor]

These +1 replies don't add to the conversation and generate quite a bit of spam. Could we use emojis on the original issue post instead?

[pierluigilenoci]

On the other hand, generating noise shows an interest in this feature which would be a killer feature compared to the competitors.

[pierluigilenoci]

Among other things, this feature would also solve several other problems with the integration of EKS with AWS SSO.

1) AWS SSO does not support API, Cloudformation or any kind of automation (it's impossible to automate the creation of Permission Set or Role) therefore in situations of complex granularity of access to the clusters it is hell to create/ manage/maintain. 2) AWS SSO doesn't even support nested groups in AAD. 3) Access to the dashboard with the token instead of with an OIDC proxy is much more complicated.

An external OIDC provider will solve all these issues.

[dlippiatt]

I think its worth the extra noise to try and get it implemented. Its big break on adoption for some of us.

[shafi-khan]

+1 for this feature.

[cmanzi]

We require user-level auditing at the API server level (not possible with aws-iam-authenticator or any other equivalent API server proxy afaik), which is part of the reason why we are using Kops right now over EKS. Interestingly, we are actually using AWS Cognito as an OIDC provider for our clusters, and it works quite well.

I'm curious as to why AWS hasn't offered native integration with Cognito User Pools - seems like a no-brainer. This feature would allow us to migrate our control planes to EKS in fairly short order.

[salamovz]

I created LDAP authentication to eks cluster using this instructions https://github.com/prydonius/dashboard-oidc-proxy

[dlippiatt]

I created LDAP authentication to eks cluster using this instructions https://github.com/prydonius/dashboard-oidc-proxy

@salamovz Did you get that to work on EKS? Did you have to make any changes?

[jlforester]

I've been looking at kubelogin which needs these two flags added to the apiserver

--oidc-issuer-url=ISSUER_URL
--oidc-client-id=YOUR_CLIENT_ID

[salamovz]

Yes it perfectly work on EKS. I changed gen-certs.sh to be: openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout server-key.pem -out server.pem -subj "/CN=kube-oidc-proxy.kube-oidc-proxy" -extensions SAN -config <(printf "[ req_distinguished_name ]\n[req]\ndistinguished_name = req_distinguished_name\n[SAN]\nsubjectAltName='DNS.1:kube-oidc-proxy.kube-oidc-proxy'") But you can use any other tool to generate certificate. In args inside of kubectl -n kube-oidc-proxy edit deploy oidc-proxy-dashboard I added: - --scopes=groups And you DONT need to uncomment --apiserver-host vi +125 dashboard/kubernetes-dashboard.yaml And when creating secrets please ensure you added '-n' to echo while creating base64 values like: echo -n "email" | base64 E.g kube-oidc-proxy/secrets.yaml

data:
  oidc.ca-pem: YW55RHVtbXlWYWx1ZUlmWW91VXNlVmFsaWRDZXJ0T25EZXg=
  oidc.issuer-url: eW91ckRFWG5hbWUudXJsLndpdGgudmFsaWQuY2VydA==
  oidc.username-claim: ZW1haWw=
  oidc.client-id: a3ViZXJuZXRlcw==
  oidc.client-secret: WlhoaGJYQnNaUzFoY0hBdGMyVmpjbVYw

[enj]

I have opened https://github.com/kubernetes/enhancements/pull/1689 as a provisional KEP to address this issue in a generic manner.

[abdellaui]

Really necessary feature for production environments for avoiding unsupported work arounds😔

[pierluigilenoci]

There is an open issue since one year for AWS SSO support on boto3. https://github.com/boto/boto3/issues/2091

[halcyondude]

bump. we're grappling with this presently. Any update from the EKS team?