[EKS] [request]: Support for OIDC endpoint with PrivateLink EKS VPC Endpoint

[heggenu]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request What do you want us to build? Ability to use the EKS VPC Endpoint together with OIDC

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? We want to provide centralized VPC Endpoints for commonly used services, when the EKS endpoint came out, we wanted to offer private connectivity to this endpoint to avoid going out to the internet.

When we deployed the central VPC Endpoint for EKS com.amazonaws.${region}.eks this causes the DNS lookup for oidc.eks.${region}.amazonaws.com. to fail as OIDC is not supported by the VPC Endpoint.

Are you currently working around this issue? How are you currently solving this problem? Use the public Endpoint. Toggle Private DNS names enabled to false for the EKS VPC endpoint. Alternatively, removing the EKS VPC Endpoint if not in use at all.

Additional context Anything else we should know?

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[SaranBalaji90]

Thank you filing the report. Can you also explain the use case for connecting to oidc.eks.${region}.amazonaws.com endpoint from the worker nodes?

Generally, when pods requests IAM credentials using the OIDC JWT vended by Kubernetes cluster, STS internally reaches out to Cluster's OIDC endpoint to validate the token. This request to OIDC endpoint is handled between the AWS services and shouldn't be originating from worker VPC.

Update: https://github.com/weaveworks/eksctl/blob/main/pkg/iam/oidc/api.go#L156 (looks like eksctl is querying the enspoint to get the CA thumbprint). Called out in their notes section https://eksctl.io/usage/eks-private-cluster/

one possible workaround at the moment for customers facing the issue with eksctl - running the following sample command using IAM VPCE can unblock you in the meantime - aws iam create-open-id-connect-provider --url <value_from_describe_cluster_api> --client-id-list sts.amazonaws.com --thumbprint-list 9e99a48a9960b14926bb7f3b02e22da2b0ab7280 to create the provider. [To obtain thumbprint https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html]

[hhsnow]

Similar to eksctl usecase, the Terraform aws_eks_cluster resource documentation recommends retrieving the CA thumbprint via this endpoint which is not accessible within the VPC if the EKS endpoint is enabled.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster.html#enabling-iam-roles-for-service-accounts

[plenti-chris-fowles]

This is really a frustrating limitation when using terraform and atlantis from within an AWS VPC.

[cydergoth]

Just spend a two days trying to figure out why Terraform being run in an AWS Cluster with Harness IaC Agent was failing with this error .... Thanks @heggenu for filing this ticket!

[hsmade]

This breaks CAPI as well. When running a management cluster for CAPI in a VPC, and it tries to create an EKS cluster with OIDC enabled.

[punkwalker]

🌟🌟🌟Workaround for partially Air-Gapped VPCs🌟🌟🌟

1. Create Route 53 outbound endpoint in VPC.

2. Create Route53 Resolver Rule and associated it under above Outbound endpoint and associate with the cluster VPC. Use following values for it:

   Rule name: <Some_Name>
   Rule type: Forward
   VPCs that use this rule: Cluster VPC
   Domain name: oidc.eks.<region>.amazonaws.com
   Outbound endpoint: <Endpoint from step 1>
   Target IP addresses: <Public Nameserver> (e.g 8.8.8.8) 
   Note: The Security Group/ NACL of Outbound endpoint (Step 1). Should allow egress traffic to target IP

3. Create EKS VPC Endpoint

With above configuration, *.eks.<region>.amazonaws.com will resolve to Private VPC IPs except oidc.eks.<region>.amazonaws.com will be resolved by Public Nameserver (Step 2) to Public IP address of OIDC.

[seansummers]

Thanks for the solution description, @punkwalker !

From the cluster (API) one can apply a similar fix, using the coredns configMap (assuming there is a NAT route to 8.8.8.8 from the control nodes:

-  forward . /etc/resolv.conf
+  forward . /etc/resolv.conf {
+    except oidc.eks.us-east-1.amazonaws.com
+  }
+  forward oidc.eks.us-east-1.amazonaws.com 8.8.8.8

[mikestef9]

We have partially solved at least one of the reasons why this request was opened in the first place

When we deployed the central VPC Endpoint for EKS com.amazonaws.${region}.eks this causes the DNS lookup for oidc.eks.${region}.amazonaws.com. to fail as OIDC is not supported by the VPC Endpoint.

EKS has introduced a new dual stack compatible endpoint for OIDC issuer url, with the following format

oidc-eks.region.api.aws

This is not currently returned as part of describe cluster. We are still debating whether to update the existing field or introduce a new field in the DescribeCluster response.

But for now, you can manually construct this URL based the existing value of https://docs.aws.amazon.com/eks/latest/APIReference/API_OIDC.html#AmazonEKS-Type-OIDC-issuer

For example if your cluster has a value of

https://oidc.eks.us-west-2.amazonaws.com/id/3187F794DDC03F340D6E663ECE675D9D

you can instead use

https://oidc-eks.us-west-2.api.aws/id/3187F794DDC03F340D6E663ECE675D9D

If you enable PrivateLink for the EKS management endpoint, this OIDC url will be accessible, but it's important to note, this URL is still only accessible publicly, not over PrivateLink. But it does solve the original reason the issue being opened, of the OIDC endpoint not being reachable at all when EKS PrivateLink was enabled.

This is currently available in commercial and GovCloud regions.

[erdeiimre]

trying from my air-gapped cluster to reach the oidc:

curl -v https://oidc-eks.eu-central-1.api.aws/id/... trys on public ip:

Trying 3.126.180.18... despite i have an endpoint for the eks endpoint and seems working (resolves it to private interface):

 curl -v https://eks.eu-central-1.api.aws
* Rebuilt URL to: https://eks.eu-central-1.api.aws/
*   Trying 172.20.74.39...

anything im missing?

[mikestef9]

Reopening the issue as I provided incorrect guidance before, please see the updated comment https://github.com/aws/containers-roadmap/issues/2038#issuecomment-2278450601