EKS Feature Request: Control Plane to Associate Private Hosted Zone for Webhooks

[robisoh88]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request What do you want us to build?

The request is to allow an option for the control plane to use the cross account function to associate a private hosted zone in order to be able to resolve private hosted domain names.

Which service(s) is this request for? This could be EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

While using a MutatingWebhookConfiguration (or ValidatingWebhookConfiguration) that calls an off-cluster endpoint through a domain in a private hosted zone, the call fails as the master nodes cannot reach route53 private hosted zone. By default the Master nodes will reach out to the public resolver only as they are hosted and managed by AWS.

Without this function, off-cluster endpoints for MutatingWebhookConfiguration will need to be resolved through public DNS resolvers.

Are you currently working around this issue? How are you currently solving this problem?

• Avoid domains in private hosted zones
• Use IP address as the endpoint

Additional context Anything else we should know?

Error seen:

E1110 13:48:30.677965 1 replica_set.go:536] sync "default/web-xxxx.xxxx" failed with Internal error occurred: failed calling webhook "xxx.xxx.xxx": Post https://xxxx.xxxx:xxxx/xxxx/?timeout=10s: context deadline exceeded

MutatingWebhookConfiguration clientConfig snippet:

...
webhooks:
- name: xxx.xxx.xxx
  clientConfig:
    url: "https://xxxx.xxxx:xxxx/xxxx/"
    caBundle: ""
  sideEffects: None
  rules:

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[peterellisjones]

We are also hitting this issue in another way:

Control Plane DNS resolution to a private hosted zone is required when configuring an OIDC provider that is deployed in the VPC and uses DNS in a private hosted zone. Our use-case for this (which I expect is a common use-case) is using Dex as a "portal" OIDC provider running on kubernetes itself - as the LB proxying traffic to Dex is in a private network using private DNS, the control plane cannot resolve the OIDC provider issuer URL.

[jackivanov]

It's been almost 2 years and we would love to see this issue at least on the roadmap

[charmingnewt]

We're also seeing this issue for webhooks and would like to see it on the roadmap.

[pwn-sf]

We are facing the same issue. Is this the ask here:

1. Provide ability to add EKS vpc to private hosted zone?

My understanding is that private hosted zone in route53 takes a list of VPCs which have visibility into it. Or is it not that simple?

[AbeOwlu]

@robisoh88 if the webhook server is discoverable using private hosted zone;

• the control plane, after a successful dns query pointing to the off-cluster and off-network (cross-account) server endpoint, has to be able to connect to this server endpoint.

• the limitation, I think would be this network connectivity consideration. The PHZ assuming is in a peered VPC, or associated PHZ with the cluster-dataplane VPC, the control plane still fails to establish connection with the webhook server, the cluster-controlplane is still in an isolated VPC

• if the webhook server is reachable publicly, for the control plane, it defeats the purpose of the private hosted zone records

[pwn-sf]

I think it is more about resolution of private hosted domain names. EKS control plane can not see the entries is all.

For my usecase, it limits the ability of EKS control plane to call an internal nlb.

[akefirad]

Yep. Our use case is to deploy Istio in a primary-remote setup (there’s an eastwest LB that seems to have to be public 😔) Interestingly in my case, the LB host is resolvable but not reachable (because it’s “internal”). So I guess it’s more than just DNS resolution.

[pwn-sf]

in my usecase, DNS resolution was the only issue. Making DNS visible was enough (although the IPs it was resolving to were private, as they belonged to an Internal LB)

[gss2002]

@pwn-sf how did you make your dns visible to the control plane node for an internal lb that is resolvable via ec2 and containers running in eks?

E0214 13:12:58.904068 10 oidc.go:335] oidc authenticator: initializing plugin: Get "https://dex.eks.dbar.hdp.example.com/.well-known/openid-configuration": dial tcp: lookup dex.eks.dbar.hdp.example.com on 127.0.0.1:53: no such host

[gss2002]

@pwn-sf how did you make your dns visible to the control plane node for an internal lb that is resolvable via ec2 and containers running in eks?

E0214 13:12:58.904068 10 oidc.go:335] oidc authenticator: initializing plugin: Get "https://dex.eks.dbar.hdp.example.com/.well-known/openid-configuration": dial tcp: lookup dex.eks.dbar.hdp.example.com on 127.0.0.1:53: no such host

Based on the case I opened with AWS today they said to ask for the feature enhancement here for the APIServer to be able to utilize a VPC's defined Route 53 Resolver/Route 53 privatezone

[pwn-sf]

did you make your dns visible to the control plane node for an internal lb

I tried with the DNS entry in a public route53 hosted zone. This way you will have the dns entry as public but it will resolve to private IPs (which is ok apparently as kube-apiserver is then able to connect to the IP of the internal LB)