[Feature] Private Cluster: Simplify BYO VNet and DNS setup by decoupling node -> control plane communication from private DNS zone.

[TomGeske]

The AKS team uncovered issues with BYO VNet + DNS setup in AKS private clusters: https://docs.microsoft.com/en-us/azure/aks/private-clusters

In some cases, AKS private cluster creation fails for BYO VNet configuration when the DNS zone is not properly attached to BYO VNet or your Hub network. You may see messages like:

Agents are unable to resolve Kubernetes API server name. It's likely custom DNS server is not correctly configured, please see https://aka.ms/aks/private-cluster#hub-and-spoke-with-custom-dns for more information

Today worker node to control plane communication requires the private DNS zone for API server name resolution.

Going forward we are working on removing private DNS zone dependency for node to control plane communication.

Current ETA for public preview is in summer 2020.

In the meantime, you may want to consider one of the workarounds:

1. https://github.com/feiskyer/aks-private-cluster-demo
2. https://github.com/rhummelmose/private-aks-dns-zone-linker-function-app

Related issues

• 1508

[Musham-Aj]

When utilizing the "None" option for the private DNS zone, the cluster creates without failing using on-prem DNS in the VNET. However, after creation it seems like the portal no longer works to interact with the cluster.

Looking at network traffic, it looks like it's trying to find a DNS record with the format -priv.portal.hcp.centralus.azmk8s.io. That record doesn't seem to get created from what I can see when using the None option.

Is that expected behavior? If so, is that GUID available anywhere to lookup programmatically so that we can create a record on our internal DNS server?

cam we create a DNS Zone post the deployment of the AKS cluster? Would that work @feiskyer @miwithro @viralpat218 @anttipo

[feiskyer]

@Musham-Aj ARM template is also supported, but you couldn't set it post the deployment as it is required during cluster provisioning.

[Musham-Aj]

@Musham-Aj ARM template is also supported, but you couldn't set it post the deployment as it is required during cluster provisioning.

May I know, what value I need to pass here then ""privateDNSZone": "string"", Should I pass "none" or "resourceId" ?

[feiskyer]

resourceId should be put if you have already created the private dns zone.

[Musham-Aj]

resourceId should be put if you have already created the private dns zone.

I had passed resourceId in the "privateDNSZone" option like this, tried in below two ways, was still facing the same issue. "privateDNSZone": "/subscriptions//resourceGroups//providers/Microsoft.Network/privateDnsZones/" & "privateDNSZone": "/subscriptions//resourceGroups//providers/Microsoft.Network/privateDnsZones//virtualNetworkLinks/****"

Still facing the same issue. @feiskyer

[Musham-Aj]

@feiskyer @miwithro Could you please suggest me.

As per the above discussions, I am understanding that we need to go for "BYO private DNS Zone" solution to overcome the error "Agents are unable to resolve Kubernetes API server name. It's likely custom DNS server is not correctly configured.". and I had followed the below approach, However, Still I am facing the same issue.

I am using ARM Templates and I had passed resourceId in the "privateDNSZone" option like this, tried in below two ways, was still facing the same issue. "privateDNSZone": "/subscriptions//resourceGroups//providers/Microsoft.Network/privateDnsZones/" & "privateDNSZone": "/subscriptions//resourceGroups//providers/Microsoft.Network/privateDnsZones//virtualNetworkLinks/****"

[manvkaur]

Hi @feiskyer ,

We have created a private DNS zone linked to cluster VNet and hub Vnet - privatelink.westeurope.azmk8s.io Create a new user assigned managed identity having access on the private DNS zone

Using ARM template API version - 2021-02-01 passing the private DNS zone resource Id - /subscriptions/<subscriptionId>/resourceGroups/<resourcegroup>/providers/Microsoft.Network/privateDnsZones/privatelink.westeurope.azmk8s.io

We are still getting the error - Agents are unable to resolve Kubernetes API server name.

[feiskyer]

please file a support ticket as customer information is required for checking cluster issues.

[Musham-Aj]

Hi @feiskyer ,

We have created a private DNS zone linked to cluster VNet and hub Vnet - privatelink.westeurope.azmk8s.io Create a new user assigned managed identity having access on the private DNS zone

Using ARM template API version - 2021-02-01 passing the private DNS zone resource Id - /subscriptions/<subscriptionId>/resourceGroups/<resourcegroup>/providers/Microsoft.Network/privateDnsZones/privatelink.westeurope.azmk8s.io

We are still getting the error - Agents are unable to resolve Kubernetes API server name.

@manvkaur

Hi,

In the arm templates, under apiServerAccessProfiles put privateDNSZone as 'none',it doesn't throw any error in deployment and it wont create the 'dnszone'. so that post creation of the aks cluster, we have to take up the private ip address (NIC card) from the MC-RESOURCEGROUP and FQDN of aks(you will get it from AKS page) and it in hosts file '/etc/hosts' from which server you want to manage the cluster.

Done. This is the only option, I could see as of now through arm templates, i tried passing 'zoneid' in multiple ways but it doesn't work.

[feiskyer]

@Musham-Aj For the failed cluster, could you login to one of the nodes and verify DNS resolution (e.g. by nslookup command)? I'm wondering why the DNS resolution doesn't work with BYO dns zone. Or, if you prefer, open a support ticket so we could check from our side.

[manvkaur]

please file a support ticket as customer information is required for checking cluster issues.

Hi @feiskyer

I have a support ticket with Azure Premium Support Team already. Sharing the case Id - 2103120060003636

[GinSiuCheng]

Putting this sample here for people looking to deploy this via TF with hub-spoke requirements (where dns resolver lives in hub): https://github.com/GinSiuCheng/private-aks-deployment/blob/master/modules/azure_kubernetes_service/main.tf

[manvkaur]

@arobass custom defined DNS subdomain would be supported soon, e.g. [custom-subdomain].privatelink.[region].azmk8s.io. Do you think this works for you?

Hi @feiskyer , Is the custom defined DNS subdomain feature available today or any expected timeline for the same? We want to leverage this capability to have a different DNS Zone for each environment.

[GinSiuCheng]

@manvkaur the custom defined dns subdomain feature is in preview. However, you can not create a separate [custom-subdomain].privatelink.[region].azmk8s.io and point private AKS to it. Effectively the feature allows you to specify the subdomain A record in the privatelink.[region].azmk8s.io dns zone.

[manvkaur]

@manvkaur the custom defined dns subdomain feature is in preview. However, you can not create a separate [custom-subdomain].privatelink.[region].azmk8s.io and point private AKS to it. Effectively the feature allows you to specify the subdomain A record in the privatelink.[region].azmk8s.io dns zone.

@GinSiuCheng

The issue that I am facing presently is:

1. Private DNS Zone and DNS resolver are in hub subscription with an AD tenant - T1
2. AKS clusters in spoke subscription S1 which is in tenant T1
3. AKS clusters in spoke subscription S2 which is in tenant T2
4. Creating separate DNS Zones for both the subscriptions S1 and S2 will cause a conflict when both them get linked to hub subscription.
5. On the other hand, if we keep the Private DNS Zone in hub subscription, I am not clear how can I do the role assignment for an SPN from S2 on the DNS Zone which is in different tenant T1

[GinSiuCheng]

@manvkaur since there is no cross tenant guest SPN/MSIs this is not possible. The scenario/issue you mentioned applies for all private endpoints and is something PGs are working towards fixing. There are a couple of workarounds for this, but not ideal.

[manvkaur]

@manvkaur since there is no cross tenant guest SPN/MSIs this is not possible. The scenario/issue you mentioned applies for all private endpoints and is something PGs are working towards fixing. There are a couple of workarounds for this, but not ideal.

@GinSiuCheng , Any work around is never ideal : ) , What are the possible options/workarounds to move forward in this scenario?

[jabbera]

If you are looking for commentary on this feature I don't understand why I can't use the fqSubdomain option with a type of none. I had to allocate a private dns zone that isn't even used for anything since we have internal dns.

[GinSiuCheng]

@manvkaur in step 4 of your setup, don't link tenant 2's private dns zone to tenant 1's hub. Tenant 2's private dns zone should act as a dummy/staging zone that is not linked to anything, but all SPs should still write records to that zone. Then you create an automation process to copy tenant 2's zone records to tenant 1's private dns zone.

Until we have cross tenant guest SPNs or private dns zone prefixes/custom namespaces for private endpoint, this problem will persist.

[palma21]

@jabbera you can use none. It won't make the fqdn empty, that's required but you can manage it on your DNS if you want and we won't create a private DNS zone. --private-dns-zone none

[jabbera]

@palma21 ill try again but the arm template deployment failed saying it didn't work with none. (It was the first thing I tried)

[manvkaur]

@manvkaur in step 4 of your setup, don't link tenant 2's private dns zone to tenant 1's hub. Tenant 2's private dns zone should act as a dummy/staging zone that is not linked to anything, but all SPs should still write records to that zone. Then you create an automation process to copy tenant 2's zone records to tenant 1's private dns zone.

Until we have cross tenant guest SPNs or private dns zone prefixes/custom namespaces for private endpoint, this problem will persist.

@GinSiuCheng, I am facing problem while updating kubernetes version, privateDNSVNetLinkReconciler retry failed while using a dummy PDNS and having A records in hub PDNS. Other operations like cluster deployments, do work.

Do you have suggestions/workarounds on operations where we will need privateDNSVNetLinkReconciler ? Also, Is there any roadmap for private dns zone prefixes/custom namespaces for private endpoint feature?

[cavemandaveman]

Is it still not possible to attach a private DNS zone with custom domain to a private AKS cluster? What is the suggested workaround when needing multiple private AKS clusters in the same region?

[danielnachtrub]

Is it still not possible to attach a private DNS zone with custom domain to a private AKS cluster? What is the suggested workaround when needing multiple private AKS clusters in the same region?

Technically it's possible to use multiple DNS zones, it's just not possible to attach them to the same vnet. So you could either create different vnet's for all clusters or consider the DNS zone "network property" and treat the DNS zone as part of a shared vnet.

[cavemandaveman]

Technically it's possible to use multiple DNS zones, it's just not possible to attach them to the same vnet. So you could either create different vnet's for all clusters or consider the DNS zone "network property" and treat the DNS zone as part of a shared vnet.

My AKS clusters are already in different vnets and different resource groups. All of my resources in Azure are tied with a VPN gateway so that we can access them from on-prem. I need to be able access my AKS clusters too, ideally by their own unique URL. And it's not like I can have multiple entries in DNS for privatelink.<location>.azmk8s.io, not to mention how that is a different domain than my organization...

[danielnachtrub]

Depending on your onprem infrastructure you can just add DNS forwarding to azure for the specific hosts/zones (in this case on a host level as the zone exists multiple times). The only important thing is to share the private dns zone across your clusters, as it's not possible to connect the zones to the same subnet otherwise.

If all clusters and you're only accessing a single azure tenant are connected to the same dns zone you can just redirect privatelink.<location>.azmk8s.io.

[cavemandaveman]

I am not sure I follow. Are you saying to add DNS forwarding to the hosts in the AKS VMSS? Those can be autoscaling, it sounds like a nightmare trying to manually add delete records as hosts appear and disappear, especially when you have multiple clusters of hosts.

In any case, I don't think it will work in my situation, as each team with an AKS cluster manages their own private DNS zone in their own RG and/or subscription - there is no way to have one shared "master" private DNS zone for all AKS clusters.

[anttipo]

What I think @danielnachtrub was referring to, is that it's fairly straightforward to get the onprem DNS infrastructure to work in tandem with Azure DNS, if you are using a VPN/ExpressRoute, by just configuring a forwarder or conditional forwarding for the privatelink.<location>.azmk8s.io zone. That, as a solution, should scale and you don't need to tinker anything with the VMSS itself or any of the nodes.

We use a similar solution with multiple clusters and yes, even though the domain hostnames differ from our organization, these .azmk8s.io ones are only for internal use and not shown to public in any way or form. When we publish customer-facing services, we use different domains and hostnames for k8s services and ingresses.

[cavemandaveman]

This is what you are referring to, right? I understand how it works with a single cluster, but I am struggling to see how it will work with multiple. From what I understand, when you BYO DNS, the private DNS zone has to be named exactly privatelink.<location>.azmk8s.io. Meaning I can't have SpokeA with VNetA, AKSClusterA and private DNS zone AKSClusterA.privatelink.<location>.azmk8s.io and then have SpokeB with VNetB, AKSClusterB and private DNS zone AKSClusterB.privatelink.<location>.azmk8s.io.

[danielnachtrub]

Somehow yes.

Either you place a DNS forwarder in each vnet and configure conditional forwarding per kubernetes cluster hostname (onprem using conditional forwarding for the host records to the dns forwarder on azure). Or you create a shared dns zone and grant all kubernetes clusters contributor rights on this zone and use a single DNS forwarder on azure.

[cavemandaveman]

So you're saying private DNS zones in the format <custom-name>.privatelink.<location>.azmk8s.io work with private AKS clusters? Because the Azure docs seem to say otherwise.

[danielnachtrub]

No - you need to use privatelink.<location>.azmk8s.io. Despite this you can configure your redirection onprem on a level below (like k8scl1.privatelink..azmk8s.io) and redirect to a dns resolver on azure that is connected to the vnet where the DNS zone for k8scl1 is located (and therefore the host record exists).

[cavemandaveman]

Hmm. So your DNS resolver in Azure is set to something similiar to this?:

zone "azmk8s.io" {
    type forward;
    forwarders {
        168.63.129.16;
    };
};

I wonder how Azure's 168.63.129.16 knows how to resolve your k8scl1.privatelink.<location>.azmk8s.io correctly

[GinSiuCheng]

This is the arch. setup @danielnachtrub and @anttipo is referring to on sharing zones:

[cavemandaveman]

@GinSiuCheng this seems undesirable to me, since service names need to be unique between ALL of your AKS clusters. For example, you can't have 2 app1, one in each cluster. Also, how do you independently control your clusters with kubectl when privatelink.<location>.azmk8s.io refers to both?

If @danielnachtrub setup works, it seems more flexible - from on-prem you can forward <clusterA>.privatelink.<location>.azmk8s.io to a DNS resolver in VNetA, forward <clusterB>.privatelink.<location>.azmk8s.io to a DNS resolver in VNetB. This way you can independently control each cluster since it has a specific URL, and you won't need to worry about having unique naming of services between all your clusters.

[GinSiuCheng]

Assuming 2 apps are called app1, the A record auto-inserted by AKS is in the format of app1-[random-guid] and when you query for specific AKS instance, it would be app1-[randomguid].privatelink.[region].azmk8s.io.

[kelly-brown]

I've got DNS on prem working for the privatelink..azmk8s.io domain, but the portal seems to have stopped using that DNS name for the 'kubernetes resources' section of the portal. It's using -priv.portal.hcp..azmk8s.io. I see no reference to this domain in the documentation and support seems to be at a loss at explaining what it is. If I resolve it from within Azure, it's resolving to the same IP as my privatelink..azmk8s.io name. Is this relevant to this issue or should I log it somewhere else? Thanks.

[feiskyer]

@kelly-brown the FQDN used in Portal's kubernetes resources is a public FQDN, so you don't need to setup private DNS record for it. It would resolve to the same PrivateEndpoint IP, hence the Portal needs to be run within a VNet that could reach to the PrivateEndpoint IP.

[kelly-brown]

@feiskyer thank you for the information. So do I just need to get my DNS admins on prem to forward this domain to Azure? There's nothing in the docs about it so I'm a little afraid to make changes without full understanding. I believe that this used to use the privatelink.[region].azmk8s.io address. What's the reason for the change?

[feiskyer]

@kelly-brown No, you don't need to setup any additional DNS records (and private dns zone) as the public FQDN would be resolvable on the internet.

[SatyKrish]

This is what you are referring to, right? I understand how it works with a single cluster, but I am struggling to see how it will work with multiple. From what I understand, when you BYO DNS, the private DNS zone has to be named exactly privatelink.<location>.azmk8s.io. Meaning I can't have SpokeA with VNetA, AKSClusterA and private DNS zone AKSClusterA.privatelink.<location>.azmk8s.io and then have SpokeB with VNetB, AKSClusterB and private DNS zone AKSClusterB.privatelink.<location>.azmk8s.io.

Hi @miwithro, I was referring to this limitation of having to create private DNS zone with exact privatelink.<location>.azmk8s.io name. This causes conflict when DNS zone is created on each Spoke VNet.

[thpang]

Any updates on this one. Seems to have been sitting idle for some time now with no clear solution. Running into this issue when using terraform to stand up the cluster with private_cluster_enabled where var.aks_private_cluster is true. Also have these items being passed to the azurerm_kubernetes_cluster resource:

  # https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions
  # az aks get-versions --location eastus -o table
  kubernetes_version              = var.kubernetes_version
  api_server_authorized_ip_ranges = var.aks_private_cluster ? [] : var.aks_cluster_endpoint_public_access_cidrs
  private_cluster_enabled         = var.aks_private_cluster ? true : false
  private_dns_zone_id             = var.aks_private_cluster ? "System" : null

[ghost]

Thank you for the feature request. I'm closing this issue as this feature has shipped and it hasn't had activity for 7 days.