API Server VNET Integration

[miwithro]

GA ETA: Q1 2025*

*ETAs are only estimates and can be subject to change

Both AKS Public and Private Cluster use a Tunnel, also a Private Cluster is currently based on Private Link. API Server VNET Integration will be enabled for both Public and Private clusters, and will allow for upgrade capabilities from Public to Private and Private to Public if required, without a redeploy.

[ghost]

@Azure/aks-pm issue needs labels

[rouke-broersma]

@miwithro could you add some documentation on what SWIFT is? We use private link for private clusters with user created private endpoints so we can connect to the cluster API from different tenants so it sounds like this could have a large impact on our workflows.

[hbuckle]

Is it the same Swift Virtual Network Connection that app service uses? We haven't found that to be particularly reliable.

[carvido1]

@miwithro could you add some documentation on what SWIFT is? We use private link for private clusters with user created private endpoints so we can connect to the cluster API from different tenants so it sounds like this could have a large impact on our workflows.

same as @miwithro , it would be nice to have clarifications from microsoft, as we are working on recreating AKS clusters using AKS private clusters. Maybe this is related ? https://www.swift.com/our-solutions/interfaces-and-integration/public-cloud

[ghost]

Action required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[miwithro]

@rouke-broersma @hbuckle @carvido1 API Server VNET Integration allows us from a topology perspective to get rid of the tunnel-front, openvpn, konnectivity pods for access. So it is a more secure by default network topology that will NOT ONLY apply to Private AKS but also Public AKS.

[rouke-broersma]

@miwithro will vnet integration replace private link or will it be an alternative to it? We currently use private link with aks to enable connectivity from non-peered vnets (multiple private endpoint connections to the same aks). Would we be able to continue using this or will we need to create an alternative approach?

[miwithro]

No, vnet integration will replace Private Link between the AKS Nodes and the Control Plane, your existing connectivity from non-peered vnets will continue as is.

[hbuckle]

So to confirm, we'll be able to create multiple vnet integrations in different vnets? Will there be a requirement for dedicated subnets like with app service? Will we still need to create private DNS zones like with private link?

[rouke-broersma]

@hbuckle how I understand it is that only the control plane will use vnet integration for the communication between the control plane and the aks nodes. Communication from clients to the control plane will still use private link. Is that correct @miwithro ?

[miwithro]

@rouke-broersma that is correct.

[phealy]

This feature was released to public preview on 2022 June 27. Documentation is available in the standard AKS documentation site.

@hbuckle one cluster will only be VNet integrated into a single VNet. You'll be able to create private link services on the kube-apiserver ILB to expose it across virtual networks, though that functionality doesn't yet work at public preview. Yes, the API server subnet must be delegated to Microsoft.ContainerService/managedClusters, but you can use it for multiple clusters as long as it's big enough. Private DNS zones will still be created by default, but the nodes and cluster workloads reach the API VIP directly by IP (since it's in a delegated subnet, the IP can't be changed, so we don't need to use DNS). You can disable provisioning of the private DNS zone and just use the public FQDN record we create by following the regular private cluster DNS setup steps.

@rouke-broersma VNet integration will be used for all communication between your private network, including nodes and out-of-cluster kubernetes clients, and the API server. Private link will only be used if you need to expose the cluster to a separate VNet without peering - e.g. you deploy AKS in an "island" unpeered virtual network, then you create a PLS pointing at kube-apiserver and create a PE to expose the API server endpoint into your main hub/spoke for connectivity. That functionality is planned but PE/PLS support is not available as of initial public preview. Importantly, it will be self-managed - AKS with API server VNet integration won't create the PLS for you, so you have complete control over how it's provisioned.

[rouke-broersma]

@phealy so if I understand you correctly the previous information provided by @miwithro was incomplete?

I had asked before if vnet integration would replace the current private link setup and was told that it would. I assume that my question was misunderstood as meaning if you enable it, it will replace private link. What I meant was, will vnet integration be an option for private clusters (ie private link or vnet integration), or will we be forced to migrate to vnet integration for private clusters?

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[kevinharing]

With this change, will it be possible to whitelist the API server subnet on a storage account firewall, so that the csi-azurefile-controller can communicate through the data plane instead of the control plane of the storage account (for creating/deleting file shares)? Or are the managed controllers hosted elsewhere?

See this issue for reference.

[PixelRobots]

@phealy Any update on when new regions will be supported in the preview?

[miwithro]

I just updated the document yesterday. We now support ALL Public regions minus "southcentralus"

[cata008]

@phealy @miwithro Any rough idea when this becomes GA?

[mr-twendt]

@miwithro The docs currently do not state which ports are used from the api server to the nodes. We are using BYO VNET and there are NSGs connected to each subnet. Therefore we need to allow the traffic from the api server to the worker nodes.

[paulgmiller]

What ports are used are actually work load dependent. web hooks, kubectl port-forward , proxy can all use different ports. kubectl exec, logs probably have standard ports on kublet.

[zhiweiv]

Just want to confirm again, Any ETA of GA?

[PixelRobots]

@miwithro or @phealy Do you have an estimate for GA?

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[cwe1ss]

comment to keep it active. any update about the ETA for GA would still be great.

[levimm]

We don't have exact GA date yet. For those who are deciding whether to create tradition private cluster or wait for apiserver vnet integration GA, they can just go and create tradition private cluster for now. We would support upgrade from tradition cluster (no matter private or not) to apiserver vnet integration.

[phealy]

@rouke-broersma @hbuckle @carvido1 API Server VNET Integration allows us from a topology perspective to get rid of the tunnel-front, openvpn, konnectivity pods for access.

Note for those reading this thread: this was correct at the time of writing, but Konnectivity will still be deployed for API Server VNet Integration clusters that utilize Azure CNI Overlay or BYOCNI (new features since that comment was written). In these scenarios communication between the API server and the nodes will still be direct, but the API server can't reach the pods directly because they're in an overlay network. Konnectivity will be used for traffic from the API server to destinations in the PodCIDR - typically for connectivity to metrics-server and for supporting kubectl port-forward, but it will also be used if any webhook configurations are created that are hosted in-cluster.

[rtsisyk]

We found a bug with disappearing kube-apiserver https://feedback.azure.com/d365community/idea/b5eb7a15-bd2b-ee11-a81c-000d3a040137

[kevinharing]

Any update on when this will be GA? It’s been a while without any updates.

[PixelRobots]

Hey @chasewilson any update on this going GA?

[chasewilson]

Unfortunately no updates yet. We're still working on the feature but have run into some dependencies we are working to overcome.

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[rtsisyk]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

This issue is actual, please don't close it.

[PixelRobots]

Hey @chasewilson any further updates on this issue?

[erezbatish-ms]

Any updates on GA date?

[kevinharing]

Really want to use this in production, any update on when it will become GA?

@miwithro @chasewilson

[rafallopatka]

Any updates on GA date?

[miwithro]

We are trying to GA in Q4 2024. We are working with closely with Azure Networking to ensure the dependencies across Azure regions are in place to support API Server VNET integration. As we get closer to the date, we will update the github link for everyone to be aware.

[microsoft-github-policy-service[bot]]

@chasewilson would you be able to assist?

[microsoft-github-policy-service[bot]]

@chasewilson would you be able to assist?

[microsoft-github-policy-service[bot]]

@miwithrow, @CocoWang-wql would you be able to assist?

[microsoft-github-policy-service[bot]]

@miwithrow, @CocoWang-wql would you be able to assist?

[simon-harvey-evelyn]

@miwithro is this still planned for Q4 2024?

[chasewilson]

@miwithro is this still planned for Q4 2024?

This is still planned for this year and we're closely watching this one with high priority.

[illrill]

Need guidance on how large the subnet should be. The docs say we need minimum /28 to function, but what do we need to scale? For example, some recommendations for clusters of 500 / 1000 / 2000 / 5000 nodes?