[BUG] AKS APIServer public IP designated as External Public IP by Azure networking

[zmalik]

Describe the bug First of all, I'm not sure how helpful it will be to open this issue in this repository or whether it should be handled by Azure Networking. However, I believe all AKS customers could be affected by this, so perhaps the AKS team can drive this potential issue to a resolution or explain this behavior.

We are observing that in our Azure AKS cluster, the AKS API Server Public IP is being designated as ExternalPublic by Azure Networking. This means that if a NAT gateway is attached to the subnet where AKS nodes are running, all communication between nodes and the AKS API Server is charged as if the control plane is running on AWS, GCP, or any other external infrastructure.

Practically, this means that as we add more load (nodes/pods/operators) to our AKS cluster or use API-intensive features like Azure NPM, the cost of operating a simple AKS cluster can become unmanageable. The communication between the cluster and the control plane is charged at $0.045 per GB.

In the default AKS setup, an AKS mutation webhook also injects the public FQDN in the kube-system namespace as environment variables to all pods. Consequently, all namespaces, except kube-system, use the in-cluster private IP address. This makes controlling this traffic cost challenging for users, as Azure does not provide much flexibility to reduce this traffic. Given that only users with significant load in the AKS cluster will attach a NAT gateway (as the default egress is insufficient under heavy load), this significantly increases the chances of overcharging. In our environment, a single node is receiving between 6GiB to 13GiB of traffic every hour from the API server. If a cluster has 250 nodes, this translates to a substantial amount of traffic and associated costs.

• $1620 daily NAT gateway charges at 6Gi for 250nodes
• $3510 daily NAT gateway charges at 13Gi for 250 nodes

This amount can significantly get out of the hands if you have Azure NPM on a big cluster or you are scaling nodes and pods to fit in 1000 nodes in AKS Cluster. And in case you have 50s of AKS clusters this all sums up.

bare in mind this is all just kube-system and systemd units(node-problem-detector/kubelet) communicating to the APIServer IP. As rest of operators are just using kubernetes.default.svc.cluster.local which resolves to internal IP address.

To Reproduce

• create an AKS cluster with default settings
• add pods and nodes which have no ingress or egress, busybox with sleep is enough. This step is important to observe NAT gateway metrics change in later step.
• attach Azure NAT gateway to the subnet of the AKS cluster
• (optional) enable NSG flow logs to observe traffic between AKS nodes and AKS APIServer
• add a route for AKS APIServer to use next hop an Azure firewall to avoid NAT Gateway
• observe how in metrics of NAT gateway there is a drastic decrease. As almost all traffic was just nodes talking to APIServer.

Here are few diagrams on the current behavior we see.

once we divert the traffic through some other route

we see a direct reduction in cost of NAT Gateway and also the metrics we see significant decrease.

Expected behavior Just like storage accounts or managed databases, when using public IP in same region, traffic is not going through the NAT Gateway, as its designated AzurePublic

We have tested this behavior and without private link, uploading a file of 100s of Gi, doesn't go through NAT gateway. And if we upload a file to a storage account from another region, which is also designated as AzurePublic in flow logs. That traffic goes through the NAT Gateway, which is fully understandable as we are dealing with inter-region traffic.

Im aware of the private AKS cluster setup, but feels like that it's not fair to not to list this additional charge in the default AKS cluster setup.

Environment (please complete the following information):

• Kubernetes version [1.27.9]

Additional context Some additional behavior we have seen in the classification of the AzurePublic or ExternalPublic

We do see that APIServer public IP of another cluster in same region is identified as AzurePublic and we are not charged for that traffic. So the nodes talking to their own APIServer are charged extra, but get free traffic to another AKS APIServer running in same Azure region for free.

There is an Azure Support ticket on-going for several weeks and first impression is that this is desired behavior. I would like to hear AKS engineering opinion on it.

[eyltl]

You sure about the nat cost & azure storage?

https://learn.microsoft.com/en-us/answers/questions/1084416/same-region-rounting-to-storage-account-from-subne

[zmalik]

You sure about the nat cost & azure storage?

yes, very sure. As in we tested it.

please take a look at recent link from May of this year: https://learn.microsoft.com/en-us/answers/questions/1663216/traffic-path-between-azure-storage-account-and-azu

here they mention:

Azure VM to Azure Storage in the Same Region: If private endpoints are not used, the Azure VM will connect to the storage account's public endpoint. However, even though it's a public endpoint, the traffic between the Azure VM and the storage account, when both are in the same region, typically remains within Microsoft's Azure internal network, not traversing the public internet. This setup leverages the Azure network, optimizing for security and performance within the same regional infrastructure.

[kamilzzz]

Storage is special kind of service where traffic path may indeed be different as far as I know.

For network flows targeting public IPs in the same region traffic is definitely traversing NAT Gateway as one use case for NAT Gateway is to avoid SNAT exhaustion. And we were using that to avoid SNAT exhaustion when targeting managed databases.

[zmalik]

some updates:

• AKS control-plane and node communication occurs over the internet unless using a private AKS cluster or preview features with apiserver pods in the user VNET.
• NSG flow logs proved unhelpful in investigating NAT Gateway traffic flows. We couldn't detect operators communicating with the APIServer via private address, even though traffic ultimately passes through the NAT Gateway.
• In June, Azure released a fix for NAT Gateway metrics, affecting costs. This primarily impacted AKS control-plane to node communication in our case. While not applicable to all users, it affects control-plane and node traffic scenarios.

This aligns with our observed NAT Gateway cost increases. AKS users globally following best practices with managed NAT Gateway may have experienced unexpected cost increases after this change.

Investigation into high traffic volume between AKS control-plane and nodes continues

[kevinnowland]

@zmalik Do you have documentation describing the fix for NAT Gateway Metrics? Thank you

[zmalik]

@kevinnowland unfortunately not. I am not aware of any public documentation for that fix.

[zmalik]

so most of the traffic is coming from the operators. Operators such as istio, kube-state-metrics, argoCD or vertical-pod-autoscaler takes the lead. Any managed add-on such ama-metrics is also present. This is understandable knowing all of these operators watch quite significant amount of objects.

This also matches with the traffic pattern where APIServer to Nodes inbound traffic is 10x more than outbound traffic. This was verified by coroot also. I did a second check by writing a small node agent which does the following:

• Runs a tcpdump where the source is the APIServer IP address for 20 seconds every 5 minutes, over a few hours
• Analyzes the framelen field and destination port (source port is always 443 and IPs are always fixed)
• In parallel, finds TCP entries in /proc, converts hexadecimal IP addresses to dotted-decimal format, and saves those connections in an in-memory map (including the PID)
• Maps the local port and remote port to the tcpdump data, as remote IP and local IP are always the same
• Fetches the /proc/pid/cgroup to get the container from there
• Determines if it is a systemd or pod, gets the container metadata using crictl, otherwise groups the systemd as a system process
• Emits metrics on bytes received per second for the namespace/container (or system process)

I did this clunky manual verification step to fully discard any errors in our analysis. I honestly feel that all of this traffic by default should not be charged by Azure. As this is coming from kubernetes watch pattern.