[BUG] AKS pushes obsolete tags to resources it manipulates

[rickardp]

Describe the bug Possibly related to https://github.com/Azure/AKS/issues/1200#issuecomment-1355189645 and https://github.com/Azure/AKS/issues/3459

I am adjusting tags on resources to comply with changes to our corporate tagging policy. I noticed that some resources, like the routing tables, pops back to being noncompliant. After looking into the activity logs, I found that it is the AKS cluster managed identity that overwrites them with the old tags that were set when the AKS cluster was created.

I cannot reliably get the subscription to be compliant now, as the tags keeps resetting to their old (noncompliant) values. For obvious reasons, recreating the cluster is not an option.

To Reproduce Steps to reproduce the behavior:

1. Create AKS cluster using kubenet with tags foo=bar
2. Observe routing tables updated with tags foo=bar
3. Change tags on all resources to foo=bar2
4. Observe routing tables get their tags foo=bar written back, even if all other resources have foo=bar2. The value foo=bar is cached somewhere.

Expected behavior I expect the tags written to always reflect the current tags on the resource, so tags can be changed.

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[rickardp]

As a workaround, I can set tags directly on the node pool using Terraform, copying them myself from the AKS resource. I am not sure how this plays with policy-set tags though, as they were already set when I applied the workaround.

[palma21]

As a workaround, I can set tags directly on the node pool using Terraform, copying them myself from the AKS resource. I am not sure how this plays with policy-set tags though, as they were already set when I applied the workaround.

this is not allowed and causes the cluster to become unsupported though.

We're looking into the OP issue. Have you opened a support ticket by any chance so we can look into the cluster?

[rickardp]

As a workaround, I can set tags directly on the node pool using Terraform, copying them myself from the AKS resource. I am not sure how this plays with policy-set tags though, as they were already set when I applied the workaround.

this is not allowed and causes the cluster to become unsupported though.

We're looking into the OP issue. Have you opened a support ticket by any chance so we can look into the cluster?

Can you please elaborate? It's a documented feature in azurerm https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster_node_pool

Are you suggesting this can break the cluster somehow?

I did not create a support ticket as we had to fix this urgently. Would creating a support ticket help fixing the original issue?

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads

[microsoft-github-policy-service[bot]]

Issue needing attention of @Azure/aks-leads