search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.61k stars 4.65k forks source link

Unable to upgrade AKS clusters: NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged #23436

Closed cippaciong closed 1 year ago

cippaciong commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

v1.5.7

AzureRM Provider Version

v3.75.0

Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Terraform Configuration Files

terraform {
  required_version = "~> 1.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.59"
    }
  }
}

data "azurerm_resource_group" "resource_group" {
  name = var.resource_group_name
}

locals {
  cname = lower(data.azurerm_resource_group.resource_group.name)
}

resource "azurerm_kubernetes_cluster" "k8s" {
  resource_group_name               = data.azurerm_resource_group.resource_group.name
  name                              = data.azurerm_resource_group.resource_group.name
  dns_prefix                        = local.cname
  location                          = data.azurerm_resource_group.resource_group.location
  kubernetes_version                = var.k8s_version
  sku_tier                          = var.sku_tier
  role_based_access_control_enabled = true

  default_node_pool {
    name                        = "default"
    temporary_name_for_rotation = "defaulttmp"
    orchestrator_version        = var.node_pool_orchestrator_version
    min_count                   = var.k8s_min_agent_count
    max_count                   = var.k8s_max_agent_count
    enable_auto_scaling         = true
    vm_size                     = var.k8s_agent_size
    os_disk_size_gb             = 50
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    terraform = true
  }
}

Debug Output/Panic Output

https://gist.github.com/cippaciong/a603d3641b2945658032a7a9a1ba8d08

GET /subscriptions/00000000-00000000-00000000-00000000/resourceGroups/REDACTED-NAME/providers/Microsoft.ContainerService/managedClusters/REDACTED-NAME?api-version=2023-04-02-preview HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.20.5 (amd64-linux) go-autorest/v14.2.1 hashicorp/go-azure-sdk/managedclusters/2023-04-02-preview HashiCorp Terraform/1.5.7 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.75.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 003465ec-c0fd-c8da-3663-8699b00493c8
Accept-Encoding: gzip: timestamp=2023-10-03T12:29:36.138+0200
2023-10-03T12:29:36.653+0200 [DEBUG] provider.terraform-provider-azurerm_v3.75.0_x5: AzureRM Response for https://management.azure.com/subscriptions/00000000-00000000-00000000-00000000/resourceGroups/REDACTED-NAME/providers/Microsoft.ContainerService/managedClusters/REDACTED-NAME?api-version=2023-04-02-preview: 
HTTP/2.0 200 OK
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 03 Oct 2023 10:29:35 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 003465ec-c0fd-c8da-3663-8699b00493c8
X-Ms-Ratelimit-Remaining-Subscription-Reads: 11986
X-Ms-Request-Id: 10592c37-6579-4764-8f4e-d6cb9bd080c4
X-Ms-Routing-Request-Id: ITALYNORTH:20231003T102936Z:954740aa-7a5c-4765-ae47-1c6ecbc05258

{
  "id": "/subscriptions/00000000-00000000-00000000-00000000/resourcegroups/REDACTED-NAME/providers/Microsoft.ContainerService/managedClusters/REDACTED-NAME",
  "location": "westeurope",
  "name": "REDACTED-NAME",
  "tags": {
   "terraform": "true"
  },
  "type": "Microsoft.ContainerService/ManagedClusters",
  "properties": {
   "provisioningState": "Succeeded",
   "powerState": {
    "code": "Running"
   },
   "kubernetesVersion": "1.24",
   "currentKubernetesVersion": "1.24.10",
   "dnsPrefix": "REDACTED-DNS",
   "fqdn": "REDACTED-DNS-308f7803.hcp.westeurope.azmk8s.io",
   "azurePortalFQDN": "REDACTED-DNS-308f7803.portal.hcp.westeurope.azmk8s.io",
   "agentPoolProfiles": [
    {
     "name": "default",
     "count": 3,
     "vmSize": "Standard_D4as_v5",
     "osDiskSizeGB": 50,
     "osDiskType": "Managed",
     "kubeletDiskType": "OS",
     "maxPods": 110,
     "type": "VirtualMachineScaleSets",
     "maxCount": 10,
     "minCount": 2,
     "enableAutoScaling": true,
     "scaleDownMode": "Delete",
     "provisioningState": "Succeeded",
     "powerState": {
      "code": "Running"
     },
     "orchestratorVersion": "1.24",
     "currentOrchestratorVersion": "1.24.10",
     "enableNodePublicIP": false,
     "enableCustomCATrust": false,
     "mode": "System",
     "enableEncryptionAtHost": false,
     "osType": "Linux",
     "osSKU": "Ubuntu",
     "nodeImageVersion": "AKSUbuntu-1804gen2containerd-202305.24.0",
     "upgradeSettings": {},
     "enableFIPS": false
    }
   ],
   "servicePrincipalProfile": {
    "clientId": "msi"
   },
   "addonProfiles": {
    "aciConnectorLinux": {
     "enabled": false,
     "config": null
    },
    "azureKeyvaultSecretsProvider": {
     "enabled": false,
     "config": null
    },
    "azurepolicy": {
     "enabled": false,
     "config": null
    },
    "httpApplicationRouting": {
     "enabled": false,
     "config": null
    },
    "ingressApplicationGateway": {
     "enabled": false,
     "config": null
    },
    "kubeDashboard": {
     "enabled": false,
     "config": null
    },
    "omsagent": {
     "enabled": false,
     "config": null
    },
    "openServiceMesh": {
     "enabled": false,
     "config": null
    }
   },
   "nodeResourceGroup": "MC_REDACTED-NAME_REDACTED-NAME_westeurope",
   "enableRBAC": true,
   "supportPlan": "KubernetesOfficial",
   "networkProfile": {
    "networkPlugin": "kubenet",
    "loadBalancerSku": "Standard",
    "loadBalancerProfile": {
     "managedOutboundIPs": {
      "count": 1
     },
     "effectiveOutboundIPs": [
      {
       "id": "/subscriptions/00000000-00000000-00000000-00000000/resourceGroups/MC_REDACTED-NAME_REDACTED-NAME_westeurope/providers/Microsoft.Network/publicIPAddresses/00000000-00000000-00000000-00000000"
      }
     ],
     "backendPoolType": "nodeIPConfiguration"
    },
    "podCidr": "10.244.0.0/16",
    "serviceCidr": "10.0.0.0/16",
    "dnsServiceIP": "10.0.0.10",
    "outboundType": "loadBalancer"
   },
   "maxAgentPools": 100,
   "identityProfile": {
    "kubeletidentity": {
     "resourceId": "/subscriptions/00000000-00000000-00000000-00000000/resourcegroups/MC_REDACTED-NAME_REDACTED-NAME_westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/REDACTED-NAME-agentpool",
     "clientId": "a188a581-a496-473d-bbd6-a7327799b941",
     "objectId": "bc896822-6e0b-46e3-9fdb-10c62a1c1494"
    }
   },
   "autoScalerProfile": {
    "balance-similar-node-groups": "false",
    "expander": "random",
    "max-empty-bulk-delete": "10",
    "max-graceful-termination-sec": "600",
    "max-node-provision-time": "15m",
    "max-total-unready-percentage": "45",
    "new-pod-scale-up-delay": "0s",
    "ok-total-unready-count": "3",
    "scale-down-delay-after-add": "10m",
    "scale-down-delay-after-delete": "10s",
    "scale-down-delay-after-failure": "3m",
    "scale-down-unneeded-time": "10m",
    "scale-down-unready-time": "20m",
    "scale-down-utilization-threshold": "0.5",
    "scan-interval": "10s",
    "skip-nodes-with-local-storage": "false",
    "skip-nodes-with-system-pods": "true"
   },
   "autoUpgradeProfile": {
    "upgradeChannel": "none"
   },
   "disableLocalAccounts": false,
   "securityProfile": {
    "imageCleaner": {
     "enabled": false,
     "intervalHours": 48
    }
   },
   "storageProfile": {
    "diskCSIDriver": {
     "enabled": true,
     "version": "v1"
    },
    "fileCSIDriver": {
     "enabled": true
    },
    "snapshotController": {
     "enabled": true
    }
   },
   "publicNetworkAccess": "Enabled",
   "oidcIssuerProfile": {
    "enabled": false
   },
   "workloadAutoScalerProfile": {}
  },
  "identity": {
   "type": "SystemAssigned",
   "principalId": "880325e6-3cf4-4b56-a82c-831098e19258",
   "tenantId": "f33e31d7-4a3b-4a9f-b4b2-3fa1d6bc4fb7"
  },
  "sku": {
   "name": "Base",
   "tier": "Standard"
  }
 }: timestamp=2023-10-03T12:29:36.653+0200
2023-10-03T12:29:36.654+0200 [DEBUG] provider.terraform-provider-azurerm_v3.75.0_x5: Upgrading the version of Kubernetes to "1.25.11"..: timestamp=2023-10-03T12:29:36.654+0200
2023-10-03T12:29:36.655+0200 [DEBUG] provider.terraform-provider-azurerm_v3.75.0_x5: AzureRM Request: 
PUT /subscriptions/00000000-00000000-00000000-00000000/resourceGroups/REDACTED-NAME/providers/Microsoft.ContainerService/managedClusters/REDACTED-NAME?api-version=2023-04-02-preview HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.20.5 (amd64-linux) go-autorest/v14.2.1 hashicorp/go-azure-sdk/managedclusters/2023-04-02-preview HashiCorp Terraform/1.5.7 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.75.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 3918
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 003465ec-c0fd-c8da-3663-8699b00493c8
Accept-Encoding: gzip

{"id":"/subscriptions/00000000-00000000-00000000-00000000/resourcegroups/REDACTED-NAME/providers/Microsoft.ContainerService/managedClusters/REDACTED-NAME","identity":{"type":"SystemAssigned","userAssignedIdentities":null},"location":"westeurope","name":"REDACTED-NAME","properties":{"addonProfiles":{"aciConnectorLinux":{"enabled":false},"azureKeyvaultSecretsProvider":{"enabled":false},"azurepolicy":{"enabled":false},"httpApplicationRouting":{"enabled":false},"ingressApplicationGateway":{"enabled":false},"kubeDashboard":{"enabled":false},"omsagent":{"enabled":false},"openServiceMesh":{"enabled":false}},"agentPoolProfiles":[{"count":3,"currentOrchestratorVersion":"1.24.10","enableAutoScaling":true,"enableCustomCATrust":false,"enableEncryptionAtHost":false,"enableFIPS":false,"enableNodePublicIP":false,"kubeletDiskType":"OS","maxCount":10,"maxPods":110,"minCount":2,"mode":"System","name":"default","nodeImageVersion":"AKSUbuntu-1804gen2containerd-202305.24.0","orchestratorVersion":"1.24","osDiskSizeGB":50,"osDiskType":"Managed","osSKU":"Ubuntu","osType":"Linux","powerState":{"code":"Running"},"provisioningState":"Succeeded","scaleDownMode":"Delete","type":"VirtualMachineScaleSets","upgradeSettings":{},"vmSize":"Standard_D4as_v5"}],"autoScalerProfile":{"balance-similar-node-groups":"false","expander":"random","max-empty-bulk-delete":"10","max-graceful-termination-sec":"600","max-node-provision-time":"15m","max-total-unready-percentage":"45","new-pod-scale-up-delay":"0s","ok-total-unready-count":"3","scale-down-delay-after-add":"10m","scale-down-delay-after-delete":"10s","scale-down-delay-after-failure":"3m","scale-down-unneeded-time":"10m","scale-down-unready-time":"20m","scale-down-utilization-threshold":"0.5","scan-interval":"10s","skip-nodes-with-local-storage":"false","skip-nodes-with-system-pods":"true"},"autoUpgradeProfile":{"upgradeChannel":"none"},"azurePortalFQDN":"REDACTED-DNS-308f7803.portal.hcp.westeurope.azmk8s.io","currentKubernetesVersion":"1.24.10","disableLocalAccounts":false,"dnsPrefix":"REDACTED-DNS","enableRBAC":true,"fqdn":"REDACTED-DNS-308f7803.hcp.westeurope.azmk8s.io","identityProfile":{"kubeletidentity":{"clientId":"a188a581-a496-473d-bbd6-a7327799b941","objectId":"bc896822-6e0b-46e3-9fdb-10c62a1c1494","resourceId":"/subscriptions/00000000-00000000-00000000-00000000/resourcegroups/MC_REDACTED-NAME_REDACTED-NAME_westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/REDACTED-NAME-agentpool"}},"kubernetesVersion":"1.25.11","maxAgentPools":100,"networkProfile":{"dnsServiceIP":"10.0.0.10","loadBalancerProfile":{"backendPoolType":"nodeIPConfiguration","effectiveOutboundIPs":[{"id":"/subscriptions/00000000-00000000-00000000-00000000/resourceGroups/MC_REDACTED-NAME_REDACTED-NAME_westeurope/providers/Microsoft.Network/publicIPAddresses/e1ab83ab-91c5-4378-9a00-d53b19578a28"}],"managedOutboundIPs":{"count":1}},"loadBalancerSku":"Standard","networkPlugin":"kubenet","outboundType":"loadBalancer","podCidr":"10.244.0.0/16","serviceCidr":"10.0.0.0/16"},"nodeResourceGroup":"MC_REDACTED-NAME_REDACTED-NAME_westeurope","oidcIssuerProfile":{"enabled":false},"powerState":{"code":"Running"},"provisioningState":"Succeeded","publicNetworkAccess":"Enabled","securityProfile":{"imageCleaner":{"enabled":false,"intervalHours":48}},"servicePrincipalProfile":{"clientId":"msi"},"storageProfile":{"diskCSIDriver":{"enabled":true,"version":"v1"},"fileCSIDriver":{"enabled":true},"snapshotController":{"enabled":true}},"supportPlan":"KubernetesOfficial","workloadAutoScalerProfile":{}},"sku":{"name":"Base","tier":"Standard"},"tags":{"terraform":"true"},"type":"Microsoft.ContainerService/ManagedClusters"}: timestamp=2023-10-03T12:29:36.654+0200
2023-10-03T12:29:36.859+0200 [TRACE] dag/walk: vertex "output.kubernetes_client_certificate (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:36.859+0200 [TRACE] dag/walk: vertex "output.kubernetes_client_key (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:36.859+0200 [TRACE] dag/walk: vertex "output.kubernetes_host (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:36.859+0200 [TRACE] dag/walk: vertex "root" is waiting for "output.kubernetes_host (expand)"
2023-10-03T12:29:36.859+0200 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:36.859+0200 [TRACE] dag/walk: vertex "output.kubernetes_password (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:36.859+0200 [TRACE] dag/walk: vertex "output.kubernetes_username (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:36.860+0200 [TRACE] dag/walk: vertex "output.kubernetes_cluster_ca_certificate (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "output.kubernetes_host (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "output.kubernetes_cluster_ca_certificate (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "output.kubernetes_username (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "output.kubernetes_client_certificate (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "output.kubernetes_password (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "output.kubernetes_client_key (expand)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "root" is waiting for "output.kubernetes_host (expand)"
2023-10-03T12:29:41.860+0200 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/azurerm\"] (close)" is waiting for "azurerm_kubernetes_cluster.k8s"
2023-10-03T12:29:44.309+0200 [DEBUG] provider.terraform-provider-azurerm_v3.75.0_x5: AzureRM Response for https://management.azure.com/subscriptions/00000000-00000000-00000000-00000000/resourceGroups/REDACTED-NAME/providers/Microsoft.ContainerService/managedClusters/REDACTED-NAME?api-version=2023-04-02-preview: 
HTTP/2.0 400 Bad Request
Content-Length: 413
Cache-Control: no-cache
Content-Type: application/json
Date: Tue, 03 Oct 2023 10:29:43 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 003465ec-c0fd-c8da-3663-8699b00493c8
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1199
X-Ms-Request-Id: 78b5883c-4476-4da7-87cb-10c8e217af4a
X-Ms-Routing-Request-Id: ITALYNORTH:20231003T102944Z:a2aaf04a-65fd-476c-b206-27d86933ebc9

{
  "code": "NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged",
  "details": null,
  "message": "Using managed cluster api, all Agent pools' OrchestratorVersion must be all specified or all unspecified. If all specified, they must be stay unchanged or the same with control plane. For agent pool specific change, please use per agent pool operations: https://aka.ms/agent-pool-rest-api",
  "subcode": ""
 }: timestamp=2023-10-03T12:29:44.309+0200
2023-10-03T12:29:44.309+0200 [TRACE] provider.terraform-provider-azurerm_v3.75.0_x5: Called downstream: @module=sdk.helper_schema tf_req_id=7b847e63-102b-4f4e-a171-97ec0b793d97 tf_resource_type=azurerm_kubernetes_cluster tf_provider_addr=provider tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:920 timestamp=2023-10-03T12:29:44.309+0200
2023-10-03T12:29:44.311+0200 [TRACE] provider.terraform-provider-azurerm_v3.75.0_x5: Received downstream response: tf_provider_addr=provider @module=sdk.proto tf_req_duration_ms=8757 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/tf5serverlogging/downstream_request.go:40 diagnostic_error_count=1 tf_proto_version=5.4 tf_resource_type=azurerm_kubernetes_cluster diagnostic_warning_count=0 tf_req_id=7b847e63-102b-4f4e-a171-97ec0b793d97 timestamp=2023-10-03T12:29:44.311+0200
2023-10-03T12:29:44.311+0200 [ERROR] provider.terraform-provider-azurerm_v3.75.0_x5: Response contains error diagnostic: tf_req_id=7b847e63-102b-4f4e-a171-97ec0b793d97 @module=sdk.proto diagnostic_detail= tf_proto_version=5.4 tf_provider_addr=provider @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 tf_resource_type=azurerm_kubernetes_cluster diagnostic_severity=ERROR diagnostic_summary="updating Kubernetes Version for Kubernetes Cluster (Subscription: "00000000-00000000-00000000-00000000"
Resource Group Name: "REDACTED-NAME"
Kubernetes Cluster Name: "REDACTED-NAME"): managedclusters.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged" Message="Using managed cluster api, all Agent pools' OrchestratorVersion must be all specified or all unspecified. If all specified, they must be stay unchanged or the same with control plane. For agent pool specific change, please use per agent pool operations: https://aka.ms/agent-pool-rest-api"" tf_rpc=ApplyResourceChange timestamp=2023-10-03T12:29:44.311+0200
2023-10-03T12:29:44.311+0200 [TRACE] provider.terraform-provider-azurerm_v3.75.0_x5: Served request: @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/tf5server/server.go:872 tf_proto_version=5.4 tf_provider_addr=provider @module=sdk.proto tf_req_id=7b847e63-102b-4f4e-a171-97ec0b793d97 tf_resource_type=azurerm_kubernetes_cluster tf_rpc=ApplyResourceChange timestamp=2023-10-03T12:29:44.311+0200

Expected Behaviour

Terraform/AKS would upgrade the cluster and the node pool from the current version (v1.24) to the new version (v1.25)

Actual Behaviour

The upgrade fails with the following error

{
  "code": "NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged",
  "details": null,
  "message": "Using managed cluster api, all Agent pools' OrchestratorVersion must be all specified or all unspecified. If all specified, they must be stay unchanged or the same with control plane. For agent pool specific change, please use per agent pool operations: https://aka.ms/agent-pool-rest-api",
  "subcode": ""
 }

According to the docs, this error should occur only when auto-upgrade is enabled, but this is not the case (link)

Steps to Reproduce

With the provided configuration, trying to upgrade Kubernetes from v1.24 to v1.25:

Important Factoids

Running terraform through terragrunt

References

Possibly related issues or PRs:

cippaciong commented 1 year ago

Hello. Just a quick update related to this issue. Since I couldn't find the reason why the upgrade was not working, I ended up doing it manually from the Azure UI. After the first upgrade from v1.24 to v1.25 was successful, I tried to do the next ones (up to v1.27) using Terraform and they worked fine. I just set both kubernetes_version and default_node_pool.orchestrator_version to the version that I wanted to upgrade to, as expected.

I'm closing the issue since it's clearly not a problem with the provider, but rather with the Azure API or due to a broken state of the cluster that was solved after the first upgrade.

If others have this issue in the future, they might consider doing the update using the UI first and then try again with terraform 🤷🏻

BertelBB commented 1 year ago

@cippaciong see my comment here: https://github.com/Azure/AKS/issues/3508#issuecomment-1568518512

cippaciong commented 1 year ago

@BertelBB I think I stumbled upon your comment while I was trying to debug my issue, but your scenario is not exactly the same as mine, although very similar.

Most of the time, the NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged error appears when automatic_channel_upgrade is enabled. In that case, your comment is 100% correct. As I saw from the trace logs, Terraform is sending two different values for kubernetes_version and default_node_pool.orchestrator_version, which is not supported by the API.

In my case though, I never enabled automatic_channel_upgrade and according to the docs it should be possible to do the two-step upgrade as Terraform is trying to do.
Or at least that's my interpretation of this section:

If you’re using cluster auto-upgrade, you can no longer upgrade the control plane first, and then upgrade the individual node pools. Cluster auto-upgrade always upgrades the control plane and the node pools together. You can't upgrade the control plane only. Running the az aks upgrade --control-plane-only command raises the following error: NotAllAgentPoolOrchestratorVersionSpecifiedAndUnchanged: Using managed cluster api, all Agent pools' OrchestratorVersion must be all specified or all unspecified. If all specified, they must be stay unchanged or the same with control plane.

MageshSrinivasulu commented 11 months ago

Facing the same issue when upgrading from 1.26.3 to 1.27.7 where I too didn't set automatic_channel_upgrade

pfarikrispy commented 11 months ago

we are suddenly seeing the same issue in an AKS cluster that was first at the 1.28 preview version and then auto-upgraded to 1.28.3 upon release. I tried hardcoding the version in terraform and reapplying but that has no effect ![Uploading image.png…]()

ashwajce commented 11 months ago

Same here while upgrading from 1.28 version and 1.28.3 version we are getting this error.

github-actions[bot] commented 7 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.