search

Azure / azure-cli

Azure Command-Line Interface
MIT License
4.01k stars 2.98k forks source link

[AKS] `az aks nodepool update`: can't change OS SKU #29520

Closed ptrautberg closed 2 months ago

ptrautberg commented 2 months ago

Describe the bug

According to docs we can now perform an in-place upgrade of node's OS - directly from CLI or via IaC. When I'm trying to use this feature and migrate from Ubuntu to Azure Linux, I'm getting Changing property 'agentPoolProfile.OSSKU' is not allowed error.

The same issue is opened for Terraform's AzureRM provider - link.

Related command

az aks nodepool update --resource-group <rg> --cluster-name <aks> --name <nodepool> --os-sku AzureLinux

Errors

(PropertyChangeNotAllowed) Changing property 'agentPoolProfile.OSSKU' is not allowed.
Code: PropertyChangeNotAllowed
Message: Changing property 'agentPoolProfile.OSSKU' is not allowed.
Target: agentPoolProfile.OSSKU

Issue script & Debug output

cli.knack.cli: Command arguments: ['aks', 'nodepool', 'update', '--resource-group', '*REDACTED*-dev-rg', '--cluster-name', '*REDACTED*-dev-aks', '--name', 'application', '--os-sku', 'AzureLinux', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7e19d0088040>, <function OutputProducer.on_global_arguments at 0x7e19d0036200>, <function CLIQuery.on_global_arguments at 0x7e19cfe4fce0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'aks': ['azure.cli.command_modules.acs', 'azure.cli.command_modules.serviceconnector']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: acs                       0.035        14        76
cli.azure.cli.core: serviceconnector          0.109        20       307
cli.azure.cli.core: Total (2)                 0.143        34       383
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 33 groups, 383 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : aks nodepool update
cli.azure.cli.core: Command table: aks nodepool update
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7e19cefba980>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-07-30.12-42-45.aks_nodepool_update.12770.log'.
az_command_data_logger: command args: aks nodepool update --resource-group {} --cluster-name {} --name {} --os-sku {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7e19cefef9c0>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7e19cf006c00>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7e19cf006d40>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7e19d00362a0>, <function CLIQuery.handle_query_parameter at 0x7e19cfe4fd80>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7e19cf006ca0>]
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ContainerServiceClient
cli.azure.cli.core.auth.persistence: build_persistence: location='/root/.azure/service_principal_entries.json', encrypt=False
cli.azure.cli.core.auth.persistence: build_persistence: location='/root/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /root/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/*REDACTED*
msal.authority: openid_config("https://login.microsoftonline.com/*REDACTED*/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/*REDACTED*/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/*REDACTED*/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/*REDACTED*/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/*REDACTED*/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/*REDACTED*/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/*REDACTED*/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/*REDACTED*/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? None
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: ef55de13-1a33-4b8b-9688-0f6733e627c7
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.ContainerService/managedClusters/*REDACTED*-dev-aks/agentPools/application?api-version=2024-02-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '38880250-4e71-11ef-9b03-00163e1e60df'
cli.azure.cli.core.sdk.policies:     'CommandName': 'aks nodepool update'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--resource-group --cluster-name --name --os-sku --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.62.0 (DEB) azsdk-python-core/1.28.0 Python/3.11.8 (Linux-6.5.0-1022-aws-x86_64-with-glibc2.35)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.ContainerService/managedClusters/*REDACTED*-dev-aks/agentPools/application?api-version=2024-02-01 HTTP/1.1" 200 1809
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '1809'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-reads': '249'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '3fe0a43a-6e8a-4953-9113-2640b660762d'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': 'fdfbe0f7-ed7d-4491-88d0-9564f1272791'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-global-reads': '3749'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'GERMANYWESTCENTRAL:20240730T124246Z:3fe0a43a-6e8a-4953-9113-2640b660762d'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: 0BF1E942644D4517AACC811AF0701949 Ref B: VIEEDGE1918 Ref C: 2024-07-30T12:42:46Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Tue, 30 Jul 2024 12:42:46 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
 "id": "/subscriptions/*REDACTED*/resourcegroups/*REDACTED*-dev-rg/providers/Microsoft.ContainerService/managedClusters/*REDACTED*-dev-aks/agentPools/application",
 "name": "application",
 "type": "Microsoft.ContainerService/managedClusters/agentPools",
 "properties": {
  "count": 1,
  "vmSize": "Standard_D4ds_v5",
  "osDiskSizeGB": 128,
  "osDiskType": "Ephemeral",
  "kubeletDiskType": "OS",
  "vnetSubnetID": "/subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.Network/virtualNetworks/*REDACTED*-dev-vnet/subnets/*REDACTED*-dev-subnet-aks_node",
  "podSubnetID": "/subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.Network/virtualNetworks/*REDACTED*-dev-vnet/subnets/*REDACTED*-dev-subnet-aks_pod",
  "maxPods": 250,
  "type": "VirtualMachineScaleSets",
  "maxCount": 50,
  "minCount": 0,
  "enableAutoScaling": true,
  "scaleDownMode": "Delete",
  "provisioningState": "Succeeded",
  "powerState": {
   "code": "Running"
  },
  "orchestratorVersion": "1.28.9",
  "currentOrchestratorVersion": "1.28.9",
  "enableNodePublicIP": false,
  "scaleSetPriority": "Spot",
  "scaleSetEvictionPolicy": "Delete",
  "spotMaxPrice": -1,
  "tags": {
   "Created by": "*REDACTED*",
   "Environment": "dev",
   "Project": "*REDACTED*",
   "Terraform": "True"
  },
  "nodeLabels": {
   "dedicatedGroup": "application",
   "kubernetes.azure.com/scalesetpriority": "spot"
  },
  "nodeTaints": [
   "kubernetes.azure.com/scalesetpriority=spot:NoSchedule"
  ],
  "mode": "User",
  "enableEncryptionAtHost": true,
  "enableUltraSSD": false,
  "osType": "Linux",
  "osSKU": "Ubuntu",
  "nodeImageVersion": "AKSUbuntu-2204gen2containerd-202407.03.0",
  "upgradeSettings": {},
  "enableFIPS": false
 }
}
cli.azure.cli.command_modules.acs.base_decorator: The intermediate 'snapshot' does not exist. Return default value 'None'.
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.ContainerService/managedClusters/*REDACTED*-dev-aks/agentPools/application?api-version=2024-02-01'
cli.azure.cli.core.sdk.policies: Request method: 'PUT'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies:     'Content-Length': '1289'
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': '38880250-4e71-11ef-9b03-00163e1e60df'
cli.azure.cli.core.sdk.policies:     'CommandName': 'aks nodepool update'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--resource-group --cluster-name --name --os-sku --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.62.0 (DEB) azsdk-python-core/1.28.0 Python/3.11.8 (Linux-6.5.0-1022-aws-x86_64-with-glibc2.35)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: {"properties": {"count": 1, "vmSize": "Standard_D4ds_v5", "osDiskSizeGB": 128, "osDiskType": "Ephemeral", "kubeletDiskType": "OS", "vnetSubnetID": "/subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.Network/virtualNetworks/*REDACTED*-dev-vnet/subnets/*REDACTED*-dev-subnet-aks_node", "podSubnetID": "/subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.Network/virtualNetworks/*REDACTED*-dev-vnet/subnets/*REDACTED*-dev-subnet-aks_pod", "maxPods": 250, "osType": "Linux", "osSKU": "AzureLinux", "maxCount": 50, "minCount": 0, "enableAutoScaling": true, "scaleDownMode": "Delete", "type": "VirtualMachineScaleSets", "mode": "User", "orchestratorVersion": "1.28.9", "upgradeSettings": {}, "powerState": {"code": "Running"}, "enableNodePublicIP": false, "scaleSetPriority": "Spot", "scaleSetEvictionPolicy": "Delete", "spotMaxPrice": -1.0, "tags": {"Created by": "*REDACTED*", "Environment": "dev", "Project": "*REDACTED*", "Terraform": "True"}, "nodeLabels": {"dedicatedGroup": "application", "kubernetes.azure.com/scalesetpriority": "spot"}, "nodeTaints": ["kubernetes.azure.com/scalesetpriority=spot:NoSchedule"], "enableEncryptionAtHost": true, "enableUltraSSD": false, "enableFIPS": false}}
urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/*REDACTED*/resourceGroups/*REDACTED*-dev-rg/providers/Microsoft.ContainerService/managedClusters/*REDACTED*-dev-aks/agentPools/application?api-version=2024-02-01 HTTP/1.1" 400 190
cli.azure.cli.core.sdk.policies: Response status: 400
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Length': '190'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-writes': '199'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '547a0052-71f6-4d44-b176-73e3c4bf044c'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '3c01dfdb-cc4e-4026-b567-7643d647394a'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'x-ms-ratelimit-remaining-subscription-global-writes': '2999'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'GERMANYNORTH:20240730T124252Z:547a0052-71f6-4d44-b176-73e3c4bf044c'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'X-Cache': 'CONFIG_NOCACHE'
cli.azure.cli.core.sdk.policies:     'X-MSEdge-Ref': 'Ref A: 50DA22E25F344B468CF3FCB165CA4497 Ref B: VIEEDGE1918 Ref C: 2024-07-30T12:42:46Z'
cli.azure.cli.core.sdk.policies:     'Date': 'Tue, 30 Jul 2024 12:42:52 GMT'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {
  "code": "PropertyChangeNotAllowed",
  "details": null,
  "message": "Changing property 'agentPoolProfile.OSSKU' is not allowed.",
  "subcode": "",
  "target": "agentPoolProfile.OSSKU"
 }
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 664, in execute
    raise ex
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 731, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 701, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 334, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/acs/custom.py", line 2397, in aks_agentpool_update
    return aks_agentpool_update_decorator.update_agentpool(agentpool)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/acs/agentpool_decorator.py", line 2011, in update_agentpool
    return sdk_no_wait(
           ^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/util.py", line 710, in sdk_no_wait
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/mgmt/containerservice/v2024_02_01/operations/_agent_pools_operations.py", line 836, in begin_create_or_update
    raw_result = self._create_or_update_initial(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/mgmt/containerservice/v2024_02_01/operations/_agent_pools_operations.py", line 719, in _create_or_update_initial
    raise HttpResponseError(response=response, error_format=ARMErrorFormat)
azure.core.exceptions.HttpResponseError: (PropertyChangeNotAllowed) Changing property 'agentPoolProfile.OSSKU' is not allowed.
Code: PropertyChangeNotAllowed
Message: Changing property 'agentPoolProfile.OSSKU' is not allowed.
Target: agentPoolProfile.OSSKU

cli.azure.cli.core.azclierror: (PropertyChangeNotAllowed) Changing property 'agentPoolProfile.OSSKU' is not allowed.
Code: PropertyChangeNotAllowed
Message: Changing property 'agentPoolProfile.OSSKU' is not allowed.
Target: agentPoolProfile.OSSKU
az_command_data_logger: (PropertyChangeNotAllowed) Changing property 'agentPoolProfile.OSSKU' is not allowed.
Code: PropertyChangeNotAllowed
Message: Changing property 'agentPoolProfile.OSSKU' is not allowed.
Target: agentPoolProfile.OSSKU
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7e19cefbac00>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 7.139 seconds (init: 0.125, invoke: 7.014)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4027 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /root/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

Migrate OS to Azure Linux

Environment Summary

azure-cli                         2.62.0

core                              2.62.0
telemetry                          1.1.0

Extensions:
account                            0.2.5
managementpartner                  0.1.3

Dependencies:
msal                              1.28.1
azure-mgmt-resource               23.1.1

Python location '/opt/az/bin/python3'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.8 (main, Jul  4 2024, 02:23:40) [GCC 11.4.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

Update using aks-preview extension also didn't work.

yonzhan commented 2 months ago

Thank you for opening this issue, we will look into it.

microsoft-github-policy-service[bot] commented 2 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dyu1208, @FumingZhang, @andyliuliming.

FumingZhang commented 2 months ago

@ptrautberg this is a server-side issue, and creating a support ticket (e.g., via Azure Portal) is a better way to handle issues like this.

I checked the code and found that this was a problem caused by a validation on the server side. It has been fixed and is being deployed. You could use this page to track the release progress of AKS. Once the fix (included in version v20240716) is deployed to the region where your cluster is located (it will take about 1 week if nothing unexpected happens), you could retry the command (no matter via azure-cli or terraform), and it should work. https://releases.aks.azure.com/webpage/index.html

Or, if you want a workaround immediately, please reach out to the support team, thanks!

ptrautberg commented 2 months ago

@FumingZhang, thanks for that. Now I see in release 2024-07-16 information that OS SKU migration is GA starting from this release - my mistake, I didn't check that.