search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.21k stars 3.81k forks source link

Disable-AzAksAddon to remove monitoring addon returns error "code":"BadRequest","message":"The request format was unexpected, a non-UserAssigned identity type should not contain: userAssignedIdentities" #17875

Open thiruma0203 opened 2 years ago

thiruma0203 commented 2 years ago

Description

Cmdlet reference shows an example for removing monitoring addon from AKS cluster. https://docs.microsoft.com/ja-jp/powershell/module/az.aks/disable-azaksaddon?view=azps-7.4.0

However, running below cmdlet returns error. Get-AzAksCluster -ResourceGroupName group -Name myCluster | Disable-AzAksAddon -Name Monitoring

"code":"BadRequest","message":"The request format was u nexpected, a non-UserAssigned identity type should not contain: userAssignedIdentities"

Repro steps:

  1. Deploy an AKS cluster from Azure portal with monitoring settings enabled.
  2. Run above command. Command fails with an error. (Repro)

Azure CLI works fine to remove monitoring addon for the same cluster. az aks disable-addons -a monitoring -n myCluster -g group

Issue script & Debug output

PS C:\WINDOWS\system32> Get-AzAksCluster -ResourceGroupName AKSTestRG -Name AKSTest2 | Disable-AzAksAddon -Name Monitoring -verbose -debug
警告: Upcoming breaking changes in the cmdlet 'Get-AzAksCluster' :
Get-AzAks will be removed in the next major release. Please use Get-AzAksCluster instead of Get-AzAks

Debug: ============================ HTTP REQUEST ============================

HTTP Method:
GET

Absolute Uri:
https://management.azure.com/subscriptions/*****/resourceGroups/AKSTestRG/providers/Microsoft.ContainerService/managedClusters/AKSTest2?api-version=2021-05-01

Headers:
x-ms-client-request-id        : 9b96a256-b393-4fb6-9023-01f7a244af04
accept-language               : en-US

Body:

Debug: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Pragma                        : no-cache
x-ms-routing-request-id       : JAPANEAST:20220421T082742Z:816e9a11-0118-44c9-af7c-89dee03f3223
x-ms-ratelimit-remaining-subscription-reads: 11998
x-ms-correlation-request-id   : 816e9a11-0118-44c9-af7c-89dee03f3223
x-ms-request-id               : 6bd6086d-7ecd-4db9-9f4c-0437c0d311b8
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Cache-Control                 : no-cache
Date                          : Thu, 21 Apr 2022 08:27:41 GMT
Server                        : nginx

Body:
{
  "id": "/subscriptions/*****/resourcegroups/AKSTestRG/providers/Microsoft.ContainerService/managedClusters/AKSTest2",
  "location": "westus2",
  "name": "AKSTest2",
  "type": "Microsoft.ContainerService/ManagedClusters",
  "properties": {
    "provisioningState": "Succeeded",
    "powerState": {
      "code": "Running"
    },
    "kubernetesVersion": "1.22.6",
    "dnsPrefix": "AKSTest2-dns",
    "fqdn": "*****.azmk8s.io",
    "azurePortalFQDN": "*****.azmk8s.io",
    "agentPoolProfiles": [
      {
        "name": "agentpool",
        "count": 3,
        "vmSize": "Standard_B4ms",
        "osDiskSizeGB": 128,
        "osDiskType": "Managed",
        "kubeletDiskType": "OS",
        "maxPods": 110,
        "type": "VirtualMachineScaleSets",
        "enableAutoScaling": false,
        "provisioningState": "Succeeded",
        "powerState": {
          "code": "Running"
        },
        "orchestratorVersion": "1.22.6",
        "enableNodePublicIP": false,
        "mode": "System",
        "osType": "Linux",
        "osSKU": "Ubuntu",
        "nodeImageVersion": "AKSUbuntu-1804gen2containerd-2022.04.05",
        "enableFIPS": false
      }
    ],
    "servicePrincipalProfile": {
      "clientId": "msi"
    },
    "addonProfiles": {
      "azureKeyvaultSecretsProvider": {
        "enabled": false,
        "config": null
      },
      "azurepolicy": {
        "enabled": false,
        "config": null
      },
      "httpApplicationRouting": {
        "enabled": false,
        "config": null
      },
      "omsagent": {
        "enabled": true,
        "config": {
          "logAnalyticsWorkspaceResourceID": "/subscriptions/*****/resourcegroups/defaultresourcegroup-wus2/providers/microsoft.operationalinsights/workspaces/defaultworkspace-*****-wus2"
        },
        "identity": {
          "resourceId": "/subscriptions/*****/resourcegroups/MC_AKSTestRG_AKSTest2_westus2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-akstest2",
          "clientId": "f4c49a05-5cf4-47f4-9830-0461a35b2e8f",
          "objectId": "5dea28bd-e1ab-4fda-914b-300d808d6c89"
        }
      }
    },
    "nodeResourceGroup": "MC_AKSTestRG_AKSTest2_westus2",
    "enableRBAC": true,
    "networkProfile": {
      "networkPlugin": "kubenet",
      "loadBalancerSku": "Standard",
      "loadBalancerProfile": {
        "managedOutboundIPs": {
          "count": 1
        },
        "effectiveOutboundIPs": [
          {
            "id": "/subscriptions/*****/resourceGroups/MC_AKSTestRG_AKSTest2_westus2/providers/Microsoft.Network/publicIPAddresses/f3a597cf-9c49-4429-bdd1-f30303035207"
          }
        ]
      },
      "podCidr": "10.244.0.0/16",
      "serviceCidr": "10.0.0.0/16",
      "dnsServiceIP": "10.0.0.10",
      "dockerBridgeCidr": "172.17.0.1/16",
      "outboundType": "loadBalancer"
    },
    "maxAgentPools": 100,
    "apiServerAccessProfile": {
      "enablePrivateCluster": false
    },
    "identityProfile": {
      "kubeletidentity": {
        "resourceId": "/subscriptions/*****/resourcegroups/MC_AKSTestRG_AKSTest2_westus2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AKSTest2-agentpool",
        "clientId": "37d7fed1-cebb-4a3b-9cbf-ebc1f2bc868a",
        "objectId": "9c7dc8ee-dc79-436b-8507-0c40974c5a7a"
      }
    }
  },
  "identity": {
    "type": "SystemAssigned",
    "principalId": "**********",
    "tenantId": "*********"
  },
  "sku": {
    "name": "Basic",
    "tier": "Free"
  }
}

Debug: ============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/*****/resourceGroups/AKSTestRG/providers/Microsoft.ContainerService/managedClusters/AKSTest2?api-version=2021-05-01

Headers:
x-ms-client-request-id        : 9b96a256-b393-4fb6-9023-01f7a244af04
accept-language               : en-US

Body:
{
  "properties": {
    "kubernetesVersion": "1.22.6",
    "dnsPrefix": "AKSTest2-dns",
    "addonProfiles": {
      "azureKeyvaultSecretsProvider": {
        "enabled": false,
        "config": {}
      },
      "azurepolicy": {
        "enabled": false,
        "config": {}
      },
      "httpApplicationRouting": {
        "enabled": false,
        "config": {}
      },
      "omsagent": {
        "enabled": false
      }
    },
    "nodeResourceGroup": "MC_AKSTestRG_AKSTest2_westus2",
    "enableRBAC": true,
    "networkProfile": {
      "networkPlugin": "kubenet",
      "podCidr": "10.244.0.0/16",
      "serviceCidr": "10.0.0.0/16",
      "dnsServiceIP": "10.0.0.10",
      "dockerBridgeCidr": "172.17.0.1/16",
      "loadBalancerSku": "Standard",
      "loadBalancerProfile": {
        "managedOutboundIPs": {
          "count": 1
        },
        "effectiveOutboundIPs": [
          {
            "id": "/subscriptions/*****/resourceGroups/MC_AKSTestRG_AKSTest2_westus2/providers/Microsoft.Network/publicIPAddresses/f3a597cf-9c49-4429-bdd1-f30303035207"
          }
        ]
      }
    },
    "apiServerAccessProfile": {
      "authorizedIPRanges": [],
      "enablePrivateCluster": false
    },
    "identityProfile": {
      "kubeletidentity": {
        "resourceId": "/subscriptions/*****/resourcegroups/MC_AKSTestRG_AKSTest2_westus2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/AKSTest2-agentpool",
        "clientId": "37d7fed1-cebb-4a3b-9cbf-ebc1f2bc868a",
        "objectId": "9c7dc8ee-dc79-436b-8507-0c40974c5a7a"
      }
    },
    "privateLinkResources": []
  },
  "identity": {
    "type": "SystemAssigned",
    "userAssignedIdentities": {}
  },
  "location": "westus2",
  "tags": {}
}

Debug: ============================ HTTP RESPONSE ============================

Status Code:
BadRequest

Headers:
Pragma                        : no-cache
x-ms-failure-cause            : gateway
x-ms-request-id               : 04d46c3a-775d-46d0-9879-06e518779f9e
x-ms-correlation-request-id   : 04d46c3a-775d-46d0-9879-06e518779f9e
x-ms-routing-request-id       : JAPANEAST:20220421T082746Z:04d46c3a-775d-46d0-9879-06e518779f9e
Strict-Transport-Security     : max-age=31536000; includeSubDomains
X-Content-Type-Options        : nosniff
Cache-Control                 : no-cache
Date                          : Thu, 21 Apr 2022 08:27:45 GMT

Body:
{
  "error": {
    "code": "FailedIdentityOperation",
    "message": "Identity operation for resource '/subscriptions/*****/resourceGroups/AKSTestRG/providers/Microsoft.ContainerService/managedClusters/AKSTest2' failed with error 'Failed to perform resource identity operation. Status:
'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"The request format was unexpected, a non-UserAssigned identity type should not contain: userAssignedIdentities\"}}'.'."
  }
}

Identity operation for resource '/subscriptions/*****/resourceGroups/AKSTestRG/providers/Microsoft.ContainerService/managedClusters/AKSTest2' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response:
 '{"error":{"code":"BadRequest","message":"The request format was unexpected, a non-UserAssigned identity type should not contain: userAssignedIdentities"}}'.'.

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.22000.613
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.22000.613
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Module versions

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     2.7.5      Az.Accounts                         {Add-AzEnvironment, Clear-AzContext, Clear-AzDefault, Connect-AzAccount...}
Script     3.1.2      Az.Aks                              {Disable-AzAksAddOn, Enable-AzAksAddOn, Get-AzAksCluster, Get-AzAksNodePool...}

Error output

Resolve-AzError
WARNING: Upcoming breaking changes in the cmdlet 'Resolve-AzError' :
The `Resolve-Error` alias will be removed in a future release.  Please change any scripts that use this alias to use `Resolve-AzError` instead.
Note : Go to https://aka.ms/azps-changewarnings for steps to suppress this breaking change warning, and other information on breaking changes in Azure PowerShell.
Resolve-AzError: Input string was not in a correct format.
dingmeng-xue commented 2 years ago

@wyunchi-ms , you can see the other properties of Identity were missing and empty userAssignedIdentities was added. Is it the similar issue we are fixing?

mhessaveva commented 1 year ago

Enable-AzAksAddon has the same result. I had an open support ticket with MS on this, and the workaround I was given was to not pipe the cluster object in, and instead just supply the Enable/Disable-AzAksAddon commandlet with -ClusterName and -ResourceGroupName instead.