aks update for enable-secret-rotation failing

mikewarr commented 2 years ago

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name az aks update Extension Name: aks-preview. Version: 0.5.85.

Errors:

The command failed with an unexpected error. Here is the traceback:
'NoneType' object does not support item assignment
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/home/mikewarr/.azure/cliextensions/aks-preview/azext_aks_preview/custom.py", line 927, in aks_update
    mc = aks_update_decorator.update_mc_profile_preview()
  File "/home/mikewarr/.azure/cliextensions/aks-preview/azext_aks_preview/managed_cluster_decorator.py", line 1931, in update_mc_profile_preview
    mc = self.update_mc_profile_default()
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/acs/managed_cluster_decorator.py", line 5674, in update_mc_profile_default
    mc = self.update_addon_profiles(mc)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/acs/managed_cluster_decorator.py", line 5588, in update_addon_profiles
    self.update_azure_keyvault_secrets_provider_addon_profile(azure_keyvault_secrets_provider_addon_profile)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/acs/managed_cluster_decorator.py", line 5522, in update_azure_keyvault_secrets_provider_addon_profile
    azure_keyvault_secrets_provider_addon_profile.config[
TypeError: 'NoneType' object does not support item assignment

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az aks update -g {} -n {} --enable-secret-rotation --rotation-poll-interval {}

Expected Behavior

Environment Summary

Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31, Ubuntu 20.04.4 LTS
Python 3.10.4
Installer: DEB

azure-cli 2.37.0

Extensions:
aks-preview 0.5.85

Dependencies:
msal 1.18.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

yonzhan commented 2 years ago

route to CXP team

navba-MSFT commented 2 years ago

@mikewarr Thanks for reaching out to us and reporting this issue. We are looking into this issue and we will provide an update.

navba-MSFT commented 2 years ago

@mikewarr Could you please also check if the secret provider addon profile exists by running az aks show -g {resource_group_name} -n {cluster_name} --query "addonProfiles"?

Action Plan: Could you please disable the addon first by running below command?

az aks disable-addons -g {resource_group_name} -n {cluster_name} -a azure-keyvault-secrets-provider

And then enable this addon by running below:

az aks enable-addons -g {resource_group_name} -n {cluster_name} -a azure-keyvault-secrets-provider --enable-secret-rotation

Awaiting your reply.

mikewarr commented 2 years ago

Sure, I ran the first command and got this:

mikewarr@COODASL3:~/akswth/WhatTheHack/039-AKSEnterpriseGrade$ az aks show -g akswth -n akswthclus --query "addonProfiles" The behavior of this command has been altered by the following extension: aks-preview { "azureKeyvaultSecretsProvider": { "config": null, "enabled": true, "identity": { "clientId": "c5c1fe71-2187-42b5-b0ad-dc87f32435fd", "objectId": "5eb5ac0c-b81b-48a3-ad15-50862985fda3", "resourceId": "/subscriptions/93b156ca-7643-4977-98b5-7a6e2430986f/resourcegroups/MC_akswth_akswthclus_westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurekeyvaultsecretsprovider-akswthclus" } }, "azurepolicy": { "config": null, "enabled": false, "identity": null }, "httpApplicationRouting": { "config": null, "enabled": false, "identity": null }, "ingressApplicationGateway": { "config": { "applicationGatewayName": "ingress-appgateway", "effectiveApplicationGatewayId": "/subscriptions/93b156ca-7643-4977-98b5-7a6e2430986f/resourceGroups/MC_akswth_akswthclus_westeurope/providers/Microsoft.Network/applicationGateways/ingress-appgateway", "subnetPrefix": "10.6.4.0/24" }, "enabled": true, "identity": { "clientId": "5fda67bf-7f93-4756-9da5-de03ead77243", "objectId": "4210ff78-ce40-4ee9-acda-678de39d2f63", "resourceId": "/subscriptions/93b156ca-7643-4977-98b5-7a6e2430986f/resourcegroups/MC_akswth_akswthclus_westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingressapplicationgateway-akswthclus" } }, "omsAgent": { "config": { "logAnalyticsWorkspaceResourceID": "/subscriptions/93b156ca-7643-4977-98b5-7a6e2430986f/resourcegroups/warr-core-rg/providers/microsoft.operationalinsights/workspaces/warrneloganalytics" }, "enabled": true, "identity": { "clientId": "b2944ef8-1fc9-4996-800b-c7e50f68b315", "objectId": "569afbc1-0524-4bb9-bd18-9c6369619765", "resourceId": "/subscriptions/93b156ca-7643-4977-98b5-7a6e2430986f/resourcegroups/MC_akswth_akswthclus_westeurope/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-akswthclus" } } }

Then I ran the disable command but that provided an error:

mikewarr@COODASL3:~/akswth/WhatTheHack/039-AKSEnterpriseGrade$ az aks disable-addons -g akswth -n akswthclus -a azure-keyvault-secrets-provider The behavior of this command has been altered by the following extension: aks-preview (BadRequest) AzureKeyvaultSecretsProvider addon cannot be disabled due to more than 0 Secret Provider Classes Code: BadRequest Message: AzureKeyvaultSecretsProvider addon cannot be disabled due to more than 0 Secret Provider Classes

navba-MSFT commented 2 years ago

@mikewarr The error is due to the secret provider class being in use on the cluster. If this won't disrupt your service, could you please remove the secret provider classes in use and then perform the above-mentioned disable and enable operations one by one?

Or alternatively, you could also try to perform the action plan to disable and enable the addon while its not in use. Perhaps in the late evening. Awaiting your reply.

navba-MSFT commented 2 years ago

@mikewarr I wanted to do quick follow-up to check if you had a chance to look at my above comment. Please let us know if you need any further assistance on this. Awaiting your reply.

navba-MSFT commented 2 years ago

@mikewarr The above PR has been filed as a permanent fix. However, please let me know if you have tried the above workaround. Awaiting your reply.

navba-MSFT commented 2 years ago

@mikewarr The PR Azure/azure-cli#23088 has been merged now. The fix should be released as per this milestone: https://github.com/Azure/azure-cli/milestone/120 . Until then you can follow my above workaround.

ozbillwang commented 1 year ago

navba-MSFT

Here is the real command we need run to remove the secret provider classes in use, after run it, you can disable the addon successfully

$ kk get secretproviderclasses

$ kubectl delete secretproviderclasses <the_name>

Azure / azure-cli-extensions