[BUG] azure container storage extension stuck in creating/deleting state

[akari-m]

Describe the bug azure container storage extension stuck in creating/deleting state

To Reproduce Steps to reproduce the behavior:

1. delete the extension in azure aks portal
2. from the notifications, it shows Successfully removed extension, but from the extension status, it shows it's in creating state, error code unable to add the configuration with configId {extension:azurecontainerstorage} due to error: {Unable to store the extension config update on the cluster's api-server : Recommendation Please ensure the config-agent exists on a healthy node : InnerError [extensionconfigs.clusterconfig.azure.com "azurecontainerstorage" is forbidden: unable to create new content in namespace acstor because it is being terminated]}

I tried to delete the extension pod via kubectl, and it shows ❯ kdel pod -n acstor etcd-azurecontainerstorage-c5zkpwp4mv --force --grace-period=0 Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (InternalError): Internal error occurred: failed calling webhook "podeviction.acstor.azure.com": failed to call webhook: Post "https://webhook-service.acstor.svc:443/pod-eviction?timeout=30s": service "webhook-service" not found

Now I could not either install/uninstall the extension, start/stop aks and az aks update did not help.

Expected behavior I hope I could delete the extension through azure aks portal or delete the whole namespace via kubectl

Screenshots

Environment (please complete the following information):

• CLI Version [e.g. 3.22]
• Kubernetes version1.28.5
• CLI Extension version [e.g. 1.7.5] if applicable
• Browser [e.g. chrome, safari] is applicable

Additional context Add any other context about the problem here.

[andyzhangx]

@edreed could you help? thx