search

Azure / azure-cli-extensions

Public Repository for Extensions of Azure CLI.
https://docs.microsoft.com/en-us/cli/azure
MIT License
382 stars 1.23k forks source link

Az Monitor for Arc k8s extension deletion/reinstall fails #3315

Open ezYakaEagle442 opened 3 years ago

ezYakaEagle442 commented 3 years ago

Extension name (the extension in question)

k8s-extension

Description of issue (in as much detail as possible)

az k8s-extension delete does not delete rsources from the cluster

The extnesion is displayed as FAILEd then in the Azure portal with errors : 

Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "omsagent" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config
Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: Secret "omsagent-secret" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config 
az version
{
  "azure-cli": "2.22.1",
  "azure-cli-core": "2.22.1",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {
    "connectedk8s": "1.1.3",
    "connectedmachine": "0.3.0",
    "k8s-configuration": "1.0.0",
    "k8s-extension": "0.2.1",
    "spring-cloud": "2.3.0"
  }
}

I di test it on a K3S cluster deployed on an Azure VM

workaround : I had to manually clean-up the environment : 

az k8s-extension delete --name azuremonitor-containers --cluster-type connectedClusters --cluster-name $azure_arc_k3s  -g $k3s_rg_name -y
k get sa -n kube-system
k delete sa omsagent -n kube-system
k get sa -n kube-system
k get secrets -n kube-system
k delete secret omsagent-secret -n kube-system
k get cm -n kube-system
k delete cm omsagent-rs-config -n kube-system
k get cm -n kube-system
k delete deploy omsagent-rs -n kube-system
k get crd
k delete crd healthstates.azmon.container.insights

curl -o disable-monitoring.sh -L https://aka.ms/disable-monitoring-bash-script
bash disable-monitoring.sh --resource-id $azureArc_K3S_ClusterResourceId --kube-context $kubeContext

Then recreate :

az monitor log-analytics workspace create -n $analytics_workspace_name --location $location -g $k3s_rg_name --verbose
export analytics_workspace_id=$(az monitor log-analytics workspace show -n $analytics_workspace_name -g $k3s_rg_name -o tsv --query id)
echo "analytics_workspace_id:" $analytics_workspace_id

az k8s-extension create --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings logAnalyticsWorkspaceResourceID=$analytics_workspace_id omsagent.resources.daemonset.limits.cpu=150m omsagent.resources.daemonset.limits.memory=600Mi omsagent.resources.deployment.limits.cpu=1 omsagent.resources.deployment.limits.memory=750Mi

az k8s-extension list --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters 
azmon_extension_state=$(az k8s-extension show --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --query 'installState')
echo "Azure Monitor extension state: " $azmon_extension_state
ghost commented 3 years ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @NarayanThiru.

Issue Details
- If the issue is to do with Azure CLI 2.0 in-particular, create an issue here at [Azure/azure-cli](https://github.com/Azure/azure-cli/issues) ### Extension name (the extension in question) k8s-extension ### Description of issue (in as much detail as possible) az k8s-extension delete does not delete rsources from the cluster The extnesion is displayed as FAILEd then in the Azure portal with errors : ```console Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "omsagent" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config ``` ```console Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: Secret "omsagent-secret" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config ``` ----- ```sh az version ``` ```console { "azure-cli": "2.22.1", "azure-cli-core": "2.22.1", "azure-cli-telemetry": "1.0.6", "extensions": { "connectedk8s": "1.1.3", "connectedmachine": "0.3.0", "k8s-configuration": "1.0.0", "k8s-extension": "0.2.1", "spring-cloud": "2.3.0" } } ``` I di test it on a K3S cluster deployed on an Azure VM workaround : I had to manually clean-up the environment : ```sh az k8s-extension delete --name azuremonitor-containers --cluster-type connectedClusters --cluster-name $azure_arc_k3s -g $k3s_rg_name -y k get sa -n kube-system k delete sa omsagent -n kube-system k get sa -n kube-system k get secrets -n kube-system k delete secret omsagent-secret -n kube-system k get cm -n kube-system k delete cm omsagent-rs-config -n kube-system k get cm -n kube-system k delete deploy omsagent-rs -n kube-system k get crd k delete crd healthstates.azmon.container.insights curl -o disable-monitoring.sh -L https://aka.ms/disable-monitoring-bash-script bash disable-monitoring.sh --resource-id $azureArc_K3S_ClusterResourceId --kube-context $kubeContext ``` Then recreate : ```sh az monitor log-analytics workspace create -n $analytics_workspace_name --location $location -g $k3s_rg_name --verbose export analytics_workspace_id=$(az monitor log-analytics workspace show -n $analytics_workspace_name -g $k3s_rg_name -o tsv --query id) echo "analytics_workspace_id:" $analytics_workspace_id az k8s-extension create --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings logAnalyticsWorkspaceResourceID=$analytics_workspace_id omsagent.resources.daemonset.limits.cpu=150m omsagent.resources.daemonset.limits.memory=600Mi omsagent.resources.deployment.limits.cpu=1 omsagent.resources.deployment.limits.memory=750Mi az k8s-extension list --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters azmon_extension_state=$(az k8s-extension show --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --query 'installState') echo "Azure Monitor extension state: " $azmon_extension_state ```
Author: ezYakaEagle442
Assignees: -
Labels: `Kubernetes Configuration`, `Service Attention`, `needs-triage`
Milestone: -
yungezz commented 3 years ago

route to appropriate team