search

Azure / deployment-stacks

Contains Deployment Stacks CLI scripts and releases
MIT License
87 stars 6 forks source link

Referenced resource gets added by mistake and afterwards deleted #81

Closed slavizh closed 1 year ago

slavizh commented 1 year ago

Describe the bug I think I have found a serious bug. I have a template that deploys a few Log Alerts. From the parameters file you specify which exact alerts to be deployed and other options for the alerts. From the parameters you can also define if the alert rules will have action groups associated. You can also define if the alert rules are scoped to subscription, resource group or Log Analytics workspace. The action groups, resource group and log analytics workspace are just defined as reference resources (existing) which means they are not deployed via this solution. Next I will describe the steps to reproduce the issue where the Log Analytics workspace is added as managed resource to the deployment stack and afterwards deployment stacks tries to delete it.

To Reproduce Steps to reproduce the behavior:

  1. First I do initial deployment where I have defined one alert rule. I have scoped that alert rule to Log Analytics workspace (let's say Log1). The alert rule also has association to action group. Deployment goes fine.
  2. Next I am changing the parameters file so the Log Analytics workspace reference is different one (for example Log2). I do deployment with that configuration but it fails due to not being able to change the scope of existing alert rule. This is ARM error of the resource and that is fine. When that fails you can see the new Log Analytics workspace (Log2) being added to the managed resource.
  3. Next I remove the previously defined rule and add a new one. That way the previous alert rule should be deleted and the new one should be added. I am not changing the referenced Log Analytics workspace (still Log2). This deployment fails not because it is not possible. The actual deployment it goes fine but the problem is that the deployment stacks is trying to remove the Log Analytics workspace (Log2) as I have mentioned in step 2 it was added to managed resource and now it seems it is removed as managed resource. The reason where there is failure is because I happen to have lock on the Log Analytics workspace (Log2) thus deployment stack is trying to delete it until it times out.

Below is output from the commands executed. This seems like serious bug where referenced resources gets added to managed resources of the stack for no reason. Names and GUIDs were redacted.

PS D:\dev\deployment-stacks\lz-analysis-services-monitoring> New-AzSubscriptionDeploymentStack -Name lz-analysis-services-monitoring -TemplateFile .\main.bicep -TemplateParameterFile .\main.parameters.json -Location 'West Europe' -OutVariable p -Verbose -Force -DeleteAll
VERBOSE: Using Bicep v0.13.1
VERBOSE: Performing the operation "Create" on target "lz-analysis-services-monitoring".
VERBOSE: 12:38:09 - Checking stack deployment status
VERBOSE: 12:38:14 - Checking stack deployment status
VERBOSE: 12:38:19 - Checking stack deployment status
VERBOSE: 12:38:24 - Checking stack deployment status
VERBOSE: 12:38:29 - Checking stack deployment status
VERBOSE: 12:38:34 - Checking stack deployment status
VERBOSE: 12:38:39 - Checking stack deployment status
VERBOSE: 12:38:45 - Checking stack deployment status
VERBOSE: 12:38:50 - Checking stack deployment status

Id                          : /subscriptions/<subscription id>/providers/Microsoft.Resources/deploymentStacks/lz-analysis-services-monitoring
Name                        : lz-analysis-services-monitoring
ProvisioningState           : succeeded
Parameters                  : {Name: resourceGroups, Type: array, Value: {
                                "name": "test-analysisserv-rg",
                                "location": "West Europe",
                                "tags": {
                                  "purpose": "mon"
                                },
                                "alertsScope": {
                                  "logAnalyticsWorkspace": {
                                    "name": "log1",
                                    "resourceGroup": "KustoTest"
                                  }
                                },
                                "alertRules": {
                                  "memoryThrashing": {
                                    "deploy": true,
                                    "frequencyInMinutes": 5,
                                    "timeWindowInMinutes": 5,
                                    "description": "test1"
                                  }
                                }
                              }}
ResourcesCleanupAction      : delete
ResourceGroupsCleanupAction : delete
DenySettingsMode            : none
Location                    : westeurope
CreationTime(UTC)           : 20.1.2023 г. 8:12:02
DeploymentId                : /subscriptions/<subscription id>/providers/Microsoft.Resources/deployments/lz-analysis-services-monitoring-2023-01-20-10-38-13-12003
Outputs                     : {Name: test, Type: String, Value: test}
Resources                   : /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg
                              /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg/providers/Microsoft.Insights/scheduledQueryRules/2c995c3f-8e42-4eaf-82c2-80b9d443b2dd

PS D:\dev\deployment-stacks\lz-analysis-services-monitoring> New-AzSubscriptionDeploymentStack -Name lz-analysis-services-monitoring -TemplateFile .\main.bicep -TemplateParameterFile .\main.parameters.json -Location 'West Europe' -OutVariable p -Verbose -Force -DeleteAll
VERBOSE: Using Bicep v0.13.1
VERBOSE: Performing the operation "Create" on target "lz-analysis-services-monitoring".
VERBOSE: 12:40:48 - Checking stack deployment status
VERBOSE: 12:40:53 - Checking stack deployment status
VERBOSE: 12:40:59 - Checking stack deployment status
VERBOSE: 12:41:04 - Checking stack deployment status
VERBOSE: 12:41:09 - Checking stack deployment status
VERBOSE: 12:41:14 - Checking stack deployment status
VERBOSE: 12:41:19 - Checking stack deployment status
VERBOSE: 12:41:24 - Checking stack deployment status
VERBOSE: 12:41:29 - Checking stack deployment status
New-AzSubscriptionDeploymentStack: 12:41:35 - The deployment 'lz-analysis-services-monitoring' failed with error(s). Showing 3 out of 3 error(s).
Error: Code=DeploymentStackUpdateFailed; Message=One or more stages of the deploymentStack failed. Correlation id: '1019f295-3127-457b-910c-083420ca133b'

Error: Code=DeploymentFailed; Message=At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.

Error: Code=DeploymentFailed; Message=At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.

Id                          : /subscriptions/<subscription id>/providers/Microsoft.Resources/deploymentStacks/lz-analysis-services-monitoring
Name                        : lz-analysis-services-monitoring
ProvisioningState           : failed
Parameters                  : {Name: resourceGroups, Type: array, Value: {
                                "name": "test-analysisserv-rg",
                                "location": "West Europe",
                                "tags": {
                                  "purpose": "mon"
                                },
                                "alertsScope": {
                                  "logAnalyticsWorkspace": {
                                    "name": "log2",
                                    "resourceGroup": "OMS"
                                  }
                                },
                                "alertRules": {
                                  "memoryThrashing": {
                                    "deploy": true,
                                    "frequencyInMinutes": 5,
                                    "timeWindowInMinutes": 5,
                                    "description": "test1"
                                  }
                                }
                              }}
ResourcesCleanupAction      : delete
ResourceGroupsCleanupAction : delete
DenySettingsMode            : none
Location                    : westeurope
CreationTime(UTC)           : 20.1.2023 г. 8:12:02
DeploymentId                : /subscriptions/<subscription id>/providers/Microsoft.Resources/deployments/lz-analysis-services-monitoring-2023-01-20-10-40-53-82215
Resources                   : /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg
                              /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg/providers/Microsoft.Insights/scheduledQueryRules/2c995c3f-8e42-4eaf-82c2-80b9d443b2dd
                              /subscriptions/<subscription id>/resourceGroups/OMS/providers/Microsoft.OperationalInsights/workspaces/log2
FailedResources             : {
                                id: /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg/providers/Microsoft.Insights/scheduledQueryRules/2c995c3f-8e42-4eaf-82c2-80b9d443b2dd
                                error: Scope can not be updated
                              }
Error                       : DeploymentFailed - At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.

PS D:\dev\deployment-stacks\lz-analysis-services-monitoring> New-AzSubscriptionDeploymentStack -Name lz-analysis-services-monitoring -TemplateFile .\main.bicep -TemplateParameterFile .\main.parameters.json -Location 'West Europe' -OutVariable p -Verbose -Force -DeleteAll
VERBOSE: Using Bicep v0.13.1
VERBOSE: Performing the operation "Create" on target "lz-analysis-services-monitoring".
VERBOSE: 12:43:48 - Checking stack deployment status
VERBOSE: 12:43:54 - Checking stack deployment status
VERBOSE: 12:43:59 - Checking stack deployment status
VERBOSE: 12:44:04 - Checking stack deployment status
VERBOSE: 12:44:09 - Checking stack deployment status
VERBOSE: 12:44:14 - Checking stack deployment status
VERBOSE: 12:44:19 - Checking stack deployment status
VERBOSE: 12:44:25 - Checking stack deployment status
VERBOSE: 12:44:30 - Checking stack deployment status
VERBOSE: 12:44:35 - Checking stack deployment status
VERBOSE: 12:44:40 - Checking stack deployment status
VERBOSE: 12:44:45 - Checking stack deployment status
VERBOSE: 12:44:50 - Checking stack deployment status
VERBOSE: 12:44:55 - Checking stack deployment status
VERBOSE: 12:45:01 - Checking stack deployment status
VERBOSE: 12:45:06 - Checking stack deployment status
VERBOSE: 12:45:11 - Checking stack deployment status
VERBOSE: 12:45:16 - Checking stack deployment status
VERBOSE: 12:45:21 - Checking stack deployment status
VERBOSE: 12:45:26 - Checking stack deployment status
VERBOSE: 12:45:31 - Checking stack deployment status
VERBOSE: 12:45:37 - Checking stack deployment status
VERBOSE: 12:45:42 - Checking stack deployment status
VERBOSE: 12:45:47 - Checking stack deployment status
VERBOSE: 12:45:52 - Checking stack deployment status
VERBOSE: 12:45:57 - Checking stack deployment status
New-AzSubscriptionDeploymentStack: 13:45:55 - The deployment 'lz-analysis-services-monitoring' failed with error(s). Showing 3 out of 3 error(s).
Error: Code=DeploymentStackUpdateFailed; Message=One or more stages of the deploymentStack failed. Correlation id: 'e5a7c849-d70b-4f90-8a81-590693583d9c'

Error: Code=DeploymentStackDeleteResourcesFailed; Message=One or more resources could not be deleted.

Error: Code=DeploymentStackDeleteResourcesFailed; Message=An unknown error occurred while trying to delete resources. These resources are still present in the stack but can be deleted manually.

Id                          : /subscriptions/<subscription id>/providers/Microsoft.Resources/deploymentStacks/lz-analysis-services-monitoring
Name                        : lz-analysis-services-monitoring
ProvisioningState           : failed
Parameters                  : {Name: resourceGroups, Type: array, Value: {
                                "name": "test-analysisserv-rg",
                                "location": "West Europe",
                                "tags": {
                                  "purpose": "mon"
                                },
                                "alertsScope": {
                                  "logAnalyticsWorkspace": {
                                    "name": "log2",
                                    "resourceGroup": "OMS"
                                  }
                                },
                                "alertRules": {
                                  "totalConnectionFailures": {
                                    "deploy": true,
                                    "frequencyInMinutes": 5,
                                    "timeWindowInMinutes": 5,
                                    "description": "test1"
                                  }
                                }
                              }}
ResourcesCleanupAction      : delete
ResourceGroupsCleanupAction : delete
DenySettingsMode            : none
Location                    : westeurope
CreationTime(UTC)           : 20.1.2023 г. 8:12:02
DeploymentId                : /subscriptions/<subscription id>/providers/Microsoft.Resources/deployments/lz-analysis-services-monitoring-2023-01-20-10-43-53-64e8b
Outputs                     : {Name: test, Type: String, Value: test}
Resources                   : /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg
                              /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg/providers/Microsoft.Insights/scheduledQueryRules/f9125df9-eb5d-4967-8517-52c68f6f9dd2
                              /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg/providers/Microsoft.Insights/scheduledQueryRules/2c995c3f-8e42-4eaf-82c2-80b9d443b2dd
                              /subscriptions/<subscription id>/resourceGroups/OMS/providers/Microsoft.OperationalInsights/workspaces/log2
FailedResources             : {
                                id: /subscriptions/<subscription id>/resourceGroups/test-analysisserv-rg/providers/Microsoft.Insights/scheduledQueryRules/2c995c3f-8e42-4eaf-82c2-80b9d443b2dd
                                error: Resource could not be deleted. Resource is still present in stack.
                              },
                              {
                                id: /subscriptions/<subscription id>/resourceGroups/OMS/providers/Microsoft.OperationalInsights/workspaces/log2
                                error: Resource could not be deleted. Resource is still present in stack.
                              }
Error                       : DeploymentStackDeleteResourcesFailed - An unknown error occurred while trying to delete resources. These resources are still present in the stack but can be deleted manually.

If needed I can provide the actual template in private. configuration you can see via input.

Expected behavior Not to put referenced resources into managed

Screenshots

Repro Environment Host OS: Windows 11 Powershell Version: 7.3.1

Server Debugging Information Correlation ID: 5641c573-fb56-4c3f-8299-5c4e71930f67, 1019f295-3127-457b-910c-083420ca133b, e5a7c849-d70b-4f90-8a81-590693583d9c Tenant ID: I can provide in private Timestamp of issue (please include time zone): see above, Easter Europe Time zone Data Center (eg, West Central US, West Europe): West Europe

Additional context Add any other context about the problem here.

snarkywolverine commented 1 year ago

We have found the issue for item 2 in this issue and are in the process of releasing the fix.