Failed to update stack due to deny assignment

[slavizh]

Describe the bug I have deployed a stack and it was successful. After trying to apply the stack again I get failure due to deny assignment. I have not changed anything in the stack's configuration. In both times I am using the same account (outlook account, not sure if that is the issue).

To Reproduce Below there is correlation ID in order to investigate what is happening. I have redacted some information. If you need some other information I can send it in e-mail. I have done some tests before with the same template and account and never got this problem.

PS D:\dev\deployment-stacks\lz-role-assignments> New-AzManagementGroupDeploymentStack -Name lz-role-assignments -ManagementGroupId Sponsor -Location 'West Europe' -DeleteAll -DenySettingsMode DenyWriteAndDelete -Force -TemplateFile .\main.bicep -TemplateParameterFile .\main.parameters.json -Verbose -OutVariable h -DeploymentSubscriptionId <sub id>
VERBOSE: Using Bicep v0.20.4
WARNING: D:\dev\deployment-stacks\lz-role-assignments\main.bicep(5,7) : Warning no-unused-params: Parameter "deploymentLocation" is declared but never used. [https://aka.ms/bicep/linter/no-unused-params]
D:\dev\deployment-stacks\lz-role-assignments\modules\role-assignment.bicep(14,28) : Warning BCP081: Resource type "Microsoft.Authorization/roleAssignments@2021-04-01-preview" does not have types available.
VERBOSE: Performing the operation "Create" on target "lz-role-assignments".
VERBOSE: 10:47:08 - Checking stack deployment status
VERBOSE: 10:47:13 - Starting DeploymentOperations polling
VERBOSE: 10:47:13 - Checking deployment status in 5 seconds
VERBOSE: 10:47:18 - DeploymentOperations polling failed
VERBOSE: 10:47:18 - Checking stack deployment status
VERBOSE: 10:47:23 - Checking stack deployment status
VERBOSE: 10:47:28 - Checking stack deployment status
New-AzManagementGroupDeploymentStack: 10:47:34 - The deployment 'lz-role-assignments' failed with error(s). Showing 3 out of 3 error(s).
Error: Code=DeploymentStackDeploymentFailed; Message=One or more resources could not be deployed. Correlation id: '8817c0cf-d42a-4626-8ee0-47562db3c080'.

Error: Code=DeploymentFailed; Message=At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.

Error: Code=DenyAssignmentAuthorizationFailed; Message=The client '<outlook account upn>' with object id '7d484d2d-c809-4896-ba1e-eb0fbafab022' has permission to perform action 'Microsoft.Authorization/roleAssignments/write' on scope '/subscriptions/<sub id>/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f'; however, the access is denied because of the deny assignment with name 'Deny assignment 'b842fe57-7da6-50f5-8bdb-7674c7d774f9' created by Deployment Stack '/providers/Microsoft.Management/managementGroups/Sponsor/providers/Microsoft.Resources/deploymentStacks/lz-role-assignments'.' and Id 'b842fe577da650f58bdb7674c7d774f9' at scope '/subscriptions/<sub id>/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f'.

WARNING: The cmdlet is in preview and under development.

Id                          : /providers/Microsoft.Management/managementGroups/Sponsor/providers/Microsoft.Resources/deploymentStacks/lz-role-assignments
Name                        : lz-role-assignments
ProvisioningState           : failed
ResourcesCleanupAction      : delete
ResourceGroupsCleanupAction : delete
DenySettingsMode            : denyWriteAndDelete
Location                    : westeurope
CreationTime(UTC)           : 22.8.2023 г. 7:29:55
DeploymentId                : /subscriptions/<sub id>/providers/Microsoft.Resources/deployments/lz-role-assignments-2023-08-22-07-47-09-ae555
Resources                   : /subscriptions/<sub id>/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f
FailedResources             : {
                                id: /subscriptions/<sub id>/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f
                                error: The client '<outlook account upn>' with object id '7d484d2d-c809-4896-ba1e-eb0fbafab022' has permission to perform action 'Microsoft.Authorization/roleAssignments/wri                              te' on scope '/subscriptions/<sub id>/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f'; however, the access is
                              denied because of the deny assignment with name 'Deny assignment 'b842fe57-7da6-50f5-8bdb-7674c7d774f9' created by Deployment Stack '/providers/Microsoft.Management/managementGro
                              ups/Sponsor/providers/Microsoft.Resources/deploymentStacks/lz-role-assignments'.' and Id 'b842fe577da650f58bdb7674c7d774f9' at scope '/subscriptions/a47e8b4b-b9d9-4927-bf68-e607d
                              fdef18b/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f'.
                              }
Error                       : One or more resources could not be deployed. Correlation id: '8817c0cf-d42a-4626-8ee0-47562db3c080'. (Code: DeploymentStackDeploymentFailed)
                               - At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. (Code:
                               DeploymentFailed)
                                 - The client '<outlook account upn>' with object id '7d484d2d-c809-4896-ba1e-eb0fbafab022' has permission to perform action 'Microsoft.Authorization/roleAssignments/write'
                               on scope '/subscriptions/<sub id>/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f'; however, the access is den
                              ied because of the deny assignment with name 'Deny assignment 'b842fe57-7da6-50f5-8bdb-7674c7d774f9' created by Deployment Stack '/providers/Microsoft.Management/managementGroups
                              /Sponsor/providers/Microsoft.Resources/deploymentStacks/lz-role-assignments'.' and Id 'b842fe577da650f58bdb7674c7d774f9' at scope '/subscriptions/a47e8b4b-b9d9-4927-bf68-e607dfde
                              f18b/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f'. (Code:DenyAssignmentAuthorizationFailed)
Parameters                  :
                              Name                  Type                       Value
                              ====================  =========================  ==========
                              roleAssignments       array                      [{"builtInRoleDefinitionId":"4fe576fe-1146-4730-92eb-48519fa6bf9f","principalId":"69daadcb-0994-4ce6-bef6-01d7857a536e","principa
                              lDisplayName":"Azure AD sync admins"}]
                              solutionVersionTag    object                     {}
                              deploymentLocation    string                     "West Europe"
                              tags                  object                     {}

Repro Environment Host OS: Windows 11 Powershell Version: 7.3.6

[dantedallag]

Hey @slavizh, thanks for creating this issue. Could you email me the template used?

[slavizh]

sent.

[sqlkabouter]

@slavizh you're deploying the stack with the option -DenySettingsMode DenyWriteAndDelete but you're not specifying any exclusions to these settings. You could try adding your own principal to -DenySettingsExcludedPrincipal.

I did notice when I has this issue myself that I had to run the deployment twice before it worked.

[slavizh]

@sqlkabouter it should not be the case. If you have permissions on the stack you should be able to manage it without having to exclude the account you run it with.

[slavizh]

@dantedallag any update?

[dantedallag]

@slavizh Still investigating. I'm able to reproduce, but it seems to be something that is only a problem with role assignments. While we continue to look into it, let us know if you have faced the same issue with any other resource type.

[slavizh]

@dantedallag ok, great. Being able to reproduce it is good and that is only scoped to role assignments.

[azcloudfarmer]

HI @slavizh - we have a fix for this rolling out. We will confirm and close this issue once complete.

[azcloudfarmer]

Hi @slavizh - quick update, the fix for this issue is currently being rolled out to all regions. We will confirm once all regions are deployed.

[azcloudfarmer]

Hi @slavizh - the fix for this has been deployed. Can you please confirm before we close the issue? Thanks!

[slavizh]

@azcloudfarmer I am still experiencing the issue. Location West Europe.

New-AzSubscriptionDeploymentStack: 13:30:19 - The deployment 'lz-role-assignments' failed with error(s). Showing 3 out of 3 error(s).
Error: Code=DeploymentStackDeploymentFailed; Message=One or more resources could not be deployed. Correlation id: '9e42c3a4-9f06-411c-ae0a-65abbd46e878'.

Error: Code=DeploymentFailed; Message=At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.

Error: Code=InvalidTemplateDeployment; Message=The template deployment failed with error: 'Deny assignment check failed for template resource 'bbc05547-14fd-5292-87b0-09c6812fad7f' of type 'Microsoft.Authorization/roleAssignments'. The client '<upn>' with object id '<GUID>' has the permission to perform action 'Microsoft.Authorization/roleAssignments/write' at scope '/subscriptions/<Sub ID>/providers/Microsoft.Authorization/roleAssignments/bbc05547-14fd-5292-87b0-09c6812fad7f' but is blocked by deny assignment.'.

[slavizh]

@azcloudfarmer nevermind. turned out that it requires the latest Az.Resources version and I was using an old one.