Azure / deployment-stacks

Contains Deployment Stacks CLI scripts and releases
MIT License
89 stars 7 forks source link

denyWriteAndDelete Deployment Stack does not apply deny assignment to resource group child scopes. #165

Closed dridderhof closed 3 months ago

dridderhof commented 4 months ago

Describe the bug

We have a Deployment Stack on Subscription Scope that contains a Resource Group and a Managed Identity, the stack is set to denyWriteAndDelete and Apply to child scopes set to true. We have excluded a SPN as principal, however this issue occured with another account that is in scope.

Previously this blocked the deployment of any new resource into this Resource Group (within the scope of the Deployment Stack), but we noticed that this is now succeeding. This is not what we expect, it should block the deployment to this RG.

The deleting of the Managed Identity is still blocked, as should be.

To Reproduce Steps to reproduce the behavior:

  1. Go to the RG in a deployment stack
  2. Click on 'Create' and deploy a resource, e.g. a Key Vault.
  3. Deployment succeeds

Expected behavior Deployment of any resources in the child scope of the managed RG should be denied.

Screenshots

Repro Environment

Server Debugging Information None.

Additional context Worked previously.

dantedallag commented 4 months ago

@dridderhof We can look into this. Do you have a correlation id for the stack PUT? I want to make sure that the deny assignments are in fact getting deployed correctly.

snarkywolverine commented 4 months ago

@dridderhof Do you have the resource ID of a stack that blocked RG deployments? As I recall, we have not supported a denyWrite permission on resource groups.

dridderhof commented 4 months ago

@dridderhof We can look into this. Do you have a correlation id for the stack PUT? I want to make sure that the deny assignments are in fact getting deployed correctly.

I think this should be the correlation Id: de3f478e-8724-4dac-b5f8-372b29497009

@snarkywolverine It's a stack on a subscription including a RG, I've added a screenshot below:

deploymentstack

Are you able to locate the resource ID from the information in the correlation Id? Otherwise please send me a private message or e-mail. I'd rather not share the resource ID publicly.

snarkywolverine commented 4 months ago

Feel free to reach out to us at ARMDeploymentStacks at microsoft dot com to share additional details.

I looked for that correlation ID but did not see any record of it in the last 14 days - should I be looking earlier? EDIT - I just saw the screenshot says "6 Mai" and can see the stack information.

Also, just to clarify... Dante was looking for a correlation ID for a deployment stack where the stack was deployed and allowed an RG deployment to succeed, while I was looking for a deployment stack that successfully blocked deployments to a protected resource group. Both data points would be really valuable for us to debug. EDIT: I assume the Correlation ID provided above (de3f...) is for the first scenario, rather than the second - is that correct?

dridderhof commented 4 months ago

Feel free to reach out to us at ARMDeploymentStacks at microsoft dot com to share additional details.

I looked for that correlation ID but did not see any record of it in the last 14 days - should I be looking earlier?

Also, just to clarify... Dante was looking for a correlation ID for a deployment stack where the stack was deployed and allowed an RG deployment to succeed, while I was looking for a deployment stack that successfully blocked deployments to a protected resource group. Both data points would be really valuable for us to debug.

Last update was from 6 May, so longer than 14 days ago that the deployment stack was updated. Just to be sure I have created a new stack, the correlation id for that is: b1bf0383-a89e-40d5-9e76-c26355a6fecc

Created a Key Vault in this stack, that succeeded. Correlation Id for that is: 5b62c9c9-49ab-435e-9c55-d9ed00cdda2b. This should have been blocked.

I am not able to locate a correlation Id for a blocked deployment? It seems to stop me already from a deployment if it is not allowed. Tried that via cli as through the portal, and is as expected blocked:

{"code": "InvalidTemplateDeployment", "message": "Deployment failed with multiple errors: 'Deny assignment check failed for template resource 'support-test' of type 'Microsoft.Resources/resourceGroups'. The client 'D.Ridderhof@xxx' with object id '3e10' has the permission to perform action 'Microsoft.Resources/subscriptions/resourceGroups/write' at scope '/subscriptions/xxxxxxxxxx/resourceGroups/support-test' but is blocked by deny assignment.:Deny assignment check failed for template resource 'zm-mi-d-engineering-development-001' of type 'Microsoft.ManagedIdentity/userAssignedIdentities'. The client 'D.Ridderhof@xxx' with object id '3e10' has the permission to perform action 'Microsoft.ManagedIdentity/userAssignedIdentities/write' at scope '/subscriptions/xxxxxxxxxx/resourceGroups/support-test/providers/Microsoft.ManagedIdentity/userAssignedIdentities/zm-mi-d-engineering-development-001' but is blocked by deny assignment.'"}

dridderhof commented 4 months ago

@snarkywolverine, @dantedallag Do you already have an update on this?

dantedallag commented 4 months ago

Hi @dridderhof,

Sorry for the delay. I've looked in this more and want to clarify a couple of things.

Though we lock RGs with either DenyDelete or DenyWriteAndDelete, we never apply DenySettingsApplyToChildScopes property to the deny assignment associated with the RG, even if it is specified in the stack request. There may be a documentation gap here that we can address. This is a design decision we made a while back and one that should have been in place for at least the last preview. Initially, we were bypassing RG deny assignments altogether, because of some complications, but settled on locking them without ever locking child scopes.

The last error you posted was an error associated with a write to the rg itself, which is explicitly protected by a deny assignment, and the managed identity, which is also explicitly protected by a deny assignment, so it makes sense that this is blocked. I'm curious to know if you have examples of deployments external to the stack failing to deploy a new resource to the rg associated and locked by your stack. Also, do you remember how long ago it was that these types of block were happening?

In the end though, our intended design does not allow locking of rg child scopes. We are happy to work with you on your solution with this constraint and are also wiling to revisit our design here if there is enough interest from the community.

dridderhof commented 4 months ago

Hi @dantedallag

Thank you for clarifying. Could you elaborate a bit more on the intended use of DenySettingsApplyToChildScopes? We wanted to accomplish the management (by a central team) of the whole RG, and that no one else was able to deploy resources under this scope.

We initially tested this in the beginning of January this year.

dridderhof commented 3 months ago

@dantedallag Any update?

azcloudfarmer commented 3 months ago

Hello @dridderhof - apologies for the delay. Some answers below:

--Deny-settings-apply-to-child-scopes: When specified, the deny setting mode configuration also applies the child scope of the managed resources. For example, a main.bicep file defines a Microsoft.Sql/servers resource (parent) and a Microsoft.Sql/servers/databases resource (child). If a deployment stack is created using the Bicep file with the deny-settings-apply-to-child-scopes setting enabled and the deny-settings-mode set to denyWriteAndDelete, you can't add any additional child resources to either the Microsoft.Sql/servers resource or the Microsoft.Sql/servers/databases resources created by the Bicep file, from outside of the deployment stack.

Please note that DenySettingsMode does not apply to Data plane operations. In some cases, there may be resources that execute certain commands via the data plane, and Deployment Stacks deny settings are not applicable. For example, Key Vaults and Storage Accounts can be created in the control plane, therefore can be managed by a Deployment Stack, however, some of their child resources such as Secrets or Blob Containers are created via the data plane.

Please also note that DenySettingsMode also does not apply to Implicitly created resources. In some cases, there may be resources created implicitly as a result of creating another. For example, and AKS cluster spins off multiple other services to support it, as does a Virtual Machine. In these cases, given those implicitly created resources are not defined in the template, they are not subject to the DenySettingsMode of the DeploymentStack.

We noticed these gaps in our docs and are working on rolling out an update this week. Thanks for catching this!

dridderhof commented 3 months ago

@azcloudfarmer Thanks for the clarification!

What I do not understand yet is that we manage the Resource Group as the parent, I thought that anything deployed in it is threated as child. For example /subscriptions/xxxxxxxxxx/resourceGroups/zm-p-rg-acrp-managed-001 and that any resource in it will be seen as child, e.g. /subscriptions/xxxxxxxxxx/resourcegroups/zm-p-rg-acrp-managed-001/providers/Microsoft.ManagedIdentity/userAssignedIdentities.

Is this scenario not supported?

snarkywolverine commented 3 months ago

As @dantedallag mentioned:

Though we lock RGs with either DenyDelete or DenyWriteAndDelete, we never apply DenySettingsApplyToChildScopes property to the deny assignment associated with the RG, even if it is specified in the stack request.

From our perspective, the goal is to protect individual resources with deny assignments, rather than entire resource groups. This design also empowers users to deploy resources to a resource group using multiple templates (since, if ApplyToChildScopes applied to the resource group, then all resources in a resource group would need to be defined in a single stack).