search

aws-solutions / aws-control-tower-customizations

The Customizations for AWS Control Tower solution combines AWS Control Tower and other highly-available, trusted AWS services to help customers more quickly set up a secure, multi-account AWS environment using AWS best practices.
https://docs.aws.amazon.com/controltower/latest/userguide/cfct-overview.html
Apache License 2.0
356 stars 205 forks source link

CfCT pipeline fails if it discovers existing resources when applying baseline(s) during AWS Account enrollment in Control Tower #77

Open ggAWS opened 2 years ago

ggAWS commented 2 years ago

Once we deploy Control Tower and CfCT solution, every time we enroll an OU, it starts enrolling the existing accounts with Control Tower. However, this invokes the CfCT pipeline and it will start applying the Baseline resources mentioned to the accounts being enrolled.

Customers would like to have these baselines for the new accounts as well which were already deployed to accounts pre CT deployment. But if you enroll the existing accounts in CT tower, CfCT tries to apply the same baselines again which results in conflicting resources and the pipeline fails. Below is an example.

Suppose an existing account has an IAM role which is a critical resource that cannot be modified or deleted. Also, you want the same role with the same name to be created in the future accounts that you are going to create via Control Tower and apply this role as a baseline via CfCT. Now, when to try to enroll an existing account in CT, the CfCT pipeline will discover that this role already exists in the account and will fail with the Conflicting resource error.

Hence, there needs to be a way for CfCT pipeline to skip the creation of these existing resources and deploy only the resources that are not present in the accounts.

rlwalker927 commented 2 years ago

+1 for this issue

premroxx commented 2 years ago

+1 A good alternative would've to provide options to ignore accounts within an OU. Say if OU has 3 accounts acc1,acc2,acc3 and you want to apply the existing baselines to new account, it would be good to ignore acc1/2 and apply to all other accounts

groverlalit commented 2 years ago

Thanks for bringing this to our attention.

Issue recap: The stack set deployment fails due to the conflicting resources in the existing accounts. Proposed solution: Provide a mechanism to exclude certain accounts in the OU to avoid this conflict.

Questions for clarification:

  1. How did we deploy the resources in the existing accounts? Stack Sets? Stacks? AWS Console?
  2. The stack set template changes will not update the stack instances in the excluded accounts. Wouldn't that add operational debt to maintain the resources in the excluded accounts?
ggAWS commented 2 years ago
  1. How did we deploy the resources in the existing accounts? Stack Sets? Stacks? AWS Console? -- I have tested it with Stacks and Stacksets.
  2. The stack set template changes will not update the stack instances in the excluded accounts. Wouldn't that add operational debt to maintain the resources in the excluded accounts? -- It would. However, if the existing resources are imperative for the workloads and customer also wants to move to CT and use CfCT, this poses as a blocker in CT adoption. We can apply workarounds as of now or build custom resources to check for existing resources but that is also an operational overhead.
ggAWS commented 2 years ago
  1. How did we deploy the resources in the existing accounts? Stack Sets? Stacks? AWS Console? -- I have tested it with Stacks and Stacksets.
  2. The stack set template changes will not update the stack instances in the excluded accounts. Wouldn't that add operational debt to maintain the resources in the excluded accounts? -- It would. However, if the existing resources are imperative for the workloads and customer also wants to move to CT and use CfCT, this poses as a blocker in CT adoption. We can apply workarounds as of now or build custom resources to check for existing resources but that is also an operational overhead.
premroxx commented 2 years ago

The scenario we come across in ProServe quite often, is that Enterprises already have CFN/CDK baselines they have developed over years. These are already applied to the existing accounts. During the CT and CFCT adoption, the biggest roadblock becomes, how do we bring in these existing accounts under CFCT. Because if we just add it at an OU level (which is ideal for automation of new accounts). the stacksets will most definitely fail since these resources/templates have already been deployed there. While in certain cases it is possible to delete the existing Stack/Stackset without breaking the existing Infra, it still requires considerable effort at an Org level. Having an option to exclude certain accounts i think might help 1) The existing baselines can be left alone, if needed 2) It gives the Customer some breathing room to bring over the existing accounts/baselines into CFCT in a phased manner, where they can delete existing Stacks/Stacksets and edit the manifest accordingly 3) If the existing baseline needs to be updated it provides additional opportunities to switch over pre-existing accounts to CFCT.

groverlalit commented 2 years ago

Appreciate the feedback. We have added this to our backlog.