Stack Set fails after accounts became suspended

[hitty5]

Describe the bug Stack set operations fails due to terminated / suspended accounts.

To Reproduce Assuming following organization:

Root
- DEV
-- Account 1 (governed with control tower)

Following manifest:

---
region: eu-central-1
version: 2021-03-15

resources:
  - name: rules
    deploy_method: stack_set
    resource_file: templates/rules.template
    deployment_targets:
      organizational_units:
        - DEV
    regions:
      - eu-central-1

Now the account 'Account 1' gets terminated, meaning the accounts is removed from control tower (=the corresponding provisioned product get deleted). This action moves the account from the origin OU to the root OU and deletes all control tower specific resources, e.g. AWSControlTowerExection role, so the stack set cannot perform any operation anymore, which lets the stack set fail.

Expected behavior The stack set detects the control tower termination of the account and removes the corresponding stack instance, like it would be with auto-deployment mode. Alternatively, for terminated accounts the stack instance could be removed with the options "--retain-stack" in case the required roles are not there anymore.

What would be right order to terminate an AWS account in combination with CT customization framework? E.g.

1. move account to SUSPENDED OU
2. run CT customizations (so the stack instance get removed)
3. terminate CT for account
4. close account

?

Please complete the following i nformation about the solution:

• [ ] Version 2.1.0
• [ ] Region: eu-central-1
• [ ] Was the solution modified from the version published on this repository? NO
• [ ] If the answer to the previous question was yes, are the changes available on GitHub?
• [ ] Have you checked your service quotas for the sevices this solution uses? YES
• [ ] Were there any errors in the CloudWatch Logs? NO

[rmsilva1973]

Perhaps feature request #90 might address this issue somewhat

[rakshb]

@hitty5 Hello, we have added this issue to our backlog.

[dlahn]

Is there a workaround for this? We have some closed accounts and our pipeline is failing because of it.

[hitty5]

@hitty5 Hello, we have added this issue to our backlog.

@rakshb You need to eliminate the "AWS Control Tower" provisioned product in service catalog before you close an account, so the accounts gets dropped from the organization and the pipeline is not recognizing suspended account anymore. Hence, suspended account are not touched by the pipeline. In your case I guess you need to reopen the closed account and remove the from service catalog.

[dlahn]

Is there any other way? We realised this a bit late, and we have some accounts that were closed many months ago which can't be recovered.

[hitty5]

Since AWS control tower customizations uses AWS stack sets you can try to delete the stack instances of the affected accounts manually.