Feature request: ImportExistingResources attribute on update-stacks task

[rene84]

Subject of the issue

I would like to easily refactor stacks, especially move resources from one stack to another that cannot be deleted (either because they contain data or because it will cause downtime)

Expected behavior

One possible implementation would be the following when executing an update-stacks task:

1. see if the stack already exists
2. query the resources in the stack
3. calculate if there are new resources in the template compared to the stack
4. query if those resources happen to already exist
5. create and apply a change-set to import those resources

Since this behavior will greatly slow down the deployment of the stack, it should be enabled with a flag. e.g.

Bastion:
  Type: update-stacks
  DependsOn: 
    - Tables
  Template: bastion.yml
  ImportExistingResources: true
  StackName: 'my-bastion-with-a-ddb-table'
  StackDescription: 'This stack will import an exiting table'
  ...

This will allow me, as per the example, to simply move a resource from the tables.yml to the bastion.yml assuming the DeletionPolicy is set to Retain. Having the flag ImportExistingResources, will only slow down the actual execution, but won't have side-effects if there is nothing to import. After having executed it, one would remove the flag from the task.

A challenge will be that such a change may not create new resources in the same action as importing existing resources as per https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html#resource-import-considerations

[rene84]

Please find the most basic set of templates and shell scripts to create, move and clean-up a LogGroup between two stacks. Assumes account and region is included in the profile that is passed as an argument to the script. Next step is to do this in Typescript.

As an aside: it proves that the resource doesn't have to be defined in exactly the same way in the target template as in the source template, because I modified the value for DeletionPolicy from Retain to Delete

1-setup.sh -> create a source and target stack. Update source stack by removing a resource that then becomes orphaned 2-move.sh -> import the orphaned resource in target stack 3-cleanup.sh -> delete both stacks

import-resources.zip

[rene84]

Let's look at potential implementations per step:

1. This is trivial and already available within org-formation. We can query cloudformation for the stack to see if if already exists. We should also know that based on our own state
2. skip. see 3
3. create a change-set of type UPDATE. The change-set will tell us if the template contains new resources compared to those in the stack. We delete this change-set if set 4 returns no results.
   • Unfortunately the change-set only specifies the resource type and logical name. Not the physicalId obviously (because it doesnt exist yet), but also not the resourceIdentifier. We need to resolve it from the template to find the identifier value. The identifier key can be obtained with aws cloudformation get-template-summary but honestly we can hard-code this as well because it will be a rather static map of resource type to attribute
4. Can we rely on an AWS service to determine if the resource already exists? Otherwise we'll have to implement getters for all resourcetypes that support importing.
   • aws configservice list-discovered-resources? I guess it's not so nice to make using Config a prerequisite to use this feature and there might be situations where the resource is too new to have been picked up by Config.
   • aws resourcegroupstaggingapi get-resources --resource-arn-list will return the resource if it exists. This requires us to calcuate the ARN based on the resourceIdentifier value which may not be trivial
   • aws resourcegroupstaggingapi get-resources --resource-type-filters then iterate through that to match on the resourceIdentifier key and value. Doublecheck if this is also true for resources without tags (cloudformation adds tags as well to most resources) edit: checked and the tags API does NOT return resources that never had tags so this also doesnt work
5. Create a change-set of type IMPORT for those resource that are in the output of both 3 and 4. Execute the change set

(6) continue normal flow

[rene84]

Added PoC code to implement step 2-move.sh in Typescript.

1-setup.sh -> create a source and target stack. Update source stack by removing a resource that then becomes orphaned ts-node 2-move.ts -> import the orphaned resource in target stack 3-cleanup.sh -> delete both stacks

import-resources.zip