Cameronsplaze / AWS-ContainerManager

CDK Architecture to spin up containers when you connect, and back down afterwards automatically.
2 stars 0 forks source link

[Bug] Route53 Hosted Zone cannot be deleted if system is on (The specified hosted zone contains non-required resource record sets and so cannot be deleted) #7

Open Cameronsplaze opened 3 months ago

Cameronsplaze commented 3 months ago

Describe the bug If the system is ON when you go to delete the stack, the Route53 DNS record is not the same as when it got created. (Ip is the public instance ip, instead of 0.0.0.0). CDK thinks its a custom resource you added, instead of something it created, and refuses to delete it.

DELETE_FAILED AWS::Route53::HostedZone
The specified hosted zone contains non-required resource record sets and so cannot be deleted.

QUICK FIX: Just delete the record with the instance IP from the console, then re-run the make cdk-destroy-leaf ... command again. (You'll notice the ip ISN'T 0.0.0.0, and that's why this happened).

To Reproduce Steps to reproduce the behavior:

  1. On an existing stack, try to connect to your container and wait for the instance to spin up
  2. Before it can spin down, delete the stack
  3. Profit

Expected behavior The stack should be able to be destroyed, regardless of what state it's in.

Additional context There's a few routes we can go to fix this:

1) Have the makefile spin down the ASG before deleting the stack. It's easiest, but feels hacky.

- Originally I did this with `aws cli` commands in the makefile like so:

  ```bash
  cdk-destroy-leaf: guard-config-file
  echo "Config File: $(config-file)"
  base_stack_name=`python3 -c "import app; print(app.base_stack_name)"`
  # Get the container ID from the config file:
  container_id=`python3 -c "import app; print(app.get_container_id('$(config-file)'))"`
  # Get the ASG Name from the Container ID:
  asg_name=$$(aws autoscaling describe-auto-scaling-groups \
    --filters "Name=tag:ContainerNameID,Values=Valheim-example" \
    --query 'AutoScalingGroups[0].AutoScalingGroupName' \
    --output text)
  # Set the desired capacity to 0:
  aws autoscaling set-desired-capacity \
    --auto-scaling-group-name $${asg_name} \
    --desired-capacity 0 \
    --honor-cooldown
  ```

  But there's no way to wait for the desired-capacity to finish that I can find.
- The other option is to move the logic into a python script, and use boto3 calls. This is tempting, but the file would have to live in the root of the project, and the makefile would probably have to use env-vars to pass in the config path to the script. Hence the hackyness of this idea.

2) Use CDK CustomResources to either delete the Route53 record, or spin down the ASG, if a delete is called on the entire stack. (Not sure if spinning down the ASG is possible, but deleting Route53 records definitely is). This does leave yet another lambda in the account per leaf stack, but is a lot more automatic than the other solution.

- CDK Custom Delete Example [here](https://medium.com/cyberark-engineering/advanced-custom-resources-with-aws-cdk-1e024d4fb2fa)
- AWS Docs [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html) (Not the greatest)