search

jckuester / awsweeper

A tool for cleaning your AWS account
Mozilla Public License 2.0
466 stars 46 forks source link

Unable to delete Cloudformation stacks #106

Open andreamaruccia opened 4 years ago

andreamaruccia commented 4 years ago

I've this config:

aws_cloudformation_stack:
  - id: awsweeper-del-me

running it I get these logs:

awsweeper --region eu-west-1 --parallel 1 --debug --force config.yaml

   • downloaded and installed provider                  name=aws version=2.59.0
   • configured provider                                name=aws version=2.59.0
   • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)
<snipping>
      • failed to import resource; trying to read resource without import error=Throttling: Rate exceeded
        status code: 400, request id: 18399ea7-5742-409a-9e62-14cf0a14cb51 id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: dca88124-cc18-4375-8aaf-39bcd0bbd6dd
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: cc2e00e1-1554-4bd3-9ca9-f235fd738e37
Error: failed to read current state of resource: Throttling: Rate exceeded
        status code: 400, request id: f78f05c8-a354-4465-92d1-5e1099bae9b5
<snipping>
      • failed to get tags        error=state is nil: <nil> id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
<snipping>
        ---
        Type: aws_cloudformation_stack
        Found: 1

                Id:             arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
                Created:        2020-06-29 09:40:07.952 +0000 UTC

        ---

   • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 1
   • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
      • unable to delete resource                          error=resource state is nil; need to call update first resource_id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack
   • TOTAL NUMBER OF DELETED RESOURCES: 0

and I can still see the cfn stack in the aws console :(

I've tried it with version 0.7 and it seems to work:

 awsweeper --region eu-west-1 --debug --force config.yaml        
   • downloaded and installed provider                  name=aws version=2.43.0
   • configured provider                                name=aws version=2.43.0
   • using region: eu-west-1  
   • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)

        ---
        Type: aws_cloudformation_stack
        Found: 1

                Id:             arn:aws:cloudformation:eu-west-1:<redacted>/:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
        Created:        2020-06-29 09:40:07.952 +0000 UTC

        ---

   • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 1
   • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
      • worker starts deleting resource                    id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72 type=aws_cloudformation_stack worker_id=5
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/awsweeper-del-me/80789cf0-b9ec-11ea-8cc3-02aed67d8a72
   • TOTAL NUMBER OF DELETED RESOURCES: 1
jckuester commented 4 years ago

Hi @andreamaruccia, I've also noticed the throttling exception that appears somehow if the AWS API is called too often. I added some retry logic that should fix the problem you are seeing: https://github.com/jckuester/awsweeper/releases/tag/v0.10.1

And yep, in 0.7 it worked because I didn't use the Terraform AWS Provider back then to support so many resources.

andreamaruccia commented 4 years ago

hi @jckuester it seems this helps already as I can delete cloudformation stacks that I've created recently. But when it comes to delete an older one, it seems that it doesn't delete it somehow.

This is what works well:

1) create a cfn stack with this yaml and name it awsweeper-test

Description: >
  Stack deployment test for awsweeper

Parameters:
  ManagedPolicyName:
    Type: String
    Default: default-managed-policy-name

Resources:
  noCloudTrail:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Sub "${ManagedPolicyName}-${AWS::Region}"
      Description: desc
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: MoleculeTest
            Effect: Deny
            Action:
              - cloudtrail:StopLogging
            Resource: "*"
Outputs:
  ExampleOutput:
    Description: This is an example output
    Value: !Ref noCloudTrail

use this configuration:

aws_cloudformation_stack:
  - id: awsweeper-test

This doesn't: 1) Take an old cloudformation template named old-rotten-poc created on 2019-04-18 14:56:06 UTC+0200 2) use this configuration:

aws_cloudformation_stack:
  - id: old-rotten-poc

I wonder if this is due to pagination or such now that the rate limiting is out of the way

jckuester commented 4 years ago

Is there any useful debug output of awsweeper for old-rotten-poc?

andreamaruccia commented 4 years ago

that's the weired thing it just says it's being deleted:

    • downloaded and installed provider                  name=aws version=2.68.0
    • configured provider                                name=aws version=2.68.0
    • SHOWING RESOURCES THAT WOULD BE DELETED (DRY RUN)
       • retrying to read current state of resource error=Throttling: Rate exceeded
    status code: 400, request id: 11699a25-0a7e-48b2-8bef-135e11ba896b
       • retrying to read current state of resource error=Throttling: Rate exceeded
    status code: 400, request id: a73faa95-4a90-41d9-957e-e283627b0f7e
<snip>
       • retrying to read current state of resource error=Throttling: Rate exceeded
    status code: 400, request id: dbe3094a-7635-457e-8069-524b87829467
       • failed to import resource; trying to read resource without import error=Throttling: Rate exceeded
    status code: 400, request id: dbe3094a-7635-457e-8069-524b87829467 id=arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stack-name>/57d1c6c0-1da0-11ea-be65-0ace5b4cd800 type=aws_cloudformation_stack
    • TOTAL NUMBER OF RESOURCES THAT WOULD BE DELETED: 14
    • USER WILL NOT BE ASKED FOR CONFIRMATION (FORCE MODE)
    • STARTING TO DELETE RESOURCES
       • start distributing resources to workers for this run
    ---
    Type: aws_cloudformation_stack
    Found: 14
        Id:     arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/6261ff90-9e85-11ea-998c-0224adcab7c8
        Created:    2020-05-25 12:43:50.775 +0000 UTC
        Id:     arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d7b48310-99a0-11ea-ac4f-0a0e6fcb5aa0
        Created:    2020-05-19 07:17:48.315 +0000 UTC
        Id:     arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d7ae6890-99a0-11ea-9143-0a0af0d573f8
        Created:    2020-05-19 07:17:48.186 +0000 UTC
        Id:     arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d7a95f80-99a0-11ea-8769-02fa7ab4594c
        Created:    2020-05-19 07:17:48.155 +0000 UTC
        Id:     arn:aws:cloudformation:eu-west-1:<redacted>:stack/<redacted-stackname>/d1bd4280-99a0-11ea-8474-0ae0d664c880
<snip>
    ---
       ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:473444336670:stack/<redacted-stackname>/3348cbf0-6dd2-11ea-9d48-0a9bdcf5c20a
       ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:473444336670:stack/<redacted-stackname>/6261ff90-9e85-11ea-998c-0224adcab7c8
       ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:473444336670:stack/<redacted-stackname>/d7a95f80-99a0-11ea-8769-02fa7ab4594c
<snip>
    • TOTAL NUMBER OF DELETED RESOURCES: 14
jckuester commented 4 years ago

@andreamaruccia, hmm, hard to say from here what's wrong. Is it just a single resource that doesn't get deleted or many? If you run awsweeper again, the resource(s) still stay?

andreamaruccia commented 4 years ago

@jckuester all the resources remain, not a single one was deleted (14 in my case) Is there a way to increase the log level somehow on the resource deletion commands?

edit: yes I ran it 2/3 times in a row, and also at night (in a schedule)

jckuester commented 4 years ago

I created a test myself and deleting of aws_cloudformation_stack resources works fine. But you said it just doesn't work with old stacks, right? Deletion also doesn't work with 0.7 then, I assume?

Is there a way to increase the log level somehow on the resource deletion commands?

Not officially, but I can make a code change to enable TRACE output of the Terraform AWS provider, which performs the destroy. I created a branch for you where I enabled TRACE output: debug/terraform-provider-trace-output

Are you able to checkout that branch and go build it or do you want me to send you a binary?

andreamaruccia commented 4 years ago

I created a test myself and deleting of aws_cloudformation_stack resources works fine. But you said it just doesn't work with old stacks, right? Deletion also doesn't work with 0.7 then, I assume?

Yes I confirm 0.7 doesn't work either. My test with 0.7 was deleting the newly created test resource

Are you able to checkout that branch and go build it or do you want me to send you a binary?

yes I will do it, thanks for enabling me to test it

andreamaruccia commented 4 years ago

Here are the logs with trace:

<snip>
        Enter a value: YES
   • STARTING TO DELETE RESOURCES
      • start distributing resources to workers for this run
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/f275c070-0a0c-11ea-a4cc-0a7ffa9bc424
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d7a95f80-99a0-11ea-8769-02fa7ab4594c
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/3891f230-6dd2-11ea-9f24-06e58f87e324
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/62fb7cd0-0a0e-11ea-b12b-0aa024806772
2020/07/02 11:36:20 [DEBUG] Waiting for state to become: [success]
2020/07/02 11:36:20 [TRACE] GRPCProvider: ApplyResourceChange
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/38b3ab00-6dd2-11ea-b898-0aac29486886
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/3348cbf0-6dd2-11ea-9d48-0a9bdcf5c20a
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d7b48310-99a0-11ea-ac4f-0a0e6fcb5aa0
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d1bd4280-99a0-11ea-8474-0ae0d664c880
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>-S3Stack-XY1NP9J0ZM1E/6261ff90-9e85-11ea-998c-0224adcab7c8
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>-SQSStack-W5K1OU0J2JJ3/d7ae6890-99a0-11ea-9143-0a0af0d573f8
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/bd4b7460-ca3f-11e9-8977-069eaf5ad9a0
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d48eb560-ca44-11e9-be29-0677a5c9174c
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/d7692af0-ca57-11e9-b63b-0a4044c481ac
      ⨯ aws_cloudformation_stack                           id=arn:aws:cloudformation:eu-west-1:<redacted-aws-account-id>:stack/<redacted-stack-name>/54209610-61d9-11e9-88f7-0a0c5b3ce4f4
   • TOTAL NUMBER OF DELETED RESOURCES: 14

I don't see anything super bad :/

jckuester commented 4 years ago

Yeah, nothing bad really to see in the TRACE logs. If Terraform says nothing, the resource should actually be deleted. Don't know what's going on there, honestly. Can you manually delete the resources in the AWS console?

andreamaruccia commented 4 years ago

@jckuester I solved the mistery, these resource have termination protection on this is why it wouldn't delete them. By removing it it was able to delete it.

I wonder what the behavior should be in this case to be honest. I can see two options to improve this: 1) Add a log when there is a protection to make it clear that it wont be deleted 2) Add a flag to remove the protection before deleting (like a force)

Anyway I am already happy to understand this

jckuester commented 4 years ago

I am glad that you could find the issue, @andreamaruccia. Let's keep this issue open, so that we don't forget about it until I have found a way to deal with deletion protection.

andreamaruccia commented 4 years ago

thank you @jckuester, sounds good

I am glad that you could find the issue, @andreamaruccia. Let's keep this issue open, so that we don't forget about it until I have found a way to deal with deletion protection.

thank you @jckuester, sounds good