search

mooyoul / aws-cdk-ses-domain-identity

Constructs for provisioning and referencing domain identities which can be used in SES RuleSets and Actions Construct.
MIT License
37 stars 13 forks source link

Does not delete cleanly #6

Closed scientific-giraffe closed 3 years ago

scientific-giraffe commented 3 years ago

When deleting the stack, the construct needs to delete its records

The specified hosted zone contains non-required resource record sets and so cannot be deleted. (Service: Route53, Status Code: 400, Request ID: bcd48189-1ac3-4ea3-8976-a264ff567e44, Extended Request ID: null)
mooyoul commented 3 years ago

Interesting. Which records are kept? I mean, Handler will remove Route53 Record also if needed.

cyberwombat commented 3 years ago

Same issue here. I have to manually delete [name='_amazonses.MYDOMAUN.', type='TXT'].

mooyoul commented 3 years ago

@cyberwombat Thanks for the information. That's weird. The Route53 Record that you mentioned is used for SES Domain Verification. It should be deleted on resource removal. Can you share your logs of handler function? You can find the logs from CloudWatch Logs Console. Log Group name contains "DomainIdentityRequestorFunction".

cyberwombat commented 3 years ago

Just manually removed stack, deployed it again and tried to destroy it.

Terminal log:

Are you sure you want to delete: SESStack (y/n)? y
SESStack: destroying...
9:32:18 PM | DELETE_FAILED        | AWS::CloudFormation::CustomResource | DomainIdentityIden...orResource3CDC595E
Received response status [FAILED] from custom resource. Message returned: Invalid request: Expected exactly one of [AliasTarget
9:32:18 PM | DELETE_FAILED        | AWS::CloudFormation::CustomResource | DomainIdentity/Ide...orResource/Default
Received response status [FAILED] from custom resource. Message returned: Invalid request: Expected exactly one of [AliasTarget
, all of [TTL, and ResourceRecords], or TrafficPolicyInstanceId], but found none in Change with [Action=DELETE, Name=dcxzz4iv7b
hysma6tlf4qm25fbkxpruy._domainkey.dev.example.com, Type=CNAME, SetIdentifier=null] (RequestId: af398e41-a632-4b24-8978-ab04
43d77e70)
9:32:24 PM | DELETE_FAILED        | AWS::CloudFormation::Stack          | SESStackThe following resource(s) failed to delete: [DomainIdentityIdentityRequestorResource3CDC595E].
 ❌  SESStack: destroy failed Error: The stack named SESStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [DomainIdentityIdentityRequestorResource3CDC595E]. )
    at Object.waitForStackDelete (/node_modules/aws-cdk/lib/api/util/cloudformation.ts:277:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at Object.destroyStack (/node_modules/aws-cdk/lib/api/deploy-stack.ts:395:28)
    at CdkToolkit.destroy (/node_modules/aws-cdk/lib/cdk-toolkit.ts:253:9)
    at initCommandLine (/node_modules/aws-cdk/bin/cdk.ts:208:9)
The stack named SESStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [DomainIdentityIdentityRequestorResource3CDC595E]. )

Lambda log:

2021-04-12T21:28:04.016-07:00   START RequestId: 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 Version: $LATEST

2021-04-12T21:28:04.190-07:00

Copy
2021-04-13T04:28:04.189Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Verifying Domain for dev.example.com
2021-04-13T04:28:04.189Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Verifying Domain for dev.example.com

2021-04-12T21:28:04.988-07:00

Copy
2021-04-13T04:28:04.988Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Creating a TXT record for verifying domain into zone Z0339125T9XHDLVBK5AJ
2021-04-13T04:28:04.988Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Creating a TXT record for verifying domain into zone Z0339125T9XHDLVBK5AJ

2021-04-12T21:28:05.347-07:00

Copy
2021-04-13T04:28:05.347Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Waiting for DNS records to commit...
2021-04-13T04:28:05.347Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Waiting for DNS records to commit...

2021-04-12T21:28:36.027-07:00

Copy
2021-04-13T04:28:36.014Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Waiting for domain verification...
2021-04-13T04:28:36.014Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Waiting for domain verification...

2021-04-12T21:29:06.227-07:00

Copy
2021-04-13T04:29:06.227Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Enabling DKIM for dev.example.com
2021-04-13T04:29:06.227Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Enabling DKIM for dev.example.com

2021-04-12T21:29:06.409-07:00

Copy
2021-04-13T04:29:06.409Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Creating 3 DNS records for verifying DKIM into zone Z0339125T9XHDLVBK5AJ
2021-04-13T04:29:06.409Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Creating 3 DNS records for verifying DKIM into zone Z0339125T9XHDLVBK5AJ

2021-04-12T21:29:06.610-07:00

Copy
2021-04-13T04:29:06.610Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Waiting for DNS records to commit...
2021-04-13T04:29:06.610Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Waiting for DNS records to commit...

2021-04-12T21:29:37.263-07:00

Copy
2021-04-13T04:29:37.263Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Waiting for DKIM verification...
2021-04-13T04:29:37.263Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Waiting for DKIM verification...

2021-04-12T21:29:37.548-07:00

Copy
2021-04-13T04:29:37.492Z    928901d0-f2ee-4a43-aec6-18aa01eb0ec6    INFO    Notifying success response...
2021-04-13T04:29:37.492Z 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 INFO Notifying success response...

2021-04-12T21:29:37.889-07:00

Copy
END RequestId: 928901d0-f2ee-4a43-aec6-18aa01eb0ec6
END RequestId: 928901d0-f2ee-4a43-aec6-18aa01eb0ec6

2021-04-12T21:29:37.889-07:00

Copy
REPORT RequestId: 928901d0-f2ee-4a43-aec6-18aa01eb0ec6  Duration: 93870.43 ms   Billed Duration: 93871 ms   Memory Size: 128 MB Max Memory Used: 84 MB  Init Duration: 233.46 ms    
REPORT RequestId: 928901d0-f2ee-4a43-aec6-18aa01eb0ec6 Duration: 93870.43 ms Billed Duration: 93871 ms Memory Size: 128 MB Max Memory Used: 84 MB Init Duration: 233.46 ms

2021-04-12T21:32:17.231-07:00

Copy
START RequestId: c6f93f62-5984-4180-b346-7053d6d72bce Version: $LATEST
START RequestId: c6f93f62-5984-4180-b346-7053d6d72bce Version: $LATEST

2021-04-12T21:32:17.269-07:00

Copy
2021-04-13T04:32:17.250Z    c6f93f62-5984-4180-b346-7053d6d72bce    INFO    Getting current DKIM state for domain dev.example.com
2021-04-13T04:32:17.250Z c6f93f62-5984-4180-b346-7053d6d72bce INFO Getting current DKIM state for domain dev.example.com

2021-04-12T21:32:17.545-07:00

Copy
2021-04-13T04:32:17.545Z    c6f93f62-5984-4180-b346-7053d6d72bce    INFO    Disabling DKIM for domain dev.example.com
2021-04-13T04:32:17.545Z c6f93f62-5984-4180-b346-7053d6d72bce INFO Disabling DKIM for domain dev.example.com

2021-04-12T21:32:17.683-07:00

Copy
2021-04-13T04:32:17.683Z    c6f93f62-5984-4180-b346-7053d6d72bce    INFO    Deleting DNS Records used for DKIM verification...
2021-04-13T04:32:17.683Z c6f93f62-5984-4180-b346-7053d6d72bce INFO Deleting DNS Records used for DKIM verification...

2021-04-12T21:32:17.772-07:00

Copy
2021-04-13T04:32:17.771Z    c6f93f62-5984-4180-b346-7053d6d72bce    ERROR   Failed to provision resource! InvalidInput: Invalid request: Expected exactly one of [AliasTarget, all of [TTL, and ResourceRecords], or TrafficPolicyInstanceId], but found none in Change with [Action=DELETE, Name=dcxzz4iv7bhysma6tlf4qm25fbkxpruy._domainkey.dev.example.com, Type=CNAME, SetIdentifier=null]
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_xml.js:53:29)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
2021-04-13T04:32:17.771Z c6f93f62-5984-4180-b346-7053d6d72bce ERROR Failed to provision resource! InvalidInput: Invalid request: Expected exactly one of [AliasTarget, all of [TTL, and ResourceRecords], or TrafficPolicyInstanceId], but found none in Change with [Action=DELETE, Name=dcxzz4iv7bhysma6tlf4qm25fbkxpruy._domainkey.dev.example.com, Type=CNAME, SetIdentifier=null] at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_xml.js:53:29) at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:688:14) at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12) at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:690:12) at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

2021-04-12T21:32:17.908-07:00

Copy
END RequestId: c6f93f62-5984-4180-b346-7053d6d72bce
END RequestId: c6f93f62-5984-4180-b346-7053d6d72bce

2021-04-12T21:32:17.908-07:00

Copy
REPORT RequestId: c6f93f62-5984-4180-b346-7053d6d72bce  Duration: 674.35 ms Billed Duration: 675 ms Memory Size: 128 MB Max Memory Used: 85 MB  
REPORT RequestId: c6f93f62-5984-4180-b346-7053d6d72bce Duration: 674.35 ms Billed Duration: 675 ms Memory Size: 128 MB Max Memory Used: 85 MB

Stack code:

 const hostedZone = HostedZone.fromLookup(this, 'HostedZone', {
      domainName,
      privateZone: false
    })

    const identity = new DnsValidatedDomainIdentity(this, 'DomainIdentity', {
      domainName,
      dkim: true,
      region: this.region,
      hostedZone
    })

I don't know if this is relevant but my setup is a sub account of main account which holds the naked domain. In the sub account Route53 has a test subdomain dev.example.com. I doubt it is an issue as it creates the stack fine and works fine. It looks like perhaps the dns deletion isn't the main issue - but a leftover from another issue which causes stack destroy to fail.

cyberwombat commented 3 years ago

I went to delete stack from console and this helpful message showed up:

This stack previously failed to delete because the following resources failed to delete. If you choose to retain resources, they will be skipped during this delete operation.

Resources to retain - optional
Selected resources will be skipped during the delete stack operation

DomainIdentityIdentityRequestorResource3CDC595E
dev.example.com
mooyoul commented 3 years ago

Thanks! Now I can tell that is definitely a bug. Route53 Record removal was failed due to invalid parameter. I didn't know that removing route53 record also requires ResourceRecords property. I omitted this parameter so it results to stack removal failure. Will fix this issue within this week.

mooyoul commented 3 years ago

This issue was fixed in v1.0.5 - Specially thanks to @Garethp for providing the fix!