search

GSA / data.gov

Main repository for the data.gov service
https://data.gov
Other
667 stars 104 forks source link

Cleanup SES instances #3511

Closed jbrown-xentity closed 3 years ago

jbrown-xentity commented 3 years ago

We're getting the following emails (11 as of 10/29). These are related to test instances in SES that did not get cleaned up. We need to figure out how to remove them, and document for future use cases.

We have been attempting to verify the DKIM setup of ses-394c1167fe0582ed.ssb-dev.data.gov for the last 3 days. We have not been able to detect the required DNS records in your DNS settings. If you still wish to use DKIM when sending through Amazon SES or Amazon Pinpoint, please confirm that the DNS records are present and retry the set-up process via the Amazon SES or Amazon Pinpoint console or the API.

For DKIM troubleshooting information, see http://docs.aws.amazon.com/ses/latest/DeveloperGuide/DKIM-problems.html .

Please note that this email only relates to the US West (Oregon) region.

How to reproduce

  1. Create a test case that doesn't get destroyed
  2. Wait 3+ days
  3. Receive email

Expected behavior

All test cases are destroyed correctly

Actual behavior

Test cases persist on the AWS account

Sketch

This is the code for our side of that... As noted, AWS is supposed to do a bunch of this on their own now. go look for SES instances in the us-west-2 region in the AWS console, and see if you can tie them back to where they came from based on creation time, tags, etc. And nuke the ones that don't seem to have corresponding records. And if it's hard to figure out where they come from, then the action item is to make sure the tags get applied by the broker.... the CSB passes them in, but we may need to add those tags on the resources we create. For example, in the EKS brokerpak, here's how the tags are applied to the clusters that get created. Here's the bit about how to tag AWS resources in Terraform generally... I suspect you can tag SES identities and Route53 zones, but not individual records. For EKS we added this document, we should probably do the same thing here.

jbrown-xentity commented 3 years ago

We are now fairly certain all the SES records are getting cleaned up properly; unfortunately something is causing DKIM to not be created properly (and even though the instances are destroyed, SES still sends an email about it). The above PR should fix it, but the Terraform doesn't want to build. @mogul will investigate along with the Terraform 0.13 upgrade...

jbrown-xentity commented 3 years ago

Actually our latest test somehow caused DKIM to succeed (see email), so something must be working...