Automate the creation/lifecycle management of RDS database instances

hackoregon / civic-devops

Master collection point for issues, procedures, and code to manage the HackOregon Civic platform

MIT License

11 stars 4 forks source link

Automate the creation/lifecycle management of RDS database instances #232

Open MikeTheCanuck opened 5 years ago

MikeTheCanuck commented 5 years ago

Summary

Integrate the manually-created 2019 RDS database instances into our CloudFormation infrastructure-as-code.

Tasks

[x] Figure out how to deploy RDS with CloudFormation
[x] Test management of RDS with CloudFormation to ensure that it keeps existing instances when we upgrade instance classes or assigned storage (so that we don't destroy existing databases)
[ ] Determine if in fact we will be duplicating per-project databases for both Staging and Production purposes
[ ] Generate and deploy the CF templates for each project's Staging databases - migrating databases as required if naming changes
[ ] Generate and deploy the CF templates for each project's new Production databases (with more restrictive requirements)

Definition of Done

All projects' required databases are live and managed by CloudFormation
All projects' databases' connectivity have been validated in required environments (internet or container VPC)

MikeTheCanuck commented 5 years ago

Transportation-2019-staging database instance has been manually configured according to #229

MikeTheCanuck commented 5 years ago

Housing-2019-staging database instance has been manually configured according to #230

MikeTheCanuck commented 5 years ago

Some clues where to find compatible CF YAML templates - I noticed a reference in the old README.md in our hackoregon-aws-infrastructure repo to /infrastructure/rds.yaml, which strongly implies that at least as of March 5, 2017 (when we originally forked from the AWS Labs repo), there was a (semi?) working RDS deployment template that fit into this CF architecture.

So while that file is now gone from the originating repo, I notice 800+ forks from that repo, and found a few promising leads in the forked projects: https://github.com/BrianHGrant/ecs-refarch-cloudformation/blob/master/infrastructure/rds.yaml https://github.com/jnorback/ecs-refarch-cloudformation/blob/ECSandUpv1/db/postgresdb.yaml

MikeTheCanuck commented 4 years ago

Nearly all the CloudFormation code has been integrated, up to the need for SSM parameters to be integrated at runtime.

This article appears to have the key to constructing SSM parameter references: https://aws.amazon.com/blogs/mt/using-aws-systems-manager-parameter-store-secure-string-parameters-in-aws-cloudformation-templates/

The only thing we require in addition is to perform a Join operation or the like to insert a specific Project Name into the parameter name in that {{resolve:... statement.

MikeTheCanuck commented 4 years ago

Validation testing done today:

created new database with RDS template - 20 GiB storage, db.t2.micro instance
manually created table, column and row in the database (to verify that the data doesn't get blown away by subsequent upgrade operations)
updated database to db.t2.small
verified the table and row still exists
updated storage to 50 GiB
verified the table and row still exists

All clear. Happy to see that RDS upgrades like this doesn't brick the underlying data.