[ENHANCEMENT] Database snapshotting logic in Circle CI configuration is complex

[eternaltyro]

Current snapshot approach in the CI pipeline

Currently, the Database snapshot logic for TM4 PostgreSQL is complex.

• We get RDS instance ID by using a script to filter existing instances by tag key aws:cloudformation:stack-name
• We then get existing snapshots for the instance ID output from the above script to fetch the latest snapshot by filtering for a deterministic snapshot name
• If a snapshot with the name *-latest does exist, then we rename it and create a new snapshot with the same name.
• Finally, during cleanup stage, we delete the old snapshot identified by a custom name.

Simplifying the snapshot process

The rough approach to simplifying and streamlining this process.

• Remove copy-db-snapshot, delete-db-snapshot, and all other logic that attempts find the age of the snapshot.
• Snapshot lifecycle management is relegated to another tool or process
• Create a new job that describes the snapshot process create-db-snapshot
• Alter workflows to include the snapshot job in production, staging, teachosm and other critical workflows as a required step on which other jobs such as backend-deploy depend
• Setup monitoring for snapshot failures.

Old snapshot cleanup

If absolutely necessary, we can cleanup old snapshots by fetching snapshots older than "now date" at run time of every production deployment.

    --query "DBSnapshots[?(SnapshotCreateTime<='2023-03-05')].[DBSnapshotIdentifier]"

Ideally, this should be relegated to AWS Backup or a scheduled script.

[dakotabenjamin]

I am considering using this orb: https://circleci.com/developer/orbs/orb/ware2go/aws-rds

[eternaltyro]

Yes, the ORB is MIT licensed too. We have to assess compatibility with OIDC though, since the orb does not seem to take AWS_SESSION_TOKEN as one of the parameters which is required for temporary credentials.

What are your thoughts @dakotabenjamin on removing cleanup steps and snapshot deletion from the CI config altogether?