Upgrade dev/test Traction instances to use CrunchyDB

bcgov / DITP-DevOps

Digital Identity and Trust Program Team's DevOps Documentation Repository

Apache License 2.0

2 stars 6 forks source link

Upgrade dev/test Traction instances to use CrunchyDB #133

Closed esune closed 1 year ago

esune commented 1 year ago

Upgrade dev/test deployments of Traction to use CrunchyDB rather than standalone PostgreSQL as database provider.

Pre-Requisites:

[x] New chart released for Traction, supporting CrunchyDB

Requirements:

[x] Innkeeper and Tenant data in the environment(s) is retained and transferred over to CrunchyDB
[x] Upgrade and validate dev
[x] Upgrade and validate test
[x] CrunchyDB supports database backups
[x] Configurations updated and stored in https://github.com/bcgov/trust-over-ip-configurations

i5okie commented 1 year ago

It's a good practice to keep production data backed up off-site, or to multiple destinations. CrunchyDB's pgBackRest support configuring multiple repos for backup storage. By default, a PVC will be used to store the backup in dev, test, and prod.

@WadeBarnes Should we setup prod with backups going to AWS S3 or BCGov's Dell EMC appliance as well?

WadeBarnes commented 1 year ago

I'm assuming the backups will be saved to their own PVC. The PVC should be provisioned as netapp-file-backup which is described here; https://github.com/BCDevOps/backup-container#backup-storage-volume, which writes to off site (cluster) backup storage automatically. Platform services will soon be recommending S3 storage for backup retention. This will be supported by BC Gov S3 buckets. @jleach should be able to provide additional details regarding the move to S3 storage.

WadeBarnes commented 1 year ago

For the backups, is there an option for notification of success/failure? The backup-container notifies us of all backup success/failure in the #ditp-backups channel. I'd like to see the same for the CrunchyDB backups.

i5okie commented 1 year ago

I can look into it.

i5okie commented 1 year ago

Deployed traction-database CrunchyDB HA cluster in dev
- Cluster is configured with 2 instances of Postgresql
- Backups are configured to keep 7 days worth of continuous backup (point-in-time restore possible)
- Backup repo PVC is created with netapp-file-backup storage class
Traction dev has been migrated to use traction-database cluster

esune commented 1 year ago

Excellent work @i5okie . We need to plan for the upgrade of test now. In the meantime, would you mind deploying the production instance (#132), since we are not worries of losing data there?

i5okie commented 1 year ago

Traction prod has been deployed on October 23rd.

i5okie commented 1 year ago

Traction test has been upgraded to use CrunchyDB. PR with updates to the deployment values: https://github.com/bcgov/trust-over-ip-configurations/pull/171