Upgrade dev/test Traction instances to use CrunchyDB

bcgov / DITP-DevOps

Digital Identity and Trust Program Team's DevOps Documentation Repository

Apache License 2.0

2 stars 5 forks source link

Upgrade dev/test Traction instances to use CrunchyDB #133

Closed esune closed 10 months ago

esune commented 11 months ago

Upgrade dev/test deployments of Traction to use CrunchyDB rather than standalone PostgreSQL as database provider.

Pre-Requisites:

[x] New chart released for Traction, supporting CrunchyDB

Requirements:

[x] Innkeeper and Tenant data in the environment(s) is retained and transferred over to CrunchyDB
[x] Upgrade and validate dev
[x] Upgrade and validate test
[x] CrunchyDB supports database backups
[x] Configurations updated and stored in https://github.com/bcgov/trust-over-ip-configurations

i5okie commented 11 months ago

It's a good practice to keep production data backed up off-site, or to multiple destinations. CrunchyDB's pgBackRest support configuring multiple repos for backup storage. By default, a PVC will be used to store the backup in dev, test, and prod.

@WadeBarnes Should we setup prod with backups going to AWS S3 or BCGov's Dell EMC appliance as well?

WadeBarnes commented 11 months ago

I'm assuming the backups will be saved to their own PVC. The PVC should be provisioned as netapp-file-backup which is described here; https://github.com/BCDevOps/backup-container#backup-storage-volume, which writes to off site (cluster) backup storage automatically. Platform services will soon be recommending S3 storage for backup retention. This will be supported by BC Gov S3 buckets. @jleach should be able to provide additional details regarding the move to S3 storage.

WadeBarnes commented 11 months ago

For the backups, is there an option for notification of success/failure? The backup-container notifies us of all backup success/failure in the #ditp-backups channel. I'd like to see the same for the CrunchyDB backups.

i5okie commented 11 months ago

I can look into it.

i5okie commented 11 months ago

Deployed traction-database CrunchyDB HA cluster in dev
- Cluster is configured with 2 instances of Postgresql
- Backups are configured to keep 7 days worth of continuous backup (point-in-time restore possible)
- Backup repo PVC is created with netapp-file-backup storage class
Traction dev has been migrated to use traction-database cluster

esune commented 11 months ago

Excellent work @i5okie . We need to plan for the upgrade of test now. In the meantime, would you mind deploying the production instance (#132), since we are not worries of losing data there?

i5okie commented 11 months ago

Traction prod has been deployed on October 23rd.

i5okie commented 10 months ago

Traction test has been upgraded to use CrunchyDB. PR with updates to the deployment values: https://github.com/bcgov/trust-over-ip-configurations/pull/171