Test database recovery strategy

[WadeBarnes]

Use the database upgrade task as an opportunity to test and flesh out the database recovery strategy.

Basic steps:

• Create a recent backup
• Upgrade the server to PostgreSQL v10, deleting all previous databases and data.
• Restore the databases and data from backup.
• Test to ensure the application, especially the wallet operations, still function as expected.

[WadeBarnes]

Database recovery support has been added to the backup-container project. During the process I was testing and refining the features to support DR type, as well as, test and migration scenarios (focusing on DR though).

Final testing included (using up to the minute backups):

• Restoring the database to a different location (different database on the same server/host), and inspecting the contents for the expected data.
• Deleting to entire contents of the persistent volume of a wallet database (to mimic a DR scenario) and restoring the data via the scripts, and then ensuring the most recent credentials could still be verified, and additional credentials could be stored without error.
  • Performing the same scenario on the application (TheOrgBook) database.
• Performing the above scenario on both databases at once.
• Performing the above scenario on both databases at once while upgrading from Postgres V9.5 to V10 before restoring.

All testing passed without issue.

[WadeBarnes]

The backup updates have been rolled out to all of the environments in the following namespaces:

• (tmp OrgBook env) devex-bcgov-dac-dev
• (tmp BC Reg Agent env) devex-bcgov-dap-dev
• devex-von-bc-registries-agent-*
• devex-von-bc-tob-*
• devex-von-*
• (ESB Agent) l4izby-*
• ontvon-von-*
• ontvon-von-ont-registries-agent-*