bcgov / cas-cif

CleanBC Industry Fund project
Apache License 2.0
6 stars 2 forks source link

As a developer, I want to use Vault to back-up openshift secrets, so that I don't lose them in the event of a cluster failure or mistake #1683

Open dleard opened 1 year ago

dleard commented 1 year ago

We should not only rely on openshift to always have our secrets. In the event of a catastrophic cluster failure there is potential to lose secrets and not be able to recover them. We should explore the use of Vault to back-up our secrets to mitigate this risk. https://docs.developer.gov.bc.ca/vault-secrets-management-service/

Acceptance Criteria

Given I am a CIF developer, When I consult the CIF documentation Then I can see which secrets are stored only on the cluster And need to be backed up

Given I am a CIF developer Then I have access to a mechanism to backup the application secrets And that mechanism is triggered daily at a minimum

Developer Checklist:

pbastia commented 1 year ago

@dleard It might be helpful to know which secrets that are stored on the cluster are currently not recoverable? Maybe we could start a list to see if vault is indeed the best way to store them, or if we should automate the provisioning of these secrets with something like terraform

My step 1 would be to evaluate which secrets are not backed up and what is the risk of losing them, so we can prioritize this correctly

dleard commented 1 year ago

Yeah, good idea. I know several off the top of my head that are randomly generated on install and only saved in OCP

pbastia commented 1 year ago

@dleard @Sepehr-Sobhani @sam-warren this would probably be applicable to the data-warehouse project as well

mikevespi commented 11 months ago

@dleard / @andrea-williams I am fully onboard for backing these things up! Given the problems Hashicorp cloud has given us, do we want to consider an alternative to Vault?

andrea-williams commented 11 months ago

100% agree with storing a backup somewhere (or multiple somewheres). However, given my dislike of devops, I'm not familiar with any alternatives to Vault because I blindfold myself to that kind of thing

andrea-williams commented 11 months ago

but I can ask around in the DDS devs online world for recommendations if you want