Investigate how to leverage Velero for Dev

[Jose-Matsuda]

Hard initial deadline of end of day Tuesday July 4th https://cloudnative.pages.cloud.statcan.ca/en/documentation/best-practices-meilleures-pratiques/how-to-guide/velero.namespace.restore/

• The other part about this task being related to kubeflow is that if we wanted to use velero we'd also need auto-sync to be turned off so it doesn't see a difference in desired vs live manifest right away and then undo the velero change.

Will close this after discussing results at tech elab.

[Jose-Matsuda]

Jose notes We'd probably want a on-demand backup that backs up the entire cluster state

Questions to answer:

1) Where do backups get saved? Where did CNS setup Velero to send backups to?

• In our velero pod we do use the plugin for microsoft azure.
• Can we get access to the RG that would allow us to see the velero storage accounts? 2) Our cluster has velero-hourly-resources backups, what is in those?
  • Looking at the spec from one made on 2023-06-30 we have the following spec where we seem to backup most things except volumes.
  • I feel like for a kubeflow upgrade which won't touch storage this hourly backup would be sufficient.
  • Maybe for jfrog if we're particular about losing data it would be more important. 3) How do I restore from a backup file?

Things to try in dev

1) Create a backup of a single namespace with resources in it in dev. 2) Delete that namespace and all resources kubectl delete all --all -n jose-test-delete 3) Attempt to restore it. velero restore create restore-name --from {name}

• get the name from the velero.io/v1/backups crd
• Note that restores are non-destructive, it won't overwrite data if it exists in cluster. 4) ...

Other Info

There are restore hooks but I don't think we need these as I dont believe there's anything special. Let's also not mess with any backup storage locations

[Jose-Matsuda]

Technical Elaboration Questions and Notes (WIP)

This is meant to be a more consumable version of my above comment, where I ramble and care less about quality.

Talking points / Questions / What would we the general development team want to know?

1. What are we currently backing up?

   • Given the spec of one of the backup resources we can determine that we backup resources under all namespaces that are not volumes.

2. How often are we backing up?

   • Hourly

3. Where is it stored?

   • kubectl describe pod velero-85db44fb6c-9kc2q -n velero-system shows that we use velero-plugin-for-microsoft-azure:v1.5.0 and uses azure credentials (also here in the terraform creating velero with the velero repo here our highest level config here).
   • Can also see the BackupStorageLocation in velero-system named cluster-name with config to an Azure bucket.
   • We might be able to see the files in azure if you have permissions to do so.
   • It is stored in a tar file with a whole bunch of config

4. How can we restore something?

   • velero restore create restore-name --from-backup {name} where restore-name is the name of what you want the restore to be called and {name} is the name of the backup you are referring to (like velero-hourly-resources-20230630120028)

5. How do we manually create a backup?

   • ez pz
   • Note that before actually backing up in the case of a disaster you want to have the Backup be ReadOnly to prevent shenaniganry

Example Run through

Using official docs. Also do not forget the --namespace velero-system flag when running any velero commands Do note that CNS has their own documentation on restoring using velero.

• Have three k9s sessions open to demonstrate. 1 for namespace, 1 for backup, 1 for restore
• Do the one that has a pv and after kubectl applying, before creating the backup make some test files in the persisted directory.
• Then make the backup
• Delete the namespace and let the PV delete itself
• Restore
• Verify the file is still there
• Cleanup

[Jose-Matsuda]

Find out if you can restore just a specific resource. The backup resource will generally have multiple resources (secrets, configmaps etc) but can we restore just secrets?

Note that by default velero won't overwrite

Just use resource filtering