Runtime Data Sync using Spectrum Fusion

[bluzarraga]

Create process for runtime data sync facilitated by Spectrum Fusion

This pr creates template restore CRs and template recipe CRs that can be used to sync CS DB and Zen runtime data across clusters.

There are also several changes to scripts like deploy-br-resources.sh and deployment files to facilitate these processes.

Assumption:

• Working spectrum fusion hub and spoke clusters
• existing backups from hub
• full identical deployment (preferably restored or the original) to one of the backups

Process:

• edit the values of the recipe CR to match expectations (anything within <>)
• apply the recipe CR to the target cluster
• edit the values of the restore CR to match expectations (anything within <>)
• apply the restore CR to the hub cluster
• Wait for restore to complete
• validate restore ran properly
  • check cpfs utils pod in operator namespace, look for relevant cleanup logs in /tmp/logs
  • check cs-db-backup or zen5-backup pods in services namespace, look for logs in cs-db/cs-db-backup/logs or zen5/zen-backup/logs for relevant restore logs (there should only be one)
• validate services are working and data

[bluzarraga]

This PR is ready for review

[qpdpQ]

deploy-br-resources.sh script looks good to me:

allen@allens-mbp schedule % ./deploy-br-resources.sh --operator-ns cs1 --im --zen --util 
[✗] No services namespace specified, using operator namespace cs1 instead. If using SOD topology, please re-run with both --operator-ns and --services-ns defined.
[INFO] Using default storage class rook-cephfs.
[INFO] Creating IM Backup/Restore resources in namespace cs1.
deployment.apps/cs-db-backup created
persistentvolumeclaim/cs-db-backup-pvc created
configmap/cs-db-br-configmap created
role.rbac.authorization.k8s.io/cs-db-backup-role created
rolebinding.rbac.authorization.k8s.io/cs-db-backup-rolebinding created
serviceaccount/cs-db-backup-sa created
[✔] Resources to backup IM deployed in namespace cs1.
[INFO] Creating Zen Backup/Restore resources in namespace cs1.
deployment.apps/zen5-backup configured
persistentvolumeclaim/zen5-backup-pvc unchanged
configmap/zen5-br-configmap unchanged
role.rbac.authorization.k8s.io/zen5-backup-role unchanged
rolebinding.rbac.authorization.k8s.io/zen5-backup-rolebinding configured
serviceaccount/zen5-backup-sa unchanged
[✔] Resources to backup Zen deployed in namespace cs1.
[✔] Backup/Restore resources created in namespace cs1.
[INFO] Creating CPFS Util Backup/Restore resources in namespace cs1.
deployment.apps/cpfs-util configured
role.rbac.authorization.k8s.io/cpfs-util-role unchanged
rolebinding.rbac.authorization.k8s.io/cpfs-util-rolebinding configured
serviceaccount/cpfs-util-sa unchanged
role.rbac.authorization.k8s.io/cpfs-util-services-role unchanged
rolebinding.rbac.authorization.k8s.io/cpfs-util-services-rolebinding configured
configmap/setup-tenant-job-configmap unchanged
persistentvolumeclaim/setup-tenant-job-pvc unchanged
role.rbac.authorization.k8s.io/setup-tenant-job-role configured
rolebinding.rbac.authorization.k8s.io/setup-tenant-job-rolebinding configured
serviceaccount/setup-tenant-job-sa unchanged
role.rbac.authorization.k8s.io/setup-tenant-job-role configured
rolebinding.rbac.authorization.k8s.io/setup-tenant-job-rolebinding configured
job.batch/setup-tenant-job configured
job.batch/setup-tenant-job patched (no change)
[✔] CPFS Util resources deployed in namespace cs1.

allen@allens-mbp schedule % ./deploy-br-resources.sh --operator-ns cs1 --im --zen --util -c
[✗] No services namespace specified, using operator namespace cs1 instead. If using SOD topology, please re-run with both --operator-ns and --services-ns defined.
# Cleaning Backup/Restore resources deployed to namespace cs1.
[INFO] Clean up IM BR resources...
deployment.apps "cs-db-backup" deleted
serviceaccount "cs-db-backup-sa" deleted
role.rbac.authorization.k8s.io "cs-db-backup-role" deleted
rolebinding.rbac.authorization.k8s.io "cs-db-backup-rolebinding" deleted
pod "cs-db-backup-65556cbb4-27d54" deleted
persistentvolumeclaim "cs-db-backup-pvc" deleted
configmap "cs-db-br-configmap" deleted
[✔] IM BR resources cleaned up.
[INFO] Clean up Zen BR resources...
deployment.apps "zen5-backup" deleted
serviceaccount "zen5-backup-sa" deleted
role.rbac.authorization.k8s.io "zen5-backup-role" deleted
rolebinding.rbac.authorization.k8s.io "zen5-backup-rolebinding" deleted
pod "zen5-backup-cf7dd5f95-hzdgp" deleted
persistentvolumeclaim "zen5-backup-pvc" deleted
configmap "zen5-br-configmap" deleted
[✔] Zen BR resources cleaned up.
[INFO] Clean up Utility BR resources...
deployment.apps "cpfs-util" deleted
role.rbac.authorization.k8s.io "cpfs-util-role" deleted
rolebinding.rbac.authorization.k8s.io "cpfs-util-rolebinding" deleted
serviceaccount "cpfs-util-sa" deleted
role.rbac.authorization.k8s.io "cpfs-util-services-role" deleted
rolebinding.rbac.authorization.k8s.io "cpfs-util-services-rolebinding" deleted
configmap "setup-tenant-job-configmap" deleted
role.rbac.authorization.k8s.io "setup-tenant-job-role" deleted
rolebinding.rbac.authorization.k8s.io "setup-tenant-job-rolebinding" deleted
serviceaccount "setup-tenant-job-sa" deleted
job.batch "setup-tenant-job" deleted
persistentvolumeclaim "setup-tenant-job-pvc" deleted
[✔] Utility BR resources cleaned up.
[✔] BR resources succesfully removed from namespace cs1.

[ibm-ci-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bluzarraga, qpdpQ

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/IBM/ibm-common-service-operator/blob/scripts-dev/OWNERS)~~ [bluzarraga,qpdpQ] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment