search

CCI-MOC / xdmod-cntr

A project to prototype the use of XDMOD with OpenStack and OpenShift on the MOC
1 stars 5 forks source link

Backups for all of the xdmod persistent volumes #86

Open rob-baron opened 1 year ago

rob-baron commented 1 year ago

As it is likely that OpenShift will occasionally need to be reinstalled, we need to back up persistent volumes so that we can bring xdmod back up more quickly.

It was suggested to use an S3 storage on the MOC's ceph cluster as it is not co-located on any of the NERC's systems.

Need to find out if the NERC has storage available for this.

rob-baron commented 1 year ago

On the xdmod side a minimal backup would be (accessible through the xdmod-ui container):

  1) tar -zcvf etc-xdmod.tgz /etc/xdmod
  2) tar -zcvf usr-share-xdmod.tgz /usr/share/xdmod
  3.0) mysqldump -h mariadb -u root -p --databases  > xdmod-db-backup.sql
  3.1) gzip xdmod-db-backup.sql
  4) oc rsync these off of the cluster

When I restored from backup on the ocp-staging environment, I got the interface back with the data.

Notes: 1) /etc/xdmod and /usr/share/xdmod infrequently change 2) the mysqldump command can be done from a dedicated backup pod. Furthermore this pod could also backup the openshift and openstack data PVs 3) it would be nice to shut down the xdmod-ui pod from another pod, mount the PVs back them up and restart it.

rob-baron commented 1 year ago

This has the same problem with sharing our configuration files.

So the basic plan is to save any directory that needs to be backed up to the database, and then downloading a mysqldump of the database.

Saving directories to the database can be done in parallel with either the shredding jobs or the xdmod-ui process.

Storing the mysqldump file can be done in a kubernetes cronjob.

rob-baron commented 1 year ago

First part is to upload tarballs to the database.

Second part is to dump the database to a location for the backup (this will be done in different issue (https://github.com/CCI-MOC/xdmod-cntr/issues/149)