backup/restore jupyterhub

[Jean-Baptiste-Lasselle]

IT is possible to find out exactly which minio bucket is matching which user:

• the pvc mentions the user name
• the persistent volume assigned by a pvc , is given by kubectl :

pierre@first-test-vm:~$ kubectl -n decoderleco get pvc
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
claim-jbadmin   Bound    pvc-66843be8-55d5-472f-a2d1-f02b877f8eb9   2Gi        RWO            csi-s3         <unset>                 59m
claim-pierre    Bound    pvc-a067b982-2c34-443a-bf04-462a82646ddf   2Gi        RWO            csi-s3         <unset>                 7m17s
hub-db-dir      Bound    pvc-c6006ac9-0212-4668-a571-7e46ddb0830d   1Gi        RWO            standard       <unset>                 8m19s

• and yandex csi driver is very simple: the name of the minio s3 bucket is exactly the name of the persistent volume.

This means that to restore a given user's data after a teraform destroy, i need to be able to force a given pvc to create a persistent volue from a given snapshot, and to use csi volume snapshotter. I think i'm right there.

If this works, then i would need to keep track a a database matching users and their volume snapshot

So well this issue is about bringing a full backup restore automation to the jupyterhub part of the platform.

[Jean-Baptiste-Lasselle]

First tooling stack i'm gonna try out is k8up + restic (note: wrestic was merged into k8up some time ago) :

• https://docs.k8up.io/k8up/2.8/references/restic-config-reference.html
• https://docs.k8up.io/k8up/2.8/how-tos/application-aware-backups.html
• https://docs.k8up.io/k8up/2.8/how-tos/restore.html
• a simple tutorial, yet exactly what i want to start with: https://docs.k8up.io/k8up/2.8/tutorials/tutorial.html
• I need then to set it all up using the Operator :
  • https://docs.k8up.io/k8up/2.8/references/operator-config-reference.html
  • https://docs.k8up.io/k8up/2.8/how-tos/installation.html
  • pour la configuration de l'opérateur sur n'importe quel service minio externe de mon choix :
  • regardes https://github.com/k8up-io/k8up/blob/1e7871c51b1a4e1647033fd54847fbecd495f525/charts/k8up/values.yaml#L30
  • puis à la page https://docs.k8up.io/k8up/2.8/references/operator-config-reference.html#_arguments_and_environment_variables :
    • --globals3endpoint value set the S3 endpoint to be used globally [$BACKUP_GLOBALS3ENDPOINT]
    • --globals3bucket value set the S3 bucket to be used globally [$BACKUP_GLOBALS3BUCKET]
    • --globalsecretaccesskey value set the S3 secret access key to be used globally [$BACKUP_GLOBALSECRETACCESSKEY]
    • --globalaccesskeyid value set the S3 access key id to be used globally [$BACKUP_GLOBALACCESSKEYID]
  • I think a priori, that the pattern is one k8sup operator per appliation, because:
    • there is one and only one bucket configuraion (actually 2, there's also the restore one)
    • also there is a parameter suggesting that the data will be restored on same mount path on all pods --datapath value, --mountpath value to which path the PVCs should get mounted in the backup container (default: "/data") [$BACKUP_DATAPATH]
    • interesting question: can this work for a multi user environment, how can k8sup make the deifference between 2 Jupyterhub users ? there is here a very interesting case to study and solution to find , here are useful references i can ry and use here :
    • https://docs.k8up.io/k8up/2.8/explanations/backup.html#_pvc_backup
    • pvc backups with annotations on pods (i add annotations one pods to make sure some are backedup and make sure some are NOT backedup ) : https://docs.k8up.io/k8up/2.8/references/annotations.html
    • how can i make sure that initContainers do not see their PVs backed up ? response is i can specify exactly which container is backedup k8up.io/backupcommand-container | Specify in which container inside pod backup should be done even there is an example : https://docs.k8up.io/k8up/2.8/how-tos/application-aware-backups.html#_pick_a_specific_container
    • can i have an application aware backup and it is scheduled at same time ? I think yes https://docs.k8up.io/k8up/2.8/how-tos/schedules.html