AWXRestore README improvement

[Commifreak]

The Restore states, that an AWX deplyoment have to be there - but I see, it must not.

I had a restore job on mine and did what was described: Get the operator ready and setup AWX instance. After starting this restore, I get an infinite loop of tasks in the operator. A quick check on the db shows up wrong password logins.

So I moved everything to the trash, setup the operator only and then started the restore: restored without problems.

Is this something worth mentioning?

[lals1]

@Commifreak Thanks for raising this issue. I have stumbled upon the same and spent hours just to understand how restore should work. Readme improvement would surely help many.

By the way, do you know how can I move the backup files i.e. postgres dump, secrets to a pvc in a new k8s cluster where restore is performed? As I understand, AWXRESTORE depends on the backup files being present in a PVC.

[iMikeG6]

Hi guys,

if I may, here's my contribution on how to backup and restore awx.

This method is a daily backup which include backups life cycle management. The backups are kept for 5 days. Older one are delete when the k8s cronjob is ran.

This only apply for restoring an AWX instance on the cluster you have your awx where you want to restore the data to. I haven't tested to restore our awx on a new k8s cluster, so it may not work for your case.

Feel free to use and modify it.

Backup AWX instance

In order to properly backup an AWX instance, follow the step down below.

First, create a new configMap with the following content

# myawx-instance-backup-configmap.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: myawx-instance-backup-configmap
  namespace: myawx-instance
data:
  script: |
    #!/bin/sh
    export KEEP_DAYS=5
    export H_KEEP_DAYS=$(echo $((24*$KEEP_DAYS)))
    export BACKUPS_DIR="/backups"
    export INSTANCE_NAME="myawx-instance"
    cat <<EOF | kubectl apply -f -
    apiVersion: awx.ansible.com/v1beta1
    kind: AWXBackup
    metadata:
      name: $INSTANCE_NAME-backup-$(date +'%Y-%m-%d-%H%M')
      namespace: $INSTANCE_NAME
    spec:
      postgres_image: postgres
      postgres_image_version: "12"
      deployment_name: $INSTANCE_NAME
      backup_storage_class: managed-nfs-storage
      backup_storage_requirements: 20Gi
    EOF
    find $BACKUPS_DIR -type d -mtime +$KEEP_DAYS -name 'tower-openshift-backup-*' -exec rm -r -- '{}' \;
    kubectl get awxbackups -n $INSTANCE_NAME -o go-template --template '{{range .items}}{{.metadata.name}} {{.metadata.creationTimestamp}}{{"\n"}}{{end}}' \
    | sed -e 's/\(^.*\) \(....-..-..\)T.*$/\1 \2/g' \
    | awk '$2 <= "'$(date -d "-$(echo $H_KEEP_DAYS):00:00" +'%Y-%m-%d')'" { print $1 }' \
    | xargs --no-run-if-empty kubectl delete awxbackups -n $INSTANCE_NAME 

:information_source: NOTE: Edit file accordingly to match your awx instance, in this exemple, change myawx-instance to whatever suit your needs.

apply configuration with kubectl

kubectl apply -f myawx-instance-backup-configmap.yaml

Now, create the crontab job

# myawx-instance-backup-cronjob.yaml
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: myawx-instance-backup-cron
  namespace: myawx-instance
spec:
  schedule: "0 0 * * *"
  successfulJobsHistoryLimit: 0
  failedJobsHistoryLimit: 0
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        metadata:
          name: myawx-instance-backup-exec
        spec:
          containers:
            - image: alpine/k8s:1.22.6
              args:
                - "/apps/backup_execution.sh"
              imagePullPolicy: IfNotPresent
              name: myawx-instance-backup-exec
              volumeMounts:
                - name: awx-backup
                  mountPath: /backups
                - name: myawx-instance-backup-configmap
                  readOnly: true
                  mountPath: /apps/backup_execution.sh
                  subPath: backup_execution.sh
          serviceAccountName: awx-operator-controller-manager
          restartPolicy: "Never"
          volumes:
            - name: myawx-instance-backup-configmap
              configMap:
                name: myawx-instance-backup-configmap
                items:
                  - key: script
                    path: backup_execution.sh
                defaultMode: 0755
            - name: awx-backup
              persistentVolumeClaim:
                claimName: myawx-instance-backup-claim

:information_source: NOTE: Edit file accordingly to match your awx instance, in this exemple, change myawx-instance to whatever suit your needs. You can also change the schedule time using the standard crontab syntax

apply configuration with kubectl

kubectl apply -f myawx-instance-backup-cronjob.yaml

Once done, you awx instance will be backup each day at 00:00am according to the schedule time you specified.

Restore AWX instance

To restore a backup of one of your awx instance, you'll first need to delete the instance deployment by removing the object in kubectl get awx -n myawx-instance

kubectl delete awx myawx-instance -n myawx-instance

optionnaly, you can also suspend the cronjob with the following

kubectl patch cronjobs myawx-instance-backup-cron -p "{\"spec\" : {\"suspend\" : true }}"

Only after that, you can create a file with the following content

# myawx-instance-restore-backup.yaml
---
apiVersion: awx.ansible.com/v1beta1
kind: AWXRestore
metadata:
  name: restore-myawx-instance
  namespace: myawx-instance
spec:
  deployment_name: myawx-instance
  backup_pvc_namespace: myawx-instance
  backup_name: "myawx-instance-backup-2022-04-14-0615"

:information_source: NOTE: backup_name should match the backup object stored in the AWXBackup crd ressource. You can list backups using kubectl get awxbackup -n myawx-instance

Apply yaml file

kubectl apply -f myawx-instance-restore-backup.yaml

You can check the status of the restore task

kubectl describe awxrestore restore-myawx-instance -n myawx-instance

once the Reason status is Successful you can delete that object

kubectl delete awxrestore restore-myawx-instance -n myawx-instance