Borg backup automated script documentation

[ericlay]

I've been using this version of the official automated backup script from the Borg docs for a little over a year. However, I recently started using BorgWarehouse. After refactoring the script, I thought it might be nice to contribute a more specific version for BorgWarehouse usage in the documentation.

The script itself stays fairly close to the OG except it keeps backup logs in the provided location, prunes logs with pruned archives and makes the two api calls to BorgWarehouse for the status and storage updates. It just needs the set up block filled out and it can work out the box. I would be happy if others contributed improvements in log verbosity, improved functionality or better integration with BorgWarehouse if possible too! (I am wondering if the repo uri "/./" ever changes?)

https://gist.github.com/ericlay/baec79bc5bccd0ce690ed088e77825ce

#!/bin/bash

# Automated Borg Backup 
# For more info and OG script check: 
# https://borgbackup.readthedocs.io/en/stable/quickstart.html#automating-backups
#
# Set up to work with BorgWarehouse:
# https://borgwarehouse.com/docs/prologue/introduction/
#
# Works best when used with SystemD timer unit

### Edit this block according to your set up ###############################
remoteUser="borgwarehouse"
remoteHost=""
remotePort=""
repository=""
domain=""
cronjobKey=""
logPath=""
export BORG_PASSPHRASE='' # See the Borg backup docs section "Passphrase notes" for more infos.
export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes # Non-interactively accept relocation of a repository
### Edit below to set your pruning schedule #################################
# Maintains 3 daily, 4 weekly, 6 monthly and 1 yearly archive of this machine
daily="3"
weekly="4"
monthly="6"
yearly="1"
#############################################################################

# Set remote repository url
export BORG_REPO="ssh://$remoteUser@$remoteHost:$remotePort/./$repository"

# Some helpers and error handling
logFile="$logPath"/"$( hostnamectl --static )"-"$( date +%Y-%m-%d-%H:%M )"
info() { printf "%s\t%s\n" "$( date +%H:%M:%S )" "$*" | tee -a "$logFile"; }
seeLog() { printf "\n%s %s" "$*" "$logFile"; }
trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM

# Remove logs of pruned backups
tidyLogs() (
    mapfile -t removeList < <(grep -w "^Pruning archive" "$logFile" | awk '{ print $4 }' )
    for ((i=0; i<${#removeList[@]}; i++)); do
    set -- "$logPath"/"${removeList[i]}"
    info "$( printf "%s %s" "Removing log for pruned archive:" "${removeList[i]}" )"
    rm "$@"
    done
)

# Backup the most important directories into an archive named after
# the machine this script is currently running on:
info "Starting backup to $BORG_REPO" 

borg create                         \
    --info                          \
    --stats                         \
    --compression lz4               \
    --exclude-caches                \
    --exclude 'home/*/.cache/*'     \
    --exclude 'var/tmp/*'           \
    --exclude '/etc/mtab/*'         \
    --exclude '/dev/*'              \
    --exclude '/lost+found/'        \
    --exclude '/mnt/*'              \
    --exclude '/proc/*'             \
    --exclude '/run/*'              \
    --exclude '/sys/*'              \
    --exclude '/tmp/*'              \
    --exclude '/home/*/.cache/mozilla/firefox/' \
    --exclude '/home/*/.local/share/Trash/' \
    --exclude '/home/*/.thumbnails/' \
                                    \
    ::'{hostname}-{now:%Y-%m-%d-%H:%M}' \
    /boot                           \
    /bin                            \
    /etc                            \
    /home                           \
    /lib                            \
    /lib64                          \
    /opt                            \
    /root                           \
    /sbin                           \
    /services                       \
    /srv                            \
    /usr                            \
    /var                            \
    2>> "$logFile"

backup_exit=$?

# Use the `prune` subcommand to maintain 3 daily, 4 weekly, 6 monthly and 1 yearly
# archive of THIS machine. The '{hostname}-*' matching is very important to
# limit prune's operation to this machine's archives and not apply to
# other machines' archives also:
info "Pruning repository" 

borg prune                          \
    --list                          \
    --info                          \
    --stats                         \
    --save-space                    \
    --glob-archives '{hostname}-*'  \
    --keep-daily    "$daily"               \
    --keep-weekly   "$weekly"               \
    --keep-monthly  "$monthly"               \
    --keep-yearly   "$yearly"               \
    2>> "$logFile"

prune_exit=$?

# Tidy up log files of pruned archives
tidyLogs

# actually free repo disk space by compacting segments
info "Compacting repository"

borg compact --info 2>> "$logFile"

compact_exit=$?

# Execute API calls to handle status and storage data for borgwarehouse
curl -Ss --request POST \
    --url "$domain/api/cronjob/checkStatus" \
    --header "Authorization: Bearer $cronjobKey" 2>> "$logFile"

curlStatusExit=$?

if [ $curlStatusExit -gt 0 ]; then 
    info "Error: Status cron not executed"
else info "Success! Status cron has been executed"
fi

curl -Ss --request POST \
    --url "$domain/api/cronjob/getStorageUsed" \
    --header "Authorization: Bearer $cronjobKey" 2>> "$logFile"

curlStorageExit=$?

if [ $curlStorageExit -gt 0 ]; then 
    info "Error: Storage cron not executed"
else info "Success! Storage cron has been executed"
fi

# use highest exit code as global exit code
global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))
global_exit=$(( compact_exit > global_exit ? compact_exit : global_exit ))

if [ ${global_exit} -eq 0 ]; then
    info "Backup, Prune, and Compact finished successfully"
    seeLog "Log available:"
elif [ ${global_exit} -eq 1 ]; then
    info "Backup, Prune, and/or Compact finished with warnings"
    seeLog "Check log for warnings:"
else
    info "Backup, Prune, and/or Compact finished with errors"
    seeLog "Check log for errors:"
fi

exit ${global_exit}

[Ravinou]

Hi @ericlay 😊

Do you have a link to the original script you use? It's a BorgBackup script, isn't it (I never use it) ?

Tell me what you'd like exactly, I didn't necessarily understand the meaning of your request ^^. Would you like me to include script generation in the setup wizard like I do for BorgMatic ?

[ericlay]

Hi, thanks for the response.

It is the BorgBackup scipt, orginally, no need to stray there. Script generation in the setup wizard would be an awesome addition. Just like the Borgmatic example.

Along with that I provided my working example, maybe slightly opinionated :) in the logs and handling of the api calls. It could also be improved by using variables from the config block to setup the pruning schedule as well. However, just an example.