Bash Borg Backup System (BBBS)

Borg wrappers for server pull mode. As distributions repositories may not always be updated, it is designed to work with the standalone version, installed in /opt/bbbs.

Features

Provide pull mode in one command on the server; the wrappers will take care of the required sub-commands
- One connection, from the server to the client
  - Clients doesn't have to know server's address
  - Datas are sent back to the server through a SSH channel
  - TCP or Unix socket
  - Note: borg serve process is not reused, 1 backup = 1 client = 1 wrapper call = 1 serve
- Call pre and post backup hooks on the client
Provide an admin wrapper to start borg with the right user (not to mess with file ownership)
As the created user is dedicated to borg, configuration files are exposed, and redirected via symlinks for borg
Installer handle borg binary retrieval (not to bother with distributions mess), user creation, directory setup
SSH provisioning handle both user and host (known_hosts) key
Ensure correct (ie. UTF-8) locale

Limitations

No real configuration file
Doesn't really handle failure (deconnection, ...)
It's ugly
Probably more ...

Requirement

bash
sudo
socat
git, to retrieve thoses scripts
curl, jq, gpg, to retrieve and verify borg binary (optional, but you'll have to provide it after install)

Instructions

One option is to git clone this repo in /opt/bbbs/ and run the installer.\ Obviously, you have to create do-backup from do-backup.example according to your needs and mark it as executable (chmod +x /opt/bbbs/do-backup).\ borg's dot path are symlinked to more accessible paths (.cache/borg -> ~/cache, .config/borg -> ~/ (to expose keys and security directly), .ssh -> ssh).

client> apt-get update && apt-get install git sudo socat curl jq
client> git clone https://github.com/u1735067/bbbs.git /opt/bbbs
client> /opt/bbbs/installer.sh install-client

server> yum install git sudo socat curl jq
server> git clone https://github.com/u1735067/bbbs.git /opt/bbbs
server> /opt/bbbs/installer.sh install-server

server> /opt/bbbs/ssh-gen-copy-key --generate --key-name server-name_hypervisor --copy --user borg -- root@server-name -p 22
server> gw_key=~borg/ssh/key_server-name_hypervisor; /opt/bbbs/ssh-gen-copy-key --generate --key-name server-name_vm-1 --copy --user borg -- root@172.16.0.1 -p 22 -o ProxyCommand="ssh -i$gw_key -W %h:%p server-name.fqdn -p 22"

server> /opt/bbbs/borg init -e authenticated-blake2 /srv/borg/server-name/hypervisor
server> /opt/bbbs/borg init -e authenticated-blake2 /srv/borg/server-name/vm-1

server> sudo -u borg /opt/bbbs/do-backup

server> /opt/bbbs/borg info /srv/borg/server-name/hypervisor

bbbs-client will call ~borg/backup-pre and ~borg/backup-pre on the client before and after running borg create, you can use them to dump a sql database and remove it for example.

To manage ssh connexion parameters, you can also use ssh_config instead, for example:

server> cat ~borg/ssh/config
Host *
    User borg

Host server-name
    Port 22
    IdentityFile ~borg/ssh/key_server-name_hypervisor

Host 172.16.*.*
    ProxyCommand ssh -F ~borg/ssh/config -W %h:%p server-name

Diagram

╔══════╦════════════════════════════════════════════════════╦══════════════════════════════════════════════════════════════╗
║      ║                       Server                       ║                            Client                            ║
╠══════╬════════════════════════════════════════════════════╬══════════════════════════════════════════════════════════════╣
║ root ║  sudo                                              ║            bbbs-client ─┬─► ~borg/backup-pre                 ║
║      ║    │                                               ║              ▲          ├─► borg create ─► wrapper ─► socat  ║
║      ║    │                                               ║              │          └─► ~borg/backup-post           │    ║
║      ║    │                                               ║              │                                          │    ║
╠══════╬════┼═══════════════════════════════════════════════╬══════════════┼══════════════════════════════════════════╪════╣
║ borg ║    └─► do-backup                                   ║              │                                          │    ║
║      ║             └─► wrapper                            ║              │                                          │    ║
║      ║                   ├──────────────────────► ssh ====╬===► ssh ──► sudo                                        │    ║
║      ║                   └─► socat ──► borg serve  ┊      ║      ┊                                                  │    ║
║      ║                          ▲                  ┊      ║      ┊                                                  │    ║
║      ║                          └──────────────────╘======╬======╛◄─────────────────────────────────────────────────┘    ║
╚══════╩════════════════════════════════════════════════════╩══════════════════════════════════════════════════════════════╝
  ===== = ssh channel

Pre & post hooks on client

*/!\ Don't forget to set executable bit (`chmod +x ~borg/backup-p`) on them!**

Mysql

Cache credentials in ~borg/.mylogin.cnf (see mysqldump, mysql_config_editor):

su borg -c "mysql_config_editor set --user=root --password"

or for MariaDB in ~borg/.my.cnf (see mysqldump):

[mysqldump]
user=mysqluser
password=secret

~borg/backup-pre:

#!/usr/bin/env bash

echo "- Dumping MySQL ..."
mkdir --parents /var/backups/mysql
# GZip directly in case the database is huge -- could be handled by borg using stdin maybe
mysqldump --all-databases --single-transaction \
    | gzip --fast --rsyncable \
    > /var/backups/mysql/borg-dump_$(date --utc "+%Y-%m-%d_%H.%M.%SZ").sql.gz
echo "- Dumping MySQL ... Done"

~borg/backup-post:

#!/usr/bin/env bash

echo "- Removing MySQL Dump ..."
rm -f /var/backups/mysql/borg-*
echo "- Removing MySQL Dump ... Done"

Splunk

See Backup indexed data, Index backup strategy.

~borg/backup-pre:

#!/usr/bin/env bash

echo "- Dumping Splunk ..."
mkdir --parents /var/backups/splunk
/opt/splunk/bin/splunk search "index=*" -output json -maxout 0 \
    | gzip --fast --rsyncable \
    > /var/backups/splunk/borg-dump_$(date --utc "+%Y-%m-%d_%H.%M.%SZ").json.gz
echo "- Dumping Splunk ... Done"

echo "- Stopping Splunk ..."
service splunk stop
#systemctl stop splunk
echo "- Stopping Splunk ... Done"

~borg/backup-post:

#!/usr/bin/env bash

echo "- Removing Splunk Dump ..."
rm -f /var/backups/splunk/borg-*
echo "- Removing Splunk Dump ... Done"

echo "- Starting Splunk ..."
service splunk start # || mail failed to restart service, check ASAP
#systemctl start splunk
echo "- Starting Splunk ... Done"

Why?

While Borg Backup Server (BBS) has been announced, it is not available yet and I needed a solution (preferably before my server crashes). Also, I don't want the clients to have to know how to reach the server. This is ugly but probably the simplest for now (when using pull mode). A correct solution could be to use python as wrapper, with libssh to setup the channel, and parsing nice configuration files.

Locale correction explanation

The wrappers borg and bbbs-client will try to set a correct locale (meaning: with UTF-8 charset).

Why?

Because Python relies on it to read filenames correctly. Most Unix filesystems seems to only care about bytes, but the convention is to use UTF-8 and most locales without the charset specifier (.utf8) doesn't not use UTF-8, so Python use that charset and fails to decode filenames properly. Also, distributions like OMV can decide to set LANG=C before executing crons, which will make Python use the ascii/ANSI encoding.

However, except for extract / fuse mouse according to the issues, borg seems to handle that correctly (even though I'm not sure why, as args.path are str, so scandir will return str which means they are decoded?).

More informations (about this mess):

How?

To do so, it'll parse locales by priority (LC_ALL > LC_CTYPE > LANG), testing if it's indicating UTF-8 charset, or if it's not trying to find the UTF-8 alternative of this locale, or ultimately unsetting it (to let the chance to lower priority locales).\ If there's no locale left, it'll then try to see if there's any UTF-8 locale available, setting LC_CTYPE, starting by C, English (en_GB, en_US, en_*) and ultimatly the first found.\ In the end, if there's really no UTF-8 locale available, none of LC_ALL, LC_CTYPE, LANG should be set, thus triggering the python 3 default UTF-8, but I wasn't able to reproduce this behavior:

# export -n LC_ALL LC_CTYPE LANG; python3 -c 'import sys; print(sys.version); print(sys.getfilesystemencoding())'
3.4.2 (default, Oct  8 2014, 10:45:20)
[GCC 4.9.1]
ascii

u1735067 / bbbs

readme

Bash Borg Backup System (BBBS)

Features

Limitations

Requirement

Instructions

Diagram

Pre & post hooks on client

Mysql

Splunk

Why?

Locale correction explanation

Why?

How?