additional-options = --extra-lsndir=/backupdir is not working

[msilva-81]

I am setting up a **new server with Holland version 1.2.10**. Below is the holland.conf file:

## Root holland config file
[holland]

## Paths where holland plugins may be found.
## Can be comma separated
plugin_dirs = /usr/share/holland/plugins

## Top level directory where backups are held
backup_directory = /backupdir

## List of enabled backup sets. Can be comma separated.
## Read from <config_dir>/backupsets/<name>.conf
# backupsets = example, traditional, parallel_backups, non_transactional
backupsets = xtrabackup

# Define a umask for file generated by holland
umask = 0007

# Define a path for holland and its spawned processes
path = /usr/local/bin:/usr/local/sbin:/bin:/sbin:/usr/bin:/usr/sbin

[logging]
## where to write the log
filename = /var/log/holland/holland.log

## debug, info, warning, error, critical (case insensitive)
level = info

## Log format using python logging module format
format = '%(asctime)s PID-%(process)s [%(levelname)s] %(message)s'

I used the config files from the current prod server as a template. Only thing I changed was a directory name, which corresponds to the hostname.

The prod server is running Holland 1.2.2

Here is the config file to perform full backups (xtrabackupfull.conf) from prod server:

[holland:backup]
backups-to-keep = 3
estimated-size-factor = 1.0
plugin = xtrabackup

[xtrabackup]
innobackupex = innobackupex
stream = xbstream
slave-info = no
additional-options = --extra-lsndir=/backupdir

[compression]
method = gzip
inline = yes
level = 1

[mysql:client]
defaults-extra-file = ~/.secret.cnf
# user = ""
# password = ""
# host = ""
# port = ""
# socket = ""

Here is the xtrabackupfull.conf file on the new server:

[holland:backup]
backups-to-keep = 3
estimated-size-factor = 1.0
plugin = xtrabackup

[xtrabackup]
innobackupex = innobackupex
stream = xbstream
slave-info = no
additional-options = --extra-lsndir=/backupdir

[compression]
method = gzip
inline = yes
level = 1

[mysql:client]
defaults-extra-file = ~/.secret.cnf
# user = ""
# password = ""
# host = ""
# port = ""
# socket = ""

According to the /var/log/holland/holland.log on prod, the following command is executed:

[INFO] Executing: /usr/bin/innobackupex --defaults-file=/backupdir/xtrabackupfull/20230217_211550/my.cnf --stream=xbstream --tmpdir=/backupdir/xtrabackupfull/20230217_211550 --no-timestamp --extra-lsndir=/backupdir /backupdir/xtrabackupfull/20230217_211550

The new server is executing this command:

Executing: /bin/innobackupex --defaults-file=/backupdir/xtrabackupfull/20230217_213706/my.cnf --stream=xbstream --tmpdir=/backupdir/xtrabackupfull/20230217_213706 --no-timestamp /backupdir/xtrabackupfull/20230217_213706

The --extra-lsndir=/backupdir/xtrabackupfull/20230217_211550 is not recognized in the command.

Has anyone dealt with this before?

I tried to find documentation on downloading an earlier version but couldn't find any. I was hoping I could uninstall the current holland package and install version 1.2.2 (matching prod). Is there a way to do that? I tried apt-get install holland=1.2.2

[soulen3]

Hello,

I'm surprised your prod server is using 1.2.2. I don't remember building packages for it, and it had some issues. In any case, this should be working with any version of Holland 1.2.

Would it be possible for you to post the output of the backup with the debug flag "-d". Note that there are some spots that might print out the password or other sensitive information. So please check the output before posting.

Steve

[msilva-81]

Hello,

The new server is now the only standalone, prod server. I do not want to run test on it at this time but I have set up a VM with the same versions of Xtrabackup, Holland Backup manager, and Percona MySQL Server.

On my Testing VM (not fixed):

Info from holland.log:

 2023-03-08 15:08:28,756 PID-209005 [INFO] Holland 1.2.10 started with pid 209005
2023-03-08 15:08:28,757 PID-209005 [INFO] --- Starting backup run ---
2023-03-08 15:08:28,758 PID-209005 [INFO] Creating backup path /test/xtrabackupfull/20230308_150828
2023-03-08 15:08:28,781 PID-209005 [INFO] * Generating mysql option file: /test/xtrabackupfull/20230308_150828/my.cnf
2023-03-08 15:08:28,781 PID-209005 [INFO]   + Added !include /etc/my.cnf
2023-03-08 15:08:28,781 PID-209005 [INFO]   + Added !include /root/.my.cnf
2023-03-08 15:08:28,781 PID-209005 [INFO]   + Added [client] section with credentials from [mysql:client] section
2023-03-08 15:08:28,786 PID-209005 [INFO] Estimated Backup Size: 160.47MB
2023-03-08 15:08:28,787 PID-209005 [INFO] Using Historic Space Estimate: Checking for information in /test/xtrabackupfull/newest/backup.conf
2023-03-08 15:08:28,787 PID-209005 [INFO] Last backup used 48.77MB
2023-03-08 15:08:28,787 PID-209005 [INFO] Adjusting estimated size to last backup total * 1.1: 53.65MB
2023-03-08 15:08:28,787 PID-209005 [INFO] Starting backup[xtrabackupfull/20230308_150828] via plugin xtrabackup
2023-03-08 15:08:28,787 PID-209005 [INFO] Executing: /bin/xtrabackup --version
2023-03-08 15:08:28,793 PID-209005 [INFO] xtrabackup: recognized server arguments: --datadir=/datadir/ --tmpdir=/tmp
2023-03-08 15:08:28,794 PID-209005 [INFO] /bin/xtrabackup version 2.4.27 based on MySQL server 5.7.40 Linux (x86_64) (revision id: aae8e82)
2023-03-08 15:08:28,795 PID-209005 [INFO] Executing: /bin/xtrabackup --version
2023-03-08 15:08:28,800 PID-209005 [INFO] xtrabackup: recognized server arguments: --datadir=/datadir/ --tmpdir=/tmp
2023-03-08 15:08:28,800 PID-209005 [INFO] /bin/xtrabackup version 2.4.27 based on MySQL server 5.7.40 Linux (x86_64) (revision id: aae8e82)
2023-03-08 15:08:28,801 PID-209005 [INFO] Executing: /bin/innobackupex --defaults-file=/test/xtrabackupfull/20230308_150828/my.cnf --tmpdir=/test/xtrabackupfull/20230308_150828 --no-timestamp /test/xtrabackupfull/20230308_150828/data
2023-03-08 15:08:28,801 PID-209005 [INFO]   > /dev/null 2 > /test/xtrabackupfull/20230308_150828/xtrabackup.log
2023-03-08 15:09:30,830 PID-209005 [INFO] Final on-disk backup size 48.77MB
2023-03-08 15:09:30,831 PID-209005 [INFO] 30.39% of estimated size 160.47MB
2023-03-08 15:09:30,832 PID-209005 [INFO] Backup completed in 1 minute, 2.04 seconds
2023-03-08 15:09:30,860 PID-209005 [INFO] Purged xtrabackupfull/20230308_145529
2023-03-08 15:09:30,860 PID-209005 [INFO] 1 backups purged
2023-03-08 15:09:30,862 PID-209005 [INFO] Released lock /etc/holland/backupsets/xtrabackupfull.conf
2023-03-08 15:09:30,862 PID-209005 [INFO] --- Ending backup run ---

The senior DBA on our team looked at the python code and she edited line 164 in the /usr/lib/python3/dist-packages/holland/backup/xtrabackup/util.py:

157         # missing executable
158         raise BackupError("pre-command %s failed: %s" % (pre_command, exc.strerror))
159
160     for line in process.stdout:
161         LOG.info("  >> %s", line)
162     returncode = process.wait()
163     if returncode != 0:
164         raise BackupError("pre-command exited with failure status [%d]" % returncode)

On the new server:, (fixed)

The xtrabackupfull.conf:

[holland:backup]
plugin = xtrabackup
backups-to-keep = 2
auto-purge-failures = yes
purge-policy = before-backup
estimated-size-factor = .25

[xtrabackup]
global-defaults = /etc/my.cnf
# innobackupex = innobackupex
# ibbackup = "" # no default
stream = xbstream
apply-logs = no
slave-info = no
safe-slave-backup = no
no-lock = no
# tmpdir = "" # no default
additional-options = --parallel=8, --extra-lsndir=/backups/ #edited /usr/lib/python3/dist-packages/holland/backup/xtrabackup/util.py for backup
pre-command = "" # no default
strict = yes

[mysql:client]
defaults-extra-file = ~/.secret.cnf
# user = "" # no default
# password = "" # no default
# socket = "" # no default
# host = "" # no default
# port = "" # no default

[compression]
method = gzip
options = ""
inline = yes
split = no
splitsize = 1
level = 1

The command that is being executed when runningholland backup xtrabackupfull (from holland.log): 2023-03-08 02:05:02,592 PID-501344 [INFO] Executing: /bin/innobackupex --defaults-file=/backups/xtrabackupfull/20230308_020502/my.cnf --stream=xbstream --tmpdir=/backups/xtrabackupfull/20230308_020502 --no-timestamp /backups/xtrabackupfull/20230308_020502 --parallel=8 --extra-lsndir=/backups

[mikegriffin]

@msilva-81 Are you perhaps using percona-xtrabackup-24?

I noticed that additional-options was not being passed via innobackupex in this case, when trying to set --parallel

This additional line solved the problem in my test:

237a238
>             args.extend(extra_opts)

The block now looks like:

    if not binary_xtrabackup:
        if basedir:
            args.extend(extra_opts)
            args.append(basedir)
    else:
        if extra_opts:
            args.extend(extra_opts)
    return args