search

deviantony / xtrabackup-scripts

Wrapper scripts (Python) for the Percona Xtrabackup tool.
http://www.percona.com/doc/percona-xtrabackup
110 stars 46 forks source link

Problem on CentOS 6.6 / Maria 10.0.16 #33

Open mrtdc opened 8 years ago

mrtdc commented 8 years ago

Hi, I wanted to ask if this is known issue, or if there is a way to get more verbose error messages to give me a hint on what is the issue.

I am able to run innobackupex and successfully generate a backup via command line, approx thus:

innobackupex --user=root --password=SecretPassGoesHere /nfs-mount/dump-path/

however when I use xtrabackup script it throws an error, thus:

./xb-backup-incremental.sh -r /nfs-mount/dump-path/-u root -p SecretPassGoesHere Command: mkdir -pv /nfs-mount/backuppc...20151206_1450 [ OK ] Starting base backup. [ INFO ] An exception occured while trying to use innobackupex.

and similarly when I look in the logfile,

[root@gpli-vsu2 xb-backup]# cd /var/log/mysql [root@gpli-vsu2 mysql]# ls -la total 12 drwxr-xr-x 2 root root 4096 Nov 25 14:08 . drwxr-xr-x. 7 root root 4096 Dec 6 03:40 .. -rw-r--r-- 1 root root 1328 Dec 6 14:50 xb-backup-incremental.log [root@gpli-vsu2 mysql]# more xb-backup-incremental.log

...

2015-12-06T18:50:17+0000 - Executing: mkdir -pv /nfs-mount/backuppc/20151206_1450 2015-12-06T18:50:17+0000 - Command: mkdir -pv /nfs-mount/backuppc... - Status: OK 2015-12-06T18:50:17+0000 - Starting base backup. - Status: INFO 2015-12-06T18:50:17+0000 - An exception occured while trying to use innobackupex. - Status: FAILED [root@gpli-vsu2 mysql]#

For ref:

host:

[root@srv mysql]# cat /etc/redhat-release CentOS release 6.6 (Final)

MariaDB Version:

Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 45064 Server version: 10.0.16-MariaDB-log MariaDB Server

Just curious what might be the next step to debug this.

Thanks!

Tim

deviantony commented 8 years ago

Have a look at the logs files of the tool, especially the --out-file option: Log file for innobackupex output (default: /var/log/mysql/xtrabackup.out)

mrtdc commented 8 years ago

Hi,

thanks for the reply!

I had posted in my query, the content of the /var/log/mysql/xtrabackup.out file already. It was equally concise. I wasn't sure if there were 'secret' options to make the script more verbose ?

Tim

Tim Chipman Fortech I.T. Solutions (902) 442-6633 http://FortechITSolutions.ca

------ Original Message ------ From: "Anthony Lapenna" notifications@github.com To: "deviantony/xtrabackup-scripts" xtrabackup-scripts@noreply.github.com Cc: "mrtdc" tim.chipman@fortechitsolutions.ca Sent: 12/8/2015 3:45:08 AM Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16 (#33)

Have a look at the logs files of the tool, especially the --out-file option: Log file for innobackupex output (default: /var/log/mysql/xtrabackup.out)

— Reply to this email directly or view it on GitHub.

deviantony commented 8 years ago

Hey Tim, I just found out that you're using the bash scripts, which are deprecated ! Is this really necessary? I recommend you to use the Python version !

mrtdc commented 8 years ago

Hi,

No, not at all necessary. Rather, it was the only version I knew existed. It seems I must have been led down the garden path, by finding ?now obsolete? web page where you discuss using the tool (bash scripts).

I believe the page was, https://deviantony.wordpress.com/2014/05/18/percona-xtrabackup-scripts-part-2/

(maybe it is easily possible to put a 'warn' message at top of that page, "Hey, the bash scripts are out-of-date, don't use them ! " :-)

Sigh! My mistake.

So - I will get the Python version, repeat my tests and let you know how it goes.

Thanks for your help with this!

Tim

Tim Chipman Fortech I.T. Solutions (902) 442-6633 http://FortechITSolutions.ca

------ Original Message ------ From: "Anthony Lapenna" notifications@github.com To: "deviantony/xtrabackup-scripts" xtrabackup-scripts@noreply.github.com Cc: "mrtdc" tim.chipman@fortechitsolutions.ca Sent: 12/8/2015 9:14:14 AM Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16 (#33)

Hey Tim, I just find out that you're using the bash scripts, which are deprecated ! Is this really necessary? I recommend you to use the Python version !

— Reply to this email directly or view it on GitHub.

deviantony commented 8 years ago

Yeaye I'll update my post !

mrtdc commented 8 years ago

Hi Anthony,

I wanted to briefly followup. (I can do this via git hub if you prefer ?) (I am now using latest-current python based scripts ..)

I've had backups / running for a while now - and thought it would be a good idea to test the restore process. (concept!)

So:

I was testing the restore-from-incrementals today. and I appear to hit a snag. Curious if this is a known issue, or if I am simply not using this restore process correctly.

Context:

---paste---

[root@gpli-vsd2 bin]# cd /nfs-mount/backuppc/ [root@gpli-vsd2 backuppc]# ls -la total 88 drwxr-xr-x 22 root root 4096 Jan 29 03:03 . drwxr-xr-x 3 root root 4096 Oct 16 15:00 .. drwxr-xr-x 3 root root 4096 Jan 10 01:00 20160110 << Base backup is inside here, see below. drwxr-xr-x 3 root root 4096 Jan 11 03:03 20160111 drwxr-xr-x 3 root root 4096 Jan 12 03:03 20160112 drwxr-xr-x 3 root root 4096 Jan 13 03:03 20160113 drwxr-xr-x 3 root root 4096 Jan 14 03:03 20160114 drwxr-xr-x 3 root root 4096 Jan 15 03:03 20160115 drwxr-xr-x 3 root root 4096 Jan 16 03:03 20160116 drwxr-xr-x 3 root root 4096 Jan 17 03:03 20160117 drwxr-xr-x 3 root root 4096 Jan 18 03:03 20160118 drwxr-xr-x 3 root root 4096 Jan 19 03:03 20160119 drwxr-xr-x 3 root root 4096 Jan 20 03:03 20160120 drwxr-xr-x 3 root root 4096 Jan 21 03:03 20160121 drwxr-xr-x 3 root root 4096 Jan 22 03:03 20160122 drwxr-xr-x 3 root root 4096 Jan 23 03:03 20160123 drwxr-xr-x 3 root root 4096 Jan 24 03:03 20160124 drwxr-xr-x 3 root root 4096 Jan 25 03:03 20160125 drwxr-xr-x 3 root root 4096 Jan 26 03:03 20160126 drwxr-xr-x 3 root root 4096 Jan 27 03:03 20160127 drwxr-xr-x 3 root root 4096 Jan 28 03:03 20160128 drwxr-xr-x 3 root root 4096 Jan 29 03:03 20160129 [root@gpli-vsd2 backuppc]# cd 20160110 [root@gpli-vsd2 20160110]# ls -la total 12 drwxr-xr-x 3 root root 4096 Jan 10 01:00 . drwxr-xr-x 22 root root 4096 Jan 29 03:03 .. drwxr-xr-x 2 root root 4096 Jan 10 03:09 INC [root@gpli-vsd2 20160110]# cd INC/ [root@gpli-vsd2 INC]# ls -la total 3009624 drwxr-xr-x 2 root root 4096 Jan 10 03:09 . drwxr-xr-x 3 root root 4096 Jan 10 01:00 .. -rw-r--r-- 1 root root 2818468399 Jan 10 01:19 base_backup_20160110_0100.tar.gz << Base backup -rw-r--r-- 1 root root 263371030 Jan 10 03:09 inc_0_backup_20160110_0303.tar.gz << First incremental, ran same day [root@gpli-vsd2 INC]#

(note other intermentals are in the other directories, as per their date stamps, sub directory INC, and the inc# name is suitably incremented, up to the last one:

[root@gpli-vsd2 INC]# pwd /nfs-mount/backuppc/20160129/INC

[root@gpli-vsd2 INC]# ls -la total 581132 drwxr-xr-x 2 root root 4096 Jan 29 03:08 . drwxr-xr-x 3 root root 4096 Jan 29 03:03 .. -rw-r--r-- 1 root root 595065971 Jan 29 03:08 inc_19_backup_20160129_0303.tar.gz

---endpaste---

So, then I try to kick off a restore ..

-- have a clean /var/lib/mysql directory -- kick off the restore command, thus,

pyxtrabackup-restore --base-archive=/nfs-mount/backuppc/20160110/INC/base_backup_20160110_0100.tar.gz \

--incremental-archive=/nfs-mount/backuppc/20160129/INC/inc_19_backup_20160129_0303.tar.gz \ --user=root --password=PASSWORD_GOES_HERE

and for reference, here is how I generated the (a) full backup once per month (b) incremental backups each night:

(A) FULL once per month /usr/bin/pyxtrabackup-inc /nfs-mount/backuppc --user=root --password=PASSWORD_GOES_HERE --tmp-dir=/var/lib/mysql/temporary

(B) Incremental all nights per month (slightly later start time than the full) /usr/bin/pyxtrabackup-inc /nfs-mount/backuppc --user=root --password=PASSWORD_GOES_HERE --tmp-dir=/var/lib/mysql/temporary --incremental

And.

When I ran the pyxtrabackup-restore ... It gave me output thus: (after a few minutes of grinding)

[root@gpli-vsd2 ~]# pyxtrabackup-restore --base-archive=/nfs-mount/backuppc/20160110/INC/base_backup_20160110_0100.tar.gz \

--incremental-archive=/nfs-mount/backuppc/20160129/INC/inc_19_backup_20160129_0303.tar.gz \ --user=root --password=PASSWORD_GOES_HERE

INFO:xtrabackup.restoration_tools:Base backup restoration time: 0:02:55.552947 - Duration: 175 ERROR:xtrabackup.restoration_tools:An error occured during an incremental backup restoration. Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line 85, in apply_incremental_backup archive_repository, prefix) File "/usr/lib/python2.6/site-packages/xtrabackup/filesystem_utils.py", line 95, in get_prefixed_file_in_dir return files[0] IndexError: list index out of range ERROR:xtrabackup.restoration_tools:An error occured during the incremental backups restoration process. Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line 71, in restore_incremental_backups self.apply_incremental_backup(repository, step) File "/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line 85, in apply_incremental_backup archive_repository, prefix) File "/usr/lib/python2.6/site-packages/xtrabackup/filesystem_utils.py", line 95, in get_prefixed_file_in_dir return files[0] IndexError: list index out of range ERROR:xtrabackup.restoration:pyxtrabackup failed. Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/xtrabackup/restoration.py", line 65, in main arguments['--restart']) File "/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line 148, in start_restoration self.restore_incremental_backups(incremental_archive) File "/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line 71, in restore_incremental_backups self.apply_incremental_backup(repository, step) File "/usr/lib/python2.6/site-packages/xtrabackup/restoration_tools.py", line 85, in apply_incremental_backup archive_repository, prefix) File "/usr/lib/python2.6/site-packages/xtrabackup/filesystem_utils.py", line 95, in get_prefixed_file_in_dir return files[0] IndexError: list index out of range [root@gpli-vsd2 ~]#

and we have output dumped to disk, not a trivial amount, but not the full amount for sure,

[root@gpli-vsd2 FAILED_restore-test]# ls -la total 10006592 drwxr-xr-x 7 root root 4096 Jan 29 14:07 . drwxr-xr-x. 21 root root 4096 Jan 29 14:09 .. -rw-r--r-- 1 root root 360 Jan 10 01:00 backup-my.cnf drwx------ 2 root root 12288 Jan 10 01:06 gpli_intranet -rw-r----- 1 mysql root 475 Jan 29 14:07 gpli-vsd2.gpli.local.err -rw-r----- 1 root root 10244587520 Jan 29 14:02 ibdata1 drwx------ 2 root root 4096 Jan 10 01:06 mysql drwxr-xr-x 2 root root 4096 Jan 10 01:06 performance_schema drwx------ 2 root root 4096 Jan 10 01:06 ssd drwxr-xr-x 2 root root 4096 Jan 10 01:06 visualization -rw-r--r-- 1 root root 40 Jan 10 01:06 xtrabackup_binlog_info -rw-r--r-- 1 root root 28 Jan 29 14:02 xtrabackup_binlog_pos_innodb -rw-r----- 1 root root 121 Jan 29 14:02 xtrabackup_checkpoints -rw-r--r-- 1 root root 688 Jan 10 01:06 xtrabackup_info -rw-r----- 1 root root 2097152 Jan 29 14:02 xtrabackup_logfile [root@gpli-vsd2 FAILED_restore-test]# du -sh . 13G .

The date-stamps on some of the files, seems to be giving me a hint - that they are as of Jan.10 versions; and a few are more current (today, jan.29)

Anyhow.

If you see anything here that immediately says,

oh gosh, that is not correct - do this, not that!

or other thoughts - any comments are certainly appreciated.

(For example? I'm not 100% clear that the incremental restore script .. is happily able to navigate the full historical data structure path for all the incrementals? or do I need to do that manually somehow? Or something else ?)

Many thanks for your assistance with this,

Tim Chipman

Tim Chipman Fortech I.T. Solutions (902) 442-6633 http://FortechITSolutions.ca

------ Original Message ------ From: "Anthony Lapenna" notifications@github.com To: "deviantony/xtrabackup-scripts" xtrabackup-scripts@noreply.github.com Cc: "mrtdc" tim.chipman@fortechitsolutions.ca Sent: 12/8/2015 9:14:14 AM Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16 (#33)

Hey Tim, I just find out that you're using the bash scripts, which are deprecated ! Is this really necessary? I recommend you to use the Python version !

— Reply to this email directly or view it on GitHub.

i A

deviantony commented 8 years ago

Hey @mrtdc, I'm not really available right now, I'll try to have a look at this issue next week !

mrtdc commented 8 years ago

Hi, no worries! Thanks for the reply. I really appreciate it. Any timeline is good, I appreciate your assistance.

Tim

Tim Chipman Fortech I.T. Solutions (902) 442-6633 http://FortechITSolutions.ca

------ Original Message ------ From: "Anthony Lapenna" notifications@github.com To: "deviantony/xtrabackup-scripts" xtrabackup-scripts@noreply.github.com Cc: "mrtdc" tim.chipman@fortechitsolutions.ca Sent: 1/30/2016 7:19:27 AM Subject: Re: [xtrabackup-scripts] Problem on CentOS 6.6 / Maria 10.0.16 (#33)

Hey @mrtdc, I'm not really available right now, I'll try to have a look at this issue next week !

— Reply to this email directly or view it on GitHub.