search

deviantony / xtrabackup-scripts

Wrapper scripts (Python) for the Percona Xtrabackup tool.
http://www.percona.com/doc/percona-xtrabackup
110 stars 49 forks source link

Not working on RHEL 7 based system #32

Open bertalanimre opened 8 years ago

bertalanimre commented 8 years ago

Found out, while trying to restore a backup, that the script is not able to run. Most likely because the command called for stopping the mysql service is wrong. It should look like this: systemctl restart mariadb

[root@PHPlist 20151123]# pyxtrabackup-restore --base-archive=/mnt/myrepo/20151123/INC/base_backup_20151123_1326.tar.gz --incremental-archive=/mnt/myrepo/20151123/INC/inc_4_backup_20151123_1400.tar.gz --user=root --password=########
ERROR:xtrabackup.restoration_tools:Unable to manage MySQL service.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/xtrabackup/restoration_tools.py", line 30, in stop_service
    self.command_executor.exec_manage_service('mysql', 'stop')
  File "/usr/lib/python2.7/site-packages/xtrabackup/command_executor.py", line 68, in exec_manage_service
    self.exec_command(command)
  File "/usr/lib/python2.7/site-packages/xtrabackup/command_executor.py", line 13, in exec_command
    stderr=subprocess.STDOUT)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
ERROR:xtrabackup.restoration:pyxtrabackup failed.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/xtrabackup/restoration.py", line 65, in main
    arguments['--restart'])
  File "/usr/lib/python2.7/site-packages/xtrabackup/restoration_tools.py", line 145, in start_restoration
    self.stop_service()
  File "/usr/lib/python2.7/site-packages/xtrabackup/restoration_tools.py", line 30, in stop_service
    self.command_executor.exec_manage_service('mysql', 'stop')
  File "/usr/lib/python2.7/site-packages/xtrabackup/command_executor.py", line 68, in exec_manage_service
    self.exec_command(command)
  File "/usr/lib/python2.7/site-packages/xtrabackup/command_executor.py", line 13, in exec_command
    stderr=subprocess.STDOUT)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1308, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

After the change of command_executor.py at line 67

    #command = ['/etc/init.d/' + service_name, action] (commented out)
    command = ['systemctl', action, 'mariadb'] (simple method of start/stop/restart command)

The backup seems to run fine:

[root@PHPlist]# pyxtrabackup-restore --base-archive=/mnt/myrepo/20151123/INC/base_backup_20151123_1326.tar.gz --incremental-archive=/mnt/myrepo/20151123/INC/inc_4_backup_20151123_1400.tar.gz --user=root --password=########
INFO:xtrabackup.restoration_tools:Base backup restoration time: 0:00:01.755533 - Duration: 1
INFO:xtrabackup.restoration_tools:Incremental step #1 restoration time: 0:00:02.021342 - Duration: 2
INFO:xtrabackup.restoration_tools:Incremental step #2 restoration time: 0:00:01.363370 - Duration: 1
INFO:xtrabackup.restoration_tools:Incremental step #3 restoration time: 0:00:01.158548 - Duration: 1
INFO:xtrabackup.restoration_tools:Incremental step #4 restoration time: 0:00:01.504667 - Duration: 1
INFO:xtrabackup.restoration_tools:Backup final preparation time: 0:00:05.507104 - Duration: 5

HOWEVER: Somewhy this did not restart the mariadb service. If I try to start it manually it gives me the stranges error I've ever seen connected to mariadb:

[root@PHPlist xtrabackup]# systemctl start mariadb
Job for mariadb.service failed. See 'systemctl status mariadb.service' and 'journalctl -xn' for details.
[root@PHPlist xtrabackup]# systemctl status mariadb.service
mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled)
   Active: failed (Result: exit-code) since h 2015-11-23 16:31:03 CET; 5s ago
  Process: 19385 ExecStartPost=/usr/libexec/mariadb-wait-ready $MAINPID (code=exited, status=1/FAILURE)
  Process: 19384 ExecStart=/usr/bin/mysqld_safe --basedir=/usr (code=exited, status=0/SUCCESS)
  Process: 19357 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=0/SUCCESS)
 Main PID: 19384 (code=exited, status=0/SUCCESS)

nov 23 16:31:02 PHPlist systemd[1]: Starting MariaDB database server...
nov 23 16:31:02 PHPlist mysqld_safe[19384]: 151123 16:31:02 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
nov 23 16:31:02 PHPlist mysqld_safe[19384]: 151123 16:31:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
nov 23 16:31:03 PHPlist mysqld_safe[19384]: 151123 16:31:03 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
nov 23 16:31:03 PHPlist systemd[1]: mariadb.service: control process exited, code=exited status=1
nov 23 16:31:03 PHPlist systemd[1]: Failed to start MariaDB database server.
nov 23 16:31:03 PHPlist systemd[1]: Unit mariadb.service entered failed state.

[root@PHPlist xtrabackup]# journalctl -xn
-- Logs begin at h 2015-11-23 11:46:25 CET, end at h 2015-11-23 16:30:29 CET. --
nov 23 16:30:29 PHPlist dbus[430]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
nov 23 16:30:29 PHPlist systemd[1]: Starting Network Manager Script Dispatcher Service...
-- Subject: Unit NetworkManager-dispatcher.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit NetworkManager-dispatcher.service has begun starting up.
nov 23 16:30:29 PHPlist dhclient[453]: bound to 10.0.0.233 -- renewal in 47 seconds.
nov 23 16:30:29 PHPlist dbus-daemon[430]: dbus[430]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
nov 23 16:30:29 PHPlist dbus[430]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
nov 23 16:30:29 PHPlist systemd[1]: Started Network Manager Script Dispatcher Service.
-- Subject: Unit NetworkManager-dispatcher.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit NetworkManager-dispatcher.service has finished starting up.
-- 
-- The start-up result is done.
nov 23 16:30:29 PHPlist nm-dispatcher[19335]: Dispatching action 'dhcp4-change' for enp0s3
nov 23 16:30:29 PHPlist systemd[1]: mariadb.service: control process exited, code=exited status=1
nov 23 16:30:29 PHPlist systemd[1]: Failed to start MariaDB database server.
-- Subject: Unit mariadb.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit mariadb.service has failed.
-- 
-- The result is failed.
nov 23 16:30:29 PHPlist systemd[1]: Unit mariadb.service entered failed state.
bertalanimre commented 8 years ago

Update: This came from the mariadb.log file

151123 17:37:09 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
151123 17:37:09 [Note] /usr/libexec/mysqld (mysqld 5.5.44-MariaDB) starting as process 3987 ...
151123 17:37:09 InnoDB: The InnoDB memory heap is disabled
151123 17:37:09 InnoDB: Mutexes and rw_locks use GCC atomic builtins
151123 17:37:09 InnoDB: Compressed tables use zlib 1.2.7
151123 17:37:09 InnoDB: Using Linux native AIO
151123 17:37:09 InnoDB: Initializing buffer pool, size = 128.0M
151123 17:37:09 InnoDB: Completed initialization of buffer pool
InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!
InnoDB: Possible causes for this error:
 (a) Incorrect log file is used or log file size is changed
 (b) In case default size is used this log file is from 10.0
 (c) Log file is corrupted or there was not enough disk space
 In case (b) you need to set innodb_log_file_size = 48M
151123 17:37:09 [ERROR] Plugin 'InnoDB' init function returned error.
151123 17:37:09 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
151123 17:37:09 [Note] Plugin 'FEEDBACK' is disabled.
151123 17:37:09 [ERROR] Unknown/unsupported storage engine: InnoDB
151123 17:37:09 [ERROR] Aborting

151123 17:37:09 [Note] /usr/libexec/mysqld: Shutdown complete

151123 17:37:09 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
bertalanimre commented 8 years ago

Found the issue:

InnoDB: Error: log file ./ib_logfile0 is of different size 0 50331648 bytes
InnoDB: than specified in the .cnf file 0 5242880 bytes!

have to set the exact same number for the backup applications my.cnf file

$ vim /etc/my.cnf
#added
innodb_log_file_size=50331648

Now the server started.

deviantony commented 8 years ago

I've edited your posts to review the format.

As discussed in Gitter, I should review how to support systemctl.

And besides, mariadb or percona-server while where at it. I think I'll let the user manage the stop/start of the database service, I'll think about it.

AnthonyWC commented 8 years ago

Ran into the same issue with xtrabackup/innobackupex in Centos 7. So this is not an issue with pyxtrabackup but rather an issue with xtrabackup not setting the log file size in the my.cnf file