Failed to save binary log: Binlog not found from /var/lib/mysql!

lestrrat / mysql-master-ha

Automatically exported from code.google.com/p/mysql-master-ha

0 stars 0 forks source link

 Hi~Yoshinor

In my Production environment

This is 4 servers in IDC room,one master and the others is slave

master:10.10.1.109
slave1:10.10.1.193
slave2:10.10.1.194
slave3+mha_manage:10.10.1.195

I can by sudo to perform root privileges,but I have not root passwd!

And all servers to disable root remote login

My mysql data directory is 700 permissions!!!!!!!

There is no problem in the master-slave replication,and SSH public key 
authentication have been configured

So when I check ssh,It's OK
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ops@B5M-D5:~$ masterha_check_ssh --conf=/etc/app1.cnf 
Sun Jan  6 16:37:32 2013 - [warning] Global configuration file 
/etc/masterha_default.cnf not found. Skipping.
Sun Jan  6 16:37:32 2013 - [info] Reading application default configurations 
from /etc/app1.cnf..
Sun Jan  6 16:37:32 2013 - [info] Reading server configurations from 
/etc/app1.cnf..
Sun Jan  6 16:37:32 2013 - [info] Starting SSH connection tests..
Sun Jan  6 16:37:34 2013 - [debug] 
Sun Jan  6 16:37:32 2013 - [debug]  Connecting via SSH from 
ops@10.10.1.109(10.10.1.109:22) to ops@10.10.1.193(10.10.1.193:22)..
Sun Jan  6 16:37:33 2013 - [debug]   ok.
Sun Jan  6 16:37:33 2013 - [debug]  Connecting via SSH from 
ops@10.10.1.109(10.10.1.109:22) to ops@10.10.1.194(10.10.1.194:22)..
Sun Jan  6 16:37:34 2013 - [debug]   ok.
Sun Jan  6 16:37:34 2013 - [debug] 
Sun Jan  6 16:37:33 2013 - [debug]  Connecting via SSH from 
ops@10.10.1.193(10.10.1.193:22) to ops@10.10.1.109(10.10.1.109:22)..
Sun Jan  6 16:37:34 2013 - [debug]   ok.
Sun Jan  6 16:37:34 2013 - [debug]  Connecting via SSH from 
ops@10.10.1.193(10.10.1.193:22) to ops@10.10.1.194(10.10.1.194:22)..
Sun Jan  6 16:37:34 2013 - [debug]   ok.
Sun Jan  6 16:37:35 2013 - [debug] 
Sun Jan  6 16:37:33 2013 - [debug]  Connecting via SSH from 
ops@10.10.1.194(10.10.1.194:22) to ops@10.10.1.109(10.10.1.109:22)..
Sun Jan  6 16:37:34 2013 - [debug]   ok.
Sun Jan  6 16:37:34 2013 - [debug]  Connecting via SSH from 
ops@10.10.1.194(10.10.1.194:22) to ops@10.10.1.193(10.10.1.193:22)..
Sun Jan  6 16:37:35 2013 - [debug]   ok.
Sun Jan  6 16:37:35 2013 - [info] All SSH connection tests passed successfully.
ops@B5M-D5:~$ 
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I have mkdir /var/log/masterha/app1/ on all Mysql servers

and also run command sudo chmod -R 777 /var/log/masterha/app1

Finally, I check mysql replication configuration
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

ops@B5M-D5:~$ masterha_check_repl --conf=/etc/app1.cnf 
Sun Jan  6 16:43:51 2013 - [warning] Global configuration file 
/etc/masterha_default.cnf not found. Skipping.
Sun Jan  6 16:43:51 2013 - [info] Reading application default configurations 
from /etc/app1.cnf..
Sun Jan  6 16:43:51 2013 - [info] Reading server configurations from 
/etc/app1.cnf..
Sun Jan  6 16:43:51 2013 - [info] MHA::MasterMonitor version 0.53.
Sun Jan  6 16:43:52 2013 - [info] Dead Servers:
Sun Jan  6 16:43:52 2013 - [info] Alive Servers:
Sun Jan  6 16:43:52 2013 - [info]   10.10.1.109(10.10.1.109:3306)
Sun Jan  6 16:43:52 2013 - [info]   10.10.1.193(10.10.1.193:3306)
Sun Jan  6 16:43:52 2013 - [info]   10.10.1.194(10.10.1.194:3306)
Sun Jan  6 16:43:52 2013 - [info] Alive Slaves:
Sun Jan  6 16:43:52 2013 - [info]   10.10.1.193(10.10.1.193:3306)  
Version=5.1.66-0+squeeze1-log (oldest major version between slaves) 
log-bin:enabled
Sun Jan  6 16:43:52 2013 - [info]     Replicating from 
10.10.1.109(10.10.1.109:3306)
Sun Jan  6 16:43:52 2013 - [info]   10.10.1.194(10.10.1.194:3306)  
Version=5.1.66-0+squeeze1-log (oldest major version between slaves) 
log-bin:enabled
Sun Jan  6 16:43:52 2013 - [info]     Replicating from 
10.10.1.109(10.10.1.109:3306)
Sun Jan  6 16:43:52 2013 - [info] Current Alive Master: 
10.10.1.109(10.10.1.109:3306)
Sun Jan  6 16:43:52 2013 - [info] Checking slave configurations..
Sun Jan  6 16:43:52 2013 - [info]  read_only=1 is not set on slave 
10.10.1.193(10.10.1.193:3306).
Sun Jan  6 16:43:52 2013 - [warning]  relay_log_purge=0 is not set on slave 
10.10.1.193(10.10.1.193:3306).
Sun Jan  6 16:43:52 2013 - [info]  read_only=1 is not set on slave 
10.10.1.194(10.10.1.194:3306).
Sun Jan  6 16:43:52 2013 - [warning]  relay_log_purge=0 is not set on slave 
10.10.1.194(10.10.1.194:3306).
Sun Jan  6 16:43:52 2013 - [info] Checking replication filtering settings..
Sun Jan  6 16:43:52 2013 - [info]  binlog_do_db= , binlog_ignore_db= 
Sun Jan  6 16:43:52 2013 - [info]  Replication filtering check ok.
Sun Jan  6 16:43:52 2013 - [info] Starting SSH connection tests..
Sun Jan  6 16:43:53 2013 - [info] All SSH connection tests passed successfully.
Sun Jan  6 16:43:53 2013 - [info] Checking MHA Node version..
Sun Jan  6 16:43:55 2013 - [info]  Version check ok.
Sun Jan  6 16:43:55 2013 - [info] Checking SSH publickey authentication 
settings on the current master..
Sun Jan  6 16:43:55 2013 - [info] HealthCheck: SSH to 10.10.1.109 is reachable.
Sun Jan  6 16:43:56 2013 - [info] Master MHA Node version is 0.53.
Sun Jan  6 16:43:56 2013 - [info] Checking recovery script configurations on 
the current master..
Sun Jan  6 16:43:56 2013 - [info]   Executing command: save_binary_logs 
--command=test --start_pos=4 --binlog_dir=/var/lib/mysql 
--output_file=/var/log/masterha/app1/save_binary_logs_test 
--manager_version=0.53 --start_file=mysql-bin.000179 
Sun Jan  6 16:43:56 2013 - [info]   Connecting to 
ops@10.10.1.109(10.10.1.109).. 
Failed to save binary log: Binlog not found from /var/lib/mysql! If you got 
this error at MHA Manager, please set 
"master_binlog_dir=/path/to/binlog_directory_of_the_master" correctly in the 
MHA Manager's configuration file and try again.
 at /usr/bin/save_binary_logs line 117
        eval {...} called at /usr/bin/save_binary_logs line 66
        main::main() called at /usr/bin/save_binary_logs line 62
Sun Jan  6 16:43:56 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, 
ln152] Master setting check failed!
Sun Jan  6 16:43:56 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, 
ln364] Master configuration failed.
Sun Jan  6 16:43:56 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, 
ln383] Error happend on checking configurations.  at 
/usr/bin/masterha_check_repl line 48
Sun Jan  6 16:43:56 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, 
ln478] Error happened on monitoring servers.
Sun Jan  6 16:43:56 2013 - [info] Got exit code 1 (Not master dead).

MySQL Replication Health is NOT OK!
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 my binlog is in /var/lib/mysql,I have not to modify it

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 This is my /etc/app1.cnf

[server default]
# mysql user and password
user=root
password=0ps.iz3n3
ssh_user=ops
repl_user=rep
repl_password=bm5123
master_binlog_dir=/var/lib/mysql
# working directory on the manager
manager_workdir=/var/log/masterha/app1
# manager log file
manager_log=/var/log/masterha/app1/app1.log
# working directory on MySQL servers
remote_workdir=/var/log/masterha/app1
[server1]
hostname=10.10.1.109
[server2]
hostname=10.10.1.193
[server3]
hostname=10.10.1.194

what's the matter? Yoshinor, Pls help me...........

Thankyou

Original issue reported on code.google.com by yanq...@b5m.com on 6 Jan 2013 at 8:49

I add "log_level=debug" it /etc/app1.cnf ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= ops@B5M-D5:~$ masterha_check_repl --conf=/etc/app1.cnf Sun Jan 6 16:51:50 2013 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sun Jan 6 16:51:50 2013 - [info] Reading application default configurations from /etc/app1.cnf.. Sun Jan 6 16:51:50 2013 - [info] Reading server configurations from /etc/app1.cnf.. Sun Jan 6 16:51:50 2013 - [info] MHA::MasterMonitor version 0.53. Sun Jan 6 16:51:50 2013 - [debug] Connecting to servers.. Sun Jan 6 16:51:50 2013 - [debug] Connected to: 10.10.1.109(10.10.1.109:3306), user=root Sun Jan 6 16:51:50 2013 - [debug] Connected to: 10.10.1.193(10.10.1.193:3306), user=root Sun Jan 6 16:51:50 2013 - [debug] Connected to: 10.10.1.194(10.10.1.194:3306), user=root Sun Jan 6 16:51:50 2013 - [debug] Comparing MySQL versions.. Sun Jan 6 16:51:50 2013 - [debug] Comparing MySQL versions done. Sun Jan 6 16:51:50 2013 - [debug] Connecting to servers done. Sun Jan 6 16:51:50 2013 - [info] Dead Servers: Sun Jan 6 16:51:50 2013 - [info] Alive Servers: Sun Jan 6 16:51:50 2013 - [info] 10.10.1.109(10.10.1.109:3306) Sun Jan 6 16:51:50 2013 - [info] 10.10.1.193(10.10.1.193:3306) Sun Jan 6 16:51:50 2013 - [info] 10.10.1.194(10.10.1.194:3306) Sun Jan 6 16:51:50 2013 - [info] Alive Slaves: Sun Jan 6 16:51:50 2013 - [info] 10.10.1.193(10.10.1.193:3306) Version=5.1.66-0+squeeze1-log (oldest major version between slaves) log-bin:enabled Sun Jan 6 16:51:50 2013 - [debug] Relay log info repository: FILE Sun Jan 6 16:51:50 2013 - [info] Replicating from 10.10.1.109(10.10.1.109:3306) Sun Jan 6 16:51:50 2013 - [info] 10.10.1.194(10.10.1.194:3306) Version=5.1.66-0+squeeze1-log (oldest major version between slaves) log-bin:enabled Sun Jan 6 16:51:50 2013 - [debug] Relay log info repository: FILE Sun Jan 6 16:51:50 2013 - [info] Replicating from 10.10.1.109(10.10.1.109:3306) Sun Jan 6 16:51:50 2013 - [info] Current Alive Master: 10.10.1.109(10.10.1.109:3306) Sun Jan 6 16:51:50 2013 - [info] Checking slave configurations.. Sun Jan 6 16:51:50 2013 - [info] read_only=1 is not set on slave 10.10.1.193(10.10.1.193:3306). Sun Jan 6 16:51:50 2013 - [warning] relay_log_purge=0 is not set on slave 10.10.1.193(10.10.1.193:3306). Sun Jan 6 16:51:50 2013 - [info] read_only=1 is not set on slave 10.10.1.194(10.10.1.194:3306). Sun Jan 6 16:51:50 2013 - [warning] relay_log_purge=0 is not set on slave 10.10.1.194(10.10.1.194:3306). Sun Jan 6 16:51:50 2013 - [info] Checking replication filtering settings.. Sun Jan 6 16:51:50 2013 - [info] binlog_do_db= , binlog_ignore_db= Sun Jan 6 16:51:50 2013 - [info] Replication filtering check ok. Sun Jan 6 16:51:50 2013 - [info] Starting SSH connection tests.. Sun Jan 6 16:51:51 2013 - [debug] Sun Jan 6 16:51:50 2013 - [debug] Connecting via SSH from ops@10.10.1.109(10.10.1.109:22) to ops@10.10.1.193(10.10.1.193:22).. Sun Jan 6 16:51:51 2013 - [debug] ok. Sun Jan 6 16:51:51 2013 - [debug] Connecting via SSH from ops@10.10.1.109(10.10.1.109:22) to ops@10.10.1.194(10.10.1.194:22).. Sun Jan 6 16:51:51 2013 - [debug] ok. Sun Jan 6 16:51:52 2013 - [debug] Sun Jan 6 16:51:50 2013 - [debug] Connecting via SSH from ops@10.10.1.193(10.10.1.193:22) to ops@10.10.1.109(10.10.1.109:22).. Sun Jan 6 16:51:51 2013 - [debug] ok. Sun Jan 6 16:51:51 2013 - [debug] Connecting via SSH from ops@10.10.1.193(10.10.1.193:22) to ops@10.10.1.194(10.10.1.194:22).. Sun Jan 6 16:51:51 2013 - [debug] ok. Sun Jan 6 16:51:52 2013 - [debug] Sun Jan 6 16:51:51 2013 - [debug] Connecting via SSH from ops@10.10.1.194(10.10.1.194:22) to ops@10.10.1.109(10.10.1.109:22).. Sun Jan 6 16:51:52 2013 - [debug] ok. Sun Jan 6 16:51:52 2013 - [debug] Connecting via SSH from ops@10.10.1.194(10.10.1.194:22) to ops@10.10.1.193(10.10.1.193:22).. Sun Jan 6 16:51:52 2013 - [debug] ok. Sun Jan 6 16:51:52 2013 - [info] All SSH connection tests passed successfully. Sun Jan 6 16:51:52 2013 - [info] Checking MHA Node version.. Sun Jan 6 16:51:52 2013 - [info] Version check ok. Sun Jan 6 16:51:52 2013 - [info] Checking SSH publickey authentication settings on the current master.. Sun Jan 6 16:51:53 2013 - [info] HealthCheck: SSH to 10.10.1.109 is reachable. Sun Jan 6 16:51:53 2013 - [info] Master MHA Node version is 0.53. Sun Jan 6 16:51:53 2013 - [info] Checking recovery script configurations on the current master.. Sun Jan 6 16:51:53 2013 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/var/log/masterha/app1/save_binary_logs_test --manager_version=0.53 --start_file=mysql-bin.000179 --debug Sun Jan 6 16:51:53 2013 - [info] Connecting to ops@10.10.1.109(10.10.1.109).. Failed to save binary log: Binlog not found from /var/lib/mysql! If you got this error at MHA Manager, please set "master_binlog_dir=/path/to/binlog_directory_of_the_master" correctly in the MHA Manager's configuration file and try again. at /usr/bin/save_binary_logs line 117 eval {...} called at /usr/bin/save_binary_logs line 66 main::main() called at /usr/bin/save_binary_logs line 62 Sun Jan 6 16:51:54 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, ln152] Master setting check failed! Sun Jan 6 16:51:54 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, ln364] Master configuration failed. Sun Jan 6 16:51:54 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. at /usr/bin/masterha_check_repl line 48 Sun Jan 6 16:51:54 2013 - [error][/usr/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers. Sun Jan 6 16:51:54 2013 - [info] Got exit code 1 (Not master dead). MySQL Replication Health is NOT OK!

Thank you! Yoshinor,It's OK now However, after a few minutes,The new mysql bin log files are still missing execute permissions I have to execute sudo chmod-R 755 /var /lib /mysql again Do I have to be performed on each server crontab, execute once every 1 minute to do chmod-R 755 /var/lib/mysql? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +++++++ -rwxr-xr-x 1 mysql mysql 832 1月 7 07:00 mysql-bin.index -rwxr-xr-x 1 mysql mysql 301 1月 6 18:13 mysqld-relay-bin.000028 -rwxr-xr-x 1 mysql mysql 270 1月 6 18:17 mysqld-relay-bin.000029 -rw-rw---- 1 mysql mysql 301 1月 7 10:00 mysqld-relay-bin.000050 -rw-rw---- 1 mysql mysql 251 1月 7 10:00 mysqld-relay-bin.000051 -rwxr-xr-x 1 mysql mysql 52 1月 7 10:00 mysqld-relay-bin.index -rwxr-xr-x 1 mysql mysql 6 12月 31 11:30 mysql_upgrade_info -rwxr-xr-x 1 mysql mysql 51 1月 7 10:00 relay-log.info drwxr-xr-x 2 mysql mysql 4096 12月 31 12:31 testdb ops@mha:/var/lib/mysq

lestrrat / mysql-master-ha

Failed to save binary log: Binlog not found from /var/lib/mysql! #52