got the error during the failover test "Applying all logs succeeded. But starting slave failed"

[GoogleCodeExporter]

**** What steps will reproduce the problem?
1. Configure the MHA with (master/master(read only)/slave(read only)) the 
details below in additional information section. Also attached the mha manager 
log output

2. run masterha_manager --conf=/etc/app1.cnf and confirmed OK with "[info] 
Ping(SELECT) succeeded, waiting until MySQL doesn't respond."

3. kill -9 the primary writable master. MHA manager kicks in to set the read 
only master writable as expect. But got the "Applying all logs succeeded. But 
starting slave failed" and the slave (read only) seems still points to the old 
failed master. is there a way that I can have MHA to change the slave (read 
only) points to the new writable master during the failover.  

**** What is the expected output? What do you see instead?
Hoping to see clean output shown failover completed.

**** What version of the product are you using? On what operating system?
OS:
Linux MHA1DBA 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 
x86_64 x86_64 x86_64 GNU/Linux

MySql:
 5.6.17-log 

MHA manager:
0.55

MHA node:
0.54

**** Please provide any additional information below.
/etc/app1.cnf
[server default]
user=root
password=test

repl_user=repl
repl_password=repl

ping_interval=1
latest_priority=0
ignore_fail=1
ssh_user=root

master_pid_file=/db/mysql_binlog/mha/master_pid.pid
remote_workdir=/db/mysql_binlog/mha/
manager_workdir=/db/mysql_binlog/mha/
master_binlog_dir=/db/mysql_binlog/

init_conf_load_script=""
shutdown_script=""
master_ip_failover_script=""
master_ip_online_change_script=""
report_script=""
log_level=debug

[server1]
hostname=10.236.114.128
candidate_master=1

[server2]
hostname=10.236.70.129

[server3]
hostname=10.236.70.131
no_master=1

Original issue reported on code.google.com by kungyi...@gmail.com on 4 Feb 2015 at 6:23

Attachments:

• app1.log

[GoogleCodeExporter]

Is MySQL replication user (repl_use and repl_password, with REPLICATION SLAVE 
privilege) created on the new master 10.236.114.128?

Original comment by Yoshinor...@gmail.com on 5 Feb 2015 at 1:04

[GoogleCodeExporter]

Appreciate your input, sir. After I ran grant replication client on *.* to repl 
identified by 'repl'; and got the below output. I reran the failover test but 
still got the same error of "but recovery on slave partially failed". see most 
bottom for the output of the privilege setting in my dbs (FYI, have confirmed 
the output is the smae from all 3 master/master/slave dbs). 

----- Failover Report -----

app1: MySQL Master failover 10.236.114.128 to 10.236.70.129

Master 10.236.114.128 is down!

Check MHA Manager logs at MHA1DBA for details.

Started automated(non-interactive) failover.
The latest slave 10.236.70.129(10.236.70.129:3306) has all relay logs for 
recovery.
Selected 10.236.70.129 as a new master.
10.236.70.129: OK: Applying all logs succeeded.
10.236.70.131: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
10.236.70.131: WARN: Applying all logs succeeded. But starting slave failed.
Master failover to 10.236.70.129(10.236.70.129:3306) done, but recovery on 
slave partially failed.

mysql> SELECT Repl_slave_priv,Repl_client_priv FROM mysql.user where 
user='repl';
+-----------------+------------------+
| Repl_slave_priv | Repl_client_priv |
+-----------------+------------------+
| Y               | Y                |
+-----------------+------------------+
1 row in set (0.00 sec)

Original comment by kungyi...@gmail.com on 6 Feb 2015 at 12:00

[GoogleCodeExporter]

I don't have shutdown_script specified. Will that be the problem? I have very 
limited knowledge on Perl. Is there any further guide on what value or 
parameter we should set for shutdown_script and master_ip_online_change_scrip. 
Thanks a lot

Original comment by kungyi...@gmail.com on 12 Feb 2015 at 12:17