We have implemented mysql MHA with Keepalived where we are able to manage successfull automatic failover most of times but we are having this odd issue while performing masterha_master_switch online where online switch gets stuck for no reason and we cancel it with CTRL+C to get over it. But sometimes it gets completed with sucessfull message as share below for both the scenarios. Please help us find why we are getting this wierd issue. I couldn't find anything in the perl scripts as well. Kindly Help.
echo -e 'config files swapped after Failover ........\n'
config files swapped after Failover ........
mutt -s ' Failover has happened in - testingserver02.teslamotors.com'
orig_master_host=testingserver01.teslamotors.com
Wed May 23 00:27:11 2018 - [info] ok.
^C
=========== Gets Stuck at this POsition =====================
Wed May 23 00:28:05 2018 - [info] Killing thread 448 on testingserver01.teslamotors.com(xxxxx.11:3306)..
Wed May 23 00:28:05 2018 - [info] ok.
Wed May 23 00:28:05 2018 - [info] Killing thread 7600 on testingserver02.teslamotors.com(xxxxx.12:3306)..
Wed May 23 00:28:05 2018 - [info] ok.
Wed May 23 00:28:05 2018 - [info] Killing thread 7608 on testingserver03.teslamotors.com(xxxxx.13:3306)..
Wed May 23 00:28:05 2018 - [info] ok.
HI Team,
We have implemented mysql MHA with Keepalived where we are able to manage successfull automatic failover most of times but we are having this odd issue while performing masterha_master_switch online where online switch gets stuck for no reason and we cancel it with CTRL+C to get over it. But sometimes it gets completed with sucessfull message as share below for both the scenarios. Please help us find why we are getting this wierd issue. I couldn't find anything in the perl scripts as well. Kindly Help.
/usr/bin/masterha_master_switch --master_state=alive --conf=/etc/app1.cnf_db01master --new_master_host=testingserver01.teslamotors.com --orig_master_is_new_slave --interactive
Failed attempt
echo -e 'config files swapped after Failover ........\n' config files swapped after Failover ........
mutt -s ' Failover has happened in - testingserver02.teslamotors.com'
orig_master_host=testingserver01.teslamotors.com Wed May 23 00:27:11 2018 - [info] ok. ^C =========== Gets Stuck at this POsition =====================
Wed May 23 00:28:05 2018 - [info] Killing thread 448 on testingserver01.teslamotors.com(xxxxx.11:3306).. Wed May 23 00:28:05 2018 - [info] ok. Wed May 23 00:28:05 2018 - [info] Killing thread 7600 on testingserver02.teslamotors.com(xxxxx.12:3306).. Wed May 23 00:28:05 2018 - [info] ok. Wed May 23 00:28:05 2018 - [info] Killing thread 7608 on testingserver03.teslamotors.com(xxxxx.13:3306).. Wed May 23 00:28:05 2018 - [info] ok.
Successfull attempt
'[' start = START ']'
'[' start = start ']'
ssh mhaadmin@testingserver01.teslamotors.com 'hostname -I | grep '\''10.20.00.19'\''' xxxxx.11 10.20.00.19
'[' 0 -eq 0 ']'
ACTIVE_MASTER=testingserver01.teslamotors.com
ssh mhaadmin@testingserver01.teslamotors.com 'sudo service keepalived stop' Redirecting to /bin/systemctl stop keepalived.service
'[' 0 -eq 0 ']'
echo -e 'Stopped keepalived in testingserver01.teslamotors.com !!!' Stopped keepalived in testingserver01.teslamotors.com !!!
sleep 5
start_keepalived testingserver01.teslamotors.com
orig_master_hostname=testingserver01.teslamotors.com
ssh mhaadmin@testingserver01.teslamotors.com 'sudo service keepalived start' Redirecting to /bin/systemctl start keepalived.service
echo -e 'Failover to testingserver01.teslamotors.com is COMPLETE.. \n\n' Failover to testingserver01.teslamotors.com is COMPLETE..
echo -e 'Swap the config files after Failover \n' Swap the config files after Failover
sudo rm -rf /etc/app1.cnf /etc/masterha_default.cnf
sudo cp /etc/app1.cnf_db02master /etc/app1.cnf
sudo cp /etc/masterha_default.cnf_db02 /etc/masterha_default.cnf
echo -e 'config files swapped after Failover ........\n' config files swapped after Failover ........
mutt -s ' Failover has happened in - testingserver01.teslamotors.com'
orig_master_host=testingserver02.teslamotors.com Wed May 23 00:35:39 2018 - [info] ok. Wed May 23 00:35:39 2018 - [info] Wed May 23 00:35:39 2018 - [info] Switching slaves in parallel.. Wed May 23 00:35:39 2018 - [info] Wed May 23 00:35:39 2018 - [info] -- Slave switch on host testingserver03.teslamotors.com(xxxxx.13:3306) started, pid: 14777 Wed May 23 00:35:39 2018 - [info] Wed May 23 00:35:41 2018 - [info] Log messages from testingserver03.teslamotors.com ... Wed May 23 00:35:41 2018 - [info] Wed May 23 00:35:39 2018 - [info] Waiting to execute all relay logs on testingserver03.teslamotors.com(xxxxx.13:3306).. Wed May 23 00:35:39 2018 - [info] master_pos_wait(mysql-bin.000015:2180) completed on testingserver03.teslamotors.com(xxxxx.13:3306). Executed 0 events. Wed May 23 00:35:39 2018 - [info] done. Wed May 23 00:35:39 2018 - [info] Resetting slave testingserver03.teslamotors.com(xxxxx.13:3306) and starting replication from the new master testingserver02.teslamotors.com(xxxxx.12:3306).. Wed May 23 00:35:39 2018 - [info] Executed CHANGE MASTER. Wed May 23 00:35:40 2018 - [info] Slave started. Wed May 23 00:35:41 2018 - [info] End of log messages from testingserver03.teslamotors.com ... Wed May 23 00:35:41 2018 - [info] Wed May 23 00:35:41 2018 - [info] -- Slave switch on host testingserver03.teslamotors.com(xxxxx.13:3306) succeeded. Wed May 23 00:35:41 2018 - [info] Unlocking all tables on the orig master: Wed May 23 00:35:41 2018 - [info] Executing UNLOCK TABLES.. Wed May 23 00:35:41 2018 - [info] ok. Wed May 23 00:35:41 2018 - [info] Starting orig master as a new slave.. Wed May 23 00:35:41 2018 - [info] Resetting slave testingserver01.teslamotors.com(xxxxx.11:3306) and starting replication from the new master testingserver02.teslamotors.com(xxxxx.12:3306).. Wed May 23 00:35:41 2018 - [info] Executed CHANGE MASTER. Wed May 23 00:35:42 2018 - [info] Slave started. Wed May 23 00:35:42 2018 - [info] All new slave servers switched successfully. Wed May 23 00:35:42 2018 - [info] Wed May 23 00:35:42 2018 - [info] Phase 5: New master cleanup phase.. Wed May 23 00:35:42 2018 - [info] Wed May 23 00:35:42 2018 - [info] testingserver02.teslamotors.com: Resetting slave info succeeded. Wed May 23 00:35:42 2018 - [info] Switching master to testingserver02.teslamotors.com(xxxxx.12:3306) completed successfully.