Open lixiaofeng1026 opened 4 years ago
The mysqlmha manual failover is a success。 [root@mysql-manager-23 conf]# masterha_master_switch --conf=/usr/local/mha-helper-master/conf/manager.conf --master_state=dead --dead_master_host=172.19.14.23
[root@mysql-manager-23 support-files]# masterha_check_repl --conf=/usr/local/mha-helper-master/conf/manager.conf
Tue Oct 29 19:52:10 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Oct 29 19:52:10 2019 - [info] Reading application default configuration from /usr/local/mha-helper-master/conf/manager.conf..
Tue Oct 29 19:52:10 2019 - [info] Reading server configuration from /usr/local/mha-helper-master/conf/manager.conf..
Tue Oct 29 19:52:10 2019 - [info] MHA::MasterMonitor version 0.58.
Tue Oct 29 19:52:10 2019 - [debug] Connecting to servers..
Tue Oct 29 19:52:11 2019 - [debug] Connected to: 172.19.14.21(172.19.14.21:3306), user=root
Tue Oct 29 19:52:11 2019 - [debug] Number of slave worker threads on host 172.19.14.21(172.19.14.21:3306): 0
Tue Oct 29 19:52:11 2019 - [debug] Connected to: 172.19.14.23(172.19.14.23:3306), user=root
Tue Oct 29 19:52:11 2019 - [debug] Number of slave worker threads on host 172.19.14.23(172.19.14.23:3306): 0
Tue Oct 29 19:52:11 2019 - [debug] Connected to: 172.19.14.22(172.19.14.22:3306), user=root
Tue Oct 29 19:52:11 2019 - [debug] Number of slave worker threads on host 172.19.14.22(172.19.14.22:3306): 0
Tue Oct 29 19:52:11 2019 - [debug] Comparing MySQL versions..
Tue Oct 29 19:52:11 2019 - [debug] Comparing MySQL versions done.
Tue Oct 29 19:52:11 2019 - [debug] Connecting to servers done.
Tue Oct 29 19:52:11 2019 - [info] GTID failover mode = 0
Tue Oct 29 19:52:11 2019 - [info] Dead Servers:
Tue Oct 29 19:52:11 2019 - [info] Alive Servers:
Tue Oct 29 19:52:11 2019 - [info] 172.19.14.21(172.19.14.21:3306)
Tue Oct 29 19:52:11 2019 - [info] 172.19.14.23(172.19.14.23:3306)
Tue Oct 29 19:52:11 2019 - [info] 172.19.14.22(172.19.14.22:3306)
Tue Oct 29 19:52:11 2019 - [info] Alive Slaves:
Tue Oct 29 19:52:11 2019 - [info] 172.19.14.23(172.19.14.23:3306) Version=8.0.12 (oldest major version between slaves) log-bin:enabled
Tue Oct 29 19:52:11 2019 - [debug] Relay log info repository: TABLE
Tue Oct 29 19:52:11 2019 - [info] Replicating from 172.19.14.21(172.19.14.21:3306)
Tue Oct 29 19:52:11 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Tue Oct 29 19:52:11 2019 - [info] 172.19.14.22(172.19.14.22:3306) Version=8.0.12 (oldest major version between slaves) log-bin:enabled
Tue Oct 29 19:52:11 2019 - [debug] Relay log info repository: TABLE
Tue Oct 29 19:52:11 2019 - [info] Replicating from 172.19.14.21(172.19.14.21:3306)
Tue Oct 29 19:52:11 2019 - [info] Not candidate for the new Master (no_master is set)
Tue Oct 29 19:52:11 2019 - [info] Current Alive Master: 172.19.14.21(172.19.14.21:3306)
Tue Oct 29 19:52:11 2019 - [info] Checking slave configurations..
Tue Oct 29 19:52:11 2019 - [info] Checking replication filtering settings..
Tue Oct 29 19:52:11 2019 - [info] binlog_do_db= apolloconfigdb,apolloportaldb, binlog_ignore_db= information_schema,mysql,performance_schema,sys
Tue Oct 29 19:52:11 2019 - [info] Replication filtering check ok.
Tue Oct 29 19:52:11 2019 - [info] GTID (with auto-pos) is not supported
Tue Oct 29 19:52:11 2019 - [info] Starting SSH connection tests..
Tue Oct 29 19:52:12 2019 - [debug]
Tue Oct 29 19:52:11 2019 - [debug] Connecting via SSH from root@172.19.14.21(172.19.14.21:11017) to root@172.19.14.23(172.19.14.23:11017)..
Tue Oct 29 19:52:11 2019 - [debug] ok.
Tue Oct 29 19:52:11 2019 - [debug] Connecting via SSH from root@172.19.14.21(172.19.14.21:11017) to root@172.19.14.22(172.19.14.22:11017)..
Tue Oct 29 19:52:12 2019 - [debug] ok.
Tue Oct 29 19:52:12 2019 - [debug]
Tue Oct 29 19:52:11 2019 - [debug] Connecting via SSH from root@172.19.14.23(172.19.14.23:11017) to root@172.19.14.21(172.19.14.21:11017)..
Tue Oct 29 19:52:12 2019 - [debug] ok.
Tue Oct 29 19:52:12 2019 - [debug] Connecting via SSH from root@172.19.14.23(172.19.14.23:11017) to root@172.19.14.22(172.19.14.22:11017)..
Tue Oct 29 19:52:12 2019 - [debug] ok.
Tue Oct 29 19:52:13 2019 - [debug]
Tue Oct 29 19:52:12 2019 - [debug] Connecting via SSH from root@172.19.14.22(172.19.14.22:11017) to root@172.19.14.21(172.19.14.21:11017)..
Tue Oct 29 19:52:12 2019 - [debug] ok.
Tue Oct 29 19:52:12 2019 - [debug] Connecting via SSH from root@172.19.14.22(172.19.14.22:11017) to root@172.19.14.23(172.19.14.23:11017)..
Tue Oct 29 19:52:13 2019 - [debug] ok.
Tue Oct 29 19:52:13 2019 - [info] All SSH connection tests passed successfully.
Tue Oct 29 19:52:13 2019 - [info] Checking MHA Node version..
Tue Oct 29 19:52:14 2019 - [info] Version check ok.
Tue Oct 29 19:52:14 2019 - [info] Checking SSH publickey authentication settings on the current master..
Tue Oct 29 19:52:14 2019 - [debug] SSH connection test to 172.19.14.21, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=5 -i /root/.ssh/id_rsa, timeout 5
Tue Oct 29 19:52:14 2019 - [info] HealthCheck: SSH to 172.19.14.21 is reachable.
Tue Oct 29 19:52:14 2019 - [info] Master MHA Node version is 0.58.
Tue Oct 29 19:52:14 2019 - [info] Checking recovery script configurations on 172.19.14.21(172.19.14.21:3306)..
Tue Oct 29 19:52:14 2019 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/mysql/data --output_file=/var/mysqlha/save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin-9501.000008 --debug
Tue Oct 29 19:52:14 2019 - [info] Connecting to root@172.19.14.21(172.19.14.21:11017)..
Creating /var/mysqlha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/mysql/data, up to mysql-bin-9501.000008
Tue Oct 29 19:52:15 2019 - [info] Binlog setting check done.
Tue Oct 29 19:52:15 2019 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Tue Oct 29 19:52:15 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=172.19.14.23 --slave_ip=172.19.14.23 --slave_port=3306 --workdir=/var/mysqlha --target_version=8.0.12 --manager_version=0.58 --relay_dir=/var/mysql/log --current_relay_log=mysql-relay-bin.000002 --debug --ssh_options='-o ServerAliveInterval=60 -o ServerAliveCountMax=20 -o StrictHostKeyChecking=no -o ConnectionAttempts=5 -o PasswordAuthentication=no -o BatchMode=yes -i /root/.ssh/id_rsa' --slave_pass=xxx
Tue Oct 29 19:52:15 2019 - [info] Connecting to root@172.19.14.23(172.19.14.23:11017)..
Checking slave recovery environment settings..
Relay log found at /var/mysql/log, up to mysql-relay-bin.000002
Temporary relay log file is /var/mysql/log/mysql-relay-bin.000002
Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue Oct 29 19:52:15 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=172.19.14.22 --slave_ip=172.19.14.22 --slave_port=3306 --workdir=/var/mysqlha --target_version=8.0.12 --manager_version=0.58 --relay_dir=/var/mysql/log --current_relay_log=mysql-relay-bin.000002 --debug --ssh_options='-o ServerAliveInterval=60 -o ServerAliveCountMax=20 -o StrictHostKeyChecking=no -o ConnectionAttempts=5 -o PasswordAuthentication=no -o BatchMode=yes -i /root/.ssh/id_rsa' --slave_pass=xxx
Tue Oct 29 19:52:15 2019 - [info] Connecting to root@172.19.14.22(172.19.14.22:11017)..
Checking slave recovery environment settings..
Relay log found at /var/mysql/log, up to mysql-relay-bin.000002
Temporary relay log file is /var/mysql/log/mysql-relay-bin.000002
Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Tue Oct 29 19:52:16 2019 - [info] Slaves settings check done.
Tue Oct 29 19:52:16 2019 - [info]
172.19.14.21(172.19.14.21:3306) (current master)
+--172.19.14.23(172.19.14.23:3306)
+--172.19.14.22(172.19.14.22:3306)
Tue Oct 29 19:52:16 2019 - [info] Checking replication health on 172.19.14.23.. Tue Oct 29 19:52:16 2019 - [info] ok. Tue Oct 29 19:52:16 2019 - [info] Checking replication health on 172.19.14.22.. Tue Oct 29 19:52:16 2019 - [info] ok. Tue Oct 29 19:52:16 2019 - [info] Checking master_ip_failover_script status: Tue Oct 29 19:52:16 2019 - [info] /usr/local/mha-helper-master/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=172.19.14.21 --orig_master_ip=172.19.14.21 --orig_master_port=3306 --orig_master_ssh_port=11017 --ssh_options='-o ServerAliveInterval=60 -o ServerAliveCountMax=20 -o StrictHostKeyChecking=no -o ConnectionAttempts=5 -o PasswordAuthentication=no -o BatchMode=yes -i /root/.ssh/id_rsa' [2019-10-29 19:52:16] Doing sanity checks Tue Oct 29 19:52:16 2019 - [info] OK. Tue Oct 29 19:52:16 2019 - [warning] shutdown_script is not defined. Tue Oct 29 19:52:16 2019 - [debug] Disconnected from 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:52:16 2019 - [debug] Disconnected from 172.19.14.23(172.19.14.23:3306) Tue Oct 29 19:52:16 2019 - [debug] Disconnected from 172.19.14.22(172.19.14.22:3306) Tue Oct 29 19:52:16 2019 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
--orig_master_ssh_user=(from 0.56, current master's ssh user) --orig_master_is_new_slave=(from 0.56, notifying whether the orig master will be new slave or not) --new_master_ssh_user=(from 0.56, new master's ssh user)
Hi, The mysqlmha online switch is not successful(masterha_master_switch --master_state=alive)。 [root@mysql-manager-23 conf]# masterha_master_switch --master_state=alive --global_conf=/usr/local/mha-helper-master/conf/global.conf --conf=/usr/local/mha-helper-master/conf/manager.conf Tue Oct 29 19:40:02 2019 - [info] MHA::MasterRotate version 0.58. Tue Oct 29 19:40:02 2019 - [info] Starting online master switch.. Tue Oct 29 19:40:02 2019 - [info] Tue Oct 29 19:40:02 2019 - [info] * Phase 1: Configuration Check Phase.. Tue Oct 29 19:40:02 2019 - [info] Tue Oct 29 19:40:02 2019 - [info] Reading default configuration from /usr/local/mha-helper-master/conf/global.conf.. Tue Oct 29 19:40:02 2019 - [info] Reading application default configuration from /usr/local/mha-helper-master/conf/manager.conf.. Tue Oct 29 19:40:02 2019 - [info] Reading server configuration from /usr/local/mha-helper-master/conf/manager.conf.. Tue Oct 29 19:40:02 2019 - [debug] Connecting to servers.. Tue Oct 29 19:40:03 2019 - [debug] Connected to: 172.19.14.21(172.19.14.21:3306), user=root Tue Oct 29 19:40:03 2019 - [debug] Number of slave worker threads on host 172.19.14.21(172.19.14.21:3306): 0 Tue Oct 29 19:40:03 2019 - [debug] Connected to: 172.19.14.23(172.19.14.23:3306), user=root Tue Oct 29 19:40:03 2019 - [debug] Number of slave worker threads on host 172.19.14.23(172.19.14.23:3306): 0 Tue Oct 29 19:40:03 2019 - [debug] Connected to: 172.19.14.22(172.19.14.22:3306), user=root Tue Oct 29 19:40:03 2019 - [debug] Number of slave worker threads on host 172.19.14.22(172.19.14.22:3306): 0 Tue Oct 29 19:40:03 2019 - [debug] Comparing MySQL versions.. Tue Oct 29 19:40:03 2019 - [debug] Comparing MySQL versions done. Tue Oct 29 19:40:03 2019 - [debug] Connecting to servers done. Tue Oct 29 19:40:03 2019 - [info] GTID failover mode = 0 Tue Oct 29 19:40:03 2019 - [info] Current Alive Master: 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:40:03 2019 - [info] Alive Slaves: Tue Oct 29 19:40:03 2019 - [info] 172.19.14.23(172.19.14.23:3306) Version=8.0.12 (oldest major version between slaves) log-bin:enabled Tue Oct 29 19:40:03 2019 - [debug] Relay log info repository: TABLE Tue Oct 29 19:40:03 2019 - [info] Replicating from 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:40:03 2019 - [info] Primary candidate for the new Master (candidate_master is set) Tue Oct 29 19:40:03 2019 - [info] 172.19.14.22(172.19.14.22:3306) Version=8.0.12 (oldest major version between slaves) log-bin:enabled Tue Oct 29 19:40:03 2019 - [debug] Relay log info repository: TABLE Tue Oct 29 19:40:03 2019 - [info] Replicating from 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:40:03 2019 - [info] Not candidate for the new Master (no_master is set)
It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 172.19.14.21(172.19.14.21:3306)? (YES/no): yes Tue Oct 29 19:40:05 2019 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time.. Tue Oct 29 19:40:05 2019 - [info] ok. Tue Oct 29 19:40:05 2019 - [info] Checking MHA is not monitoring or doing failover.. Tue Oct 29 19:40:05 2019 - [info] Checking replication health on 172.19.14.23.. Tue Oct 29 19:40:05 2019 - [info] ok. Tue Oct 29 19:40:05 2019 - [info] Checking replication health on 172.19.14.22.. Tue Oct 29 19:40:05 2019 - [info] ok. Tue Oct 29 19:40:05 2019 - [info] Searching new master from slaves.. Tue Oct 29 19:40:05 2019 - [info] Candidate masters from the configuration file: Tue Oct 29 19:40:05 2019 - [info] 172.19.14.21(172.19.14.21:3306) Version=8.0.12 log-bin:enabled Tue Oct 29 19:40:05 2019 - [info] 172.19.14.23(172.19.14.23:3306) Version=8.0.12 (oldest major version between slaves) log-bin:enabled Tue Oct 29 19:40:05 2019 - [debug] Relay log info repository: TABLE Tue Oct 29 19:40:05 2019 - [info] Replicating from 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:40:05 2019 - [info] Primary candidate for the new Master (candidate_master is set) Tue Oct 29 19:40:05 2019 - [info] Non-candidate masters: Tue Oct 29 19:40:05 2019 - [info] 172.19.14.22(172.19.14.22:3306) Version=8.0.12 (oldest major version between slaves) log-bin:enabled Tue Oct 29 19:40:05 2019 - [debug] Relay log info repository: TABLE Tue Oct 29 19:40:05 2019 - [info] Replicating from 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:40:05 2019 - [info] Not candidate for the new Master (no_master is set) Tue Oct 29 19:40:05 2019 - [info] Searching from candidate_master slaves which have received the latest relay log events.. Tue Oct 29 19:40:05 2019 - [info] From: 172.19.14.21(172.19.14.21:3306) (current master) +--172.19.14.23(172.19.14.23:3306) +--172.19.14.22(172.19.14.22:3306)
To: 172.19.14.23(172.19.14.23:3306) (new master) +--172.19.14.22(172.19.14.22:3306)
Starting master switch from 172.19.14.21(172.19.14.21:3306) to 172.19.14.23(172.19.14.23:3306)? (yes/NO): yes Tue Oct 29 19:40:07 2019 - [info] Checking whether 172.19.14.23(172.19.14.23:3306) is ok for the new master.. Tue Oct 29 19:40:07 2019 - [info] ok. Tue Oct 29 19:40:07 2019 - [info] * Phase 1: Configuration Check Phase completed. Tue Oct 29 19:40:07 2019 - [info] Tue Oct 29 19:40:07 2019 - [debug] Disconnected from 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:40:07 2019 - [info] Phase 2: Rejecting updates Phase.. Tue Oct 29 19:40:07 2019 - [info] Tue Oct 29 19:40:07 2019 - [info] Executing master ip online change script to disable write on the current master: Tue Oct 29 19:40:07 2019 - [info] /usr/local/mha-helper-master/scripts/master_ip_online_change --command=stop --orig_master_host=172.19.14.21 --orig_master_ip=172.19.14.21 --orig_master_port=3306 --orig_master_user='root' --new_master_host=172.19.14.23 --new_master_ip=172.19.14.23 --new_master_port=3306 --new_master_user='root' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_ssh_port=11017 --ssh_options='-o ServerAliveInterval=60 -o ServerAliveCountMax=20 -o StrictHostKeyChecking=no -o ConnectionAttempts=5 -o PasswordAuthentication=no -o BatchMode=yes -i /root/.ssh/id_rsa' --new_master_ssh_port=11017 --orig_master_password=xxx --new_master_password=xxx Usage: master_ip_online_change [options]
master_ip_online_change: error: no such option: --orig_master_ssh_user Tue Oct 29 19:40:07 2019 - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/bin/masterha_master_switch line 53. Tue Oct 29 19:40:07 2019 - [debug] Already disconnected from 172.19.14.21(172.19.14.21:3306) Tue Oct 29 19:40:07 2019 - [debug] Disconnected from 172.19.14.23(172.19.14.23:3306) Tue Oct 29 19:40:07 2019 - [debug] Disconnected from 172.19.14.22(172.19.14.22:3306)