search

jayjanssen / Percona-Pacemaker-Resource-Agents

Pacemaker HA Resource Agents by Percona
28 stars 9 forks source link

node status readable=0 ,after node online -> standby -> online #7

Open zhaoshenju opened 11 years ago

zhaoshenju commented 11 years ago

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:39:09 2013 Stack: openais Current DC: jboss-204 - partition with quorum Version: 1.0.12-unknown 3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-204 reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-205 writer_vip (ocf::heartbeat:IPaddr2): Started jboss-204 Master/Slave Set: ms_MySQL Masters: [ jboss-204 ] Slaves: [ jboss-203 jboss-205 ]

[root@jboss-203 ~]# crm node standby jboss-204 [root@jboss-203 ~]# [root@jboss-203 ~]# crm node online jboss-204 [root@jboss-203 ~]#

after a few minutes.

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:34 2013 Stack: openais Current DC: jboss-204 - partition with quorum Version: 1.0.12-unknown 3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205 reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203 writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203 Master/Slave Set: ms_MySQL Masters: [ jboss-203 ] Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# crm resource migrate reader_vip_1 jboss-204

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:59 2013 Stack: openais Current DC: jboss-204 - partition with quorum Version: 1.0.12-unknown 3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205 reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203 writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203 Master/Slave Set: ms_MySQL Masters: [ jboss-203 ] Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# cibadmin -Q | grep readable | grep nvpair

      <nvpair name="readable" id="status-jboss-204-readable" value="0"/>
      <nvpair id="status-jboss-205-readable" name="readable" value="1"/>

but on node jboss-204, mysql slave is ok,seconds_behind_master=0 why readable=0?

[root@jboss-203 ~]# crm configure show node jboss-203 \ attributes p_mysql_mysql_master_IP="192.168.10.203" \ attributes standby="off" node jboss-204 \ attributes p_mysql_mysql_master_IP="192.168.10.204" \ attributes standby="off" node jboss-205 \ attributes p_mysql_mysql_master_IP="192.168.10.205" \ attributes standby="off" primitive p_mysql ocf:percona:mysql \ params config="/app/mysql/etc/my.cnf" pid="/data/mysql/data/mysql.pid" socket="/data/mysql/tmp/mysql.sock" replication_user="slave" replication_passwd="hGrXFdoEnWnW" max_slave_lag="180" evict_outdated_slaves="false" binary="/app/mysql/bin/mysqld" datadir="/data/mysql/data/" test_user="prm_test_user" test_passwd="XvrhWjtmIhzs" client_binary="/app/mysql/bin/mysql" \ op monitor interval="5s" role="Master" OCF_CHECK_LEVEL="1" \ op monitor interval="2s" role="Slave" OCF_CHECK_LEVEL="1" \ op start interval="0" timeout="120s" \ op stop interval="0" timeout="120s" primitive reader_vip_1 ocf:heartbeat:IPaddr2 \ params ip="192.168.10.101" nic="eth0" \ op monitor interval="10s" primitive reader_vip_2 ocf:heartbeat:IPaddr2 \ params ip="192.168.10.102" nic="eth0" \ op monitor interval="10s" primitive reader_vip_3 ocf:heartbeat:IPaddr2 \ params ip="192.168.10.105" nic="eth0" \ op monitor interval="10s" primitive writer_vip ocf:heartbeat:IPaddr2 \ params ip="192.168.10.107" nic="eth0" \ op monitor interval="10s" ms ms_MySQL p_mysql \ meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1" notify="true" globally-unique="false" target-role="Master" is-managed="true" location loc-No-reader-vip-2 reader_vip_2 \ rule $id="rule-no-reader-vip-2" -inf: readable eq 0 location loc-No-reader-vip-3 reader_vip_3 \ rule $id="rule-no-reader-vip-3" -inf: readable eq 0 location loc-no-reader-vip-1 reader_vip_1 \ rule $id="rule-no-reader-vip-1" -inf: readable eq 0 colocation writer_vip_on_master inf: writer_vip ms_MySQL:Master order ms_MySQL_promote_before_vip inf: ms_MySQL:promote writer_vip:start property $id="cib-bootstrap-options" \ dc-version="1.0.12-unknown" \ cluster-infrastructure="openais" \ expected-quorum-votes="3" \ stonith-enabled="false" \ no-quorum-policy="ignore" property $id="mysql_replication" \ p_mysql_REPL_INFO="192.168.10.203|mysql-bin.000012|1252"

y-trudeau commented 11 years ago

Hi, create the file /tmp/mysql.ocf.ra.debug/log on 204 and look what is going wrong with the monitor operation.

Regards,

Yves

Le 2013-04-09 07:22, zhaoshenju a écrit :

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:39:09 2013 Stack: openais Current DC: jboss-204 - partition with quorum Version: 1.0.12-unknown 3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-204 reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-205 writer_vip (ocf::heartbeat:IPaddr2): Started jboss-204 Master/Slave Set: ms_MySQL Masters: [ jboss-204 ] Slaves: [ jboss-203 jboss-205 ]

[root@jboss-203 ~]# crm node standby jboss-204 [root@jboss-203 ~]# [root@jboss-203 ~]# crm node online jboss-204 [root@jboss-203 ~]#

after a few minutes.

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:34 2013 Stack: openais Current DC: jboss-204 - partition with quorum Version: 1.0.12-unknown 3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205 reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203 writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203 Master/Slave Set: ms_MySQL Masters: [ jboss-203 ] Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# crm resource migrate reader_vip_1 jboss-204

[root@jboss-203 ~]# crm status

Last updated: Fri Mar 29 23:40:59 2013 Stack: openais Current DC: jboss-204 - partition with quorum Version: 1.0.12-unknown 3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-205 reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203 writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203 Master/Slave Set: ms_MySQL Masters: [ jboss-203 ] Slaves: [ jboss-205 jboss-204 ]

[root@jboss-203 ~]# cibadmin -Q | grep readable | grep nvpair

but on node jboss-204, mysql slave is ok,seconds_behind_master=0 why readable=0?

[root@jboss-203 ~]# crm configure show node jboss-203 \ attributes p_mysql_mysql_master_IP="192.168.10.203" \ attributes standby="off" node jboss-204 \ attributes p_mysql_mysql_master_IP="192.168.10.204" \ attributes standby="off" node jboss-205 \ attributes p_mysql_mysql_master_IP="192.168.10.205" \ attributes standby="off" primitive p_mysql ocf:percona:mysql \ params config="/app/mysql/etc/my.cnf" pid="/data/mysql/data/mysql.pid" socket="/data/mysql/tmp/mysql.sock" replication_user="slave" replication_passwd="hGrXFdoEnWnW" max_slave_lag="180" evict_outdated_slaves="false" binary="/app/mysql/bin/mysqld" datadir="/data/mysql/data/" test_user="prm_test_user" test_passwd="XvrhWjtmIhzs" client_binary="/app/mysql/bin/mysql" \ op monitor interval="5s" role="Master" OCF_CHECK_LEVEL="1" \ op monitor interval="2s" role="Slave" OCF_CHECK_LEVEL="1" \ op start interval="0" timeout="120s" \ op stop interval="0" timeout="120s" primitive reader_vip_1 ocf:heartbeat:IPaddr2 \ params ip="192.168.10.101" nic="eth0" \ op monitor interval="10s" primitive reader_vip_2 ocf:heartbeat:IPaddr2 \ params ip="192.168.10.102" nic="eth0" \ op monitor interval="10s" primitive reader_vip_3 ocf:heartbeat:IPaddr2 \ params ip="192.168.10.105" nic="eth0" \ op monitor interval="10s" primitive writer_vip ocf:heartbeat:IPaddr2 \ params ip="192.168.10.107" nic="eth0" \ op monitor interval="10s" ms ms_MySQL p_mysql \ meta master-max="1" master-node-max="1" clone-max="3" clone-node-max="1" notify="true" globally-unique="false" target-role="Master" is-managed="true" location loc-No-reader-vip-2 reader_vip_2 \ rule $id="rule-no-reader-vip-2" -inf: readable eq 0 location loc-No-reader-vip-3 reader_vip_3 \ rule $id="rule-no-reader-vip-3" -inf: readable eq 0 location loc-no-reader-vip-1 reader_vip_1 \ rule $id="rule-no-reader-vip-1" -inf: readable eq 0 colocation writer_vip_on_master inf: writer_vip ms_MySQL:Master order ms_MySQL_promote_before_vip inf: ms_MySQL:promote writer_vip:start property $id="cib-bootstrap-options" \ dc-version="1.0.12-unknown" \ cluster-infrastructure="openais" \ expected-quorum-votes="3" \ stonith-enabled="false" \ no-quorum-policy="ignore" property $id="mysql_replication" \ p_mysql_REPL_INFO="192.168.10.203|mysql-bin.000012|1252"

— Reply to this email directly or view it on GitHub https://github.com/jayjanssen/Percona-Pacemaker-Resource-Agents/issues/7.

zhaoshenju commented 11 years ago

hi,y-trudeau:

Very happy to see your reply!

Not only in the jboss-204 this is the case,in all other slave such as jboss-205 is this

[root@jboss-203 /]# crm status

Last updated: Wed Apr 10 15:50:36 2013 Stack: openais Current DC: jboss-203 - partition with quorum Version: 1.0.12-unknown 3 Nodes configured, 3 expected votes

5 Resources configured.

Online: [ jboss-203 jboss-204 jboss-205 ]

reader_vip_1 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_2 (ocf::heartbeat:IPaddr2): Started jboss-203 reader_vip_3 (ocf::heartbeat:IPaddr2): Started jboss-203 writer_vip (ocf::heartbeat:IPaddr2): Started jboss-203 Master/Slave Set: ms_MySQL Masters: [ jboss-203 ] Slaves: [ jboss-204 jboss-205 ] [root@jboss-203 /]# [root@jboss-203 /]# cibadmin -Q | grep readable | grep nvpair 

      <nvpair id="status-jboss-204-readable" name="readable" value="0"/>
      <nvpair id="status-jboss-205-readable" name="readable" value="0"/>

[root@jboss-203 /]#

I run the "bash -x /usr/lib/ocf/resource.d/percona/mysql monitor" command output is as follows

+/usr/lib/ocf/resource.d/percona/mysql:1508::date Wed Apr 10 16:00:45 CST 2013 +/usr/lib/ocf/resource.d/percona/mysql:1509::echo monitor monitor +/usr/lib/ocf/resource.d/percona/mysql:1510::env +/usr/lib/ocf/resource.d/percona/mysql:1510::grep OCF_ +/usr/lib/ocf/resource.d/percona/mysql:1510::sort +/usr/lib/ocf/resource.d/percona/mysql:1511::set -x +/usr/lib/ocf/resource.d/percona/mysql:1517::case "$1" in +/usr/lib/ocf/resource.d/percona/mysql:1524::mysql_validate +/usr/lib/ocf/resource.d/percona/mysql:945:mysql_validate:check_binary /usr/bin/safe_mysqld /usr/lib/ocf/resource.d/percona/mysql: line 945: check_binary: command not found +/usr/lib/ocf/resource.d/percona/mysql:946:mysql_validate:check_binary mysql /usr/lib/ocf/resource.d/percona/mysql: line 946: check_binary: command not found +/usr/lib/ocf/resource.d/percona/mysql:948:mysql_validate:'[' '!' -f /etc/my.cnf ']' +/usr/lib/ocf/resource.d/percona/mysql:949:mysql_validate:ocf_log err 'Config /etc/my.cnf doesn'\''t exist' /usr/lib/ocf/resource.d/percona/mysql: line 949: ocf_log: command not found +/usr/lib/ocf/resource.d/percona/mysql:950:mysql_validate:return +/usr/lib/ocf/resource.d/percona/mysql:1525::rc=127 +/usr/lib/ocf/resource.d/percona/mysql:1526::LSB_STATUS_STOPPED=3 +/usr/lib/ocf/resource.d/percona/mysql:1527::'[' 127 -ne 0 ']' +/usr/lib/ocf/resource.d/percona/mysql:1528::case "$1" in +/usr/lib/ocf/resource.d/percona/mysql:1530::exit

Whether it is at master or at slave, are not implemented to mysql_monitor function position exit.

However, in the rest of the output of the file /tmp/mysql.ocf.ra.debug/log containing text:

  • __OCF_MSG='MySQL monitor succeeded (master)' (at master) or
  • __OCF_MSG='MySQL monitor succeeded' (at slave)

This indicates function mysql_monitor is executed

I need to provide what information? Look forward to your reply!

zhaoshenju commented 11 years ago 
      <nvpair id="status-jboss-203-readable" name="readable" value="1"/>
      <nvpair id="status-jboss-204-readable" name="readable" value="0"/>
      <nvpair id="status-jboss-205-readable" name="readable" value="0"/>

This is the output of the command "cibadmin -Q | grep readable | grep nvpair"

y-trudeau commented 11 years ago

Have you enable the trace file?

mkdir /tmp/mysql.ocf.ra.debug touch /tmp/mysql.ocf.ra.debug/log

look in the "log" file and see what the monitor op is doing.

Le 2013-04-10 05:34, zhaoshenju a écrit :

| 

   <nvpair id="status-jboss-205-readable" name="readable" value="0"/>

|

This is the output of the command "cibadmin -Q | grep readable | grep nvpair"

— Reply to this email directly or view it on GitHub https://github.com/jayjanssen/Percona-Pacemaker-Resource-Agents/issues/7#issuecomment-16164153.