search

epermana / tungsten-replicator

Automatically exported from code.google.com/p/tungsten-replicator
1 stars 0 forks source link

multi_trepctl misses some fan-in datasources after update #1051

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?

1. install a fan-in  topology using tungsten-replicator-2.2.1
./tools/tpm install fan_in \
    --reset \
    --replication-user=tungsten \
    --replication-password=secret \
    --home-directory=/home/tungsten/installs/cookbook \
    --repl-disable-relay-logs=true \
    --masters=host1,host2,host3 \
    --slaves=host4 \
    --master-services=alpha,bravo,charlie \
    --topology=fan-in \
    --start
2. run multi_trepctl

/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/scripts/multi_trep
ctl
| host  | servicename | role   | state  | appliedlastseqno | appliedlatency |
| host1 | alpha       | master | ONLINE |                0 |          0.239 |
| host2 | bravo       | master | ONLINE |                0 |          0.555 |
| host3 | charlie     | master | ONLINE |                0 |          0.482 |
| host4 | alpha       | slave  | ONLINE |                0 |         12.896 |
| host4 | bravo       | slave  | ONLINE |                0 |          8.896 |
| host4 | charlie     | slave  | ONLINE |                0 |          4.894 |

3. unpack the latest tungsten-replicator 3.0.0 and run an upgrade
./tools/tpm update  --hosts=host1,host2,host3,host4 
--directory=/home/tungsten/installs/cookbook

# (note: I get the same result with CONTINUENT_PROFILES+ update or with tpm 
fetch + update)

4. run multi_trepctl again

What is the expected output?

The same output that we had before updating

What do you see instead?

Some masters are missing:

/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/scripts/multi_trep
ctl
| host  | servicename | role   | state  | appliedlastseqno | appliedlatency |
| host1 | alpha       | master | ONLINE |                2 |          0.038 |
| host4 | alpha       | slave  | ONLINE |                2 |          4.252 |
| host4 | bravo       | slave  | ONLINE |                2 |          0.898 |
| host4 | charlie     | slave  | ONLINE |                2 |          1.122 |

What is the possible cause?

N/A

What is the proposed solution?

N/A

Original issue reported on code.google.com by g.maxia on 29 Oct 2014 at 12:25

GoogleCodeExporter commented 9 years ago 
After some more tests, I see that multi_trepctl behaves in the same erratic way 
even with a fresh installation using TR 3.0. This is a regression

Original comment by g.maxia on 29 Oct 2014 at 12:47

GoogleCodeExporter commented 9 years ago

Original comment by jeff.m...@continuent.com on 29 Oct 2014 at 12:49

GoogleCodeExporter commented 9 years ago 
Please run `multi_trepctl -v` and post the output.

Original comment by jeff.m...@continuent.com on 29 Oct 2014 at 1:16

GoogleCodeExporter commented 9 years ago 
multi_trepctl -v
DEBUG >> Execute `cat /home/tungsten/installs/cookbook/tungsten/.lock`
DEBUG >> RC: 0, Result: /home/tungsten/installs/cookbook, No STDERR
DEBUG >> Initialize TungstenInstall from /home/tungsten/installs/cookbook
DEBUG >> Execute `/home/tungsten/installs/cookbook/tungsten/tools/tpm query 
staging`
DEBUG >> RC: 0, Result: 
tungsten@ubuntu1:/home/tungsten/tinstall/tungsten-replicator-3.0.0-497, No 
STDERR
DEBUG >> Execute `/home/tungsten/installs/cookbook/tungsten/tools/tpm query 
values user host_name host_enable_replicator host_enable_manager 
host_enable_connector repl_rmi_port mgr_rmi_port mgr_api mgr_api_port 
mgr_api_address preferred_path`
DEBUG >> RC: 0, Result: {
  "host_enable_connector": "false",
  "host_enable_manager": "false",
  "host_enable_replicator": "true",
  "host_name": "host1",
  "mgr_api": "true",
  "mgr_api_address": "127.0.0.1",
  "mgr_api_port": "8090",
  "mgr_rmi_port": "9997",
  "preferred_path": "",
  "repl_rmi_port": "10000",
  "user": "tungsten"
}, No STDERR
DEBUG >> Begin 
/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/scripts/multi_trep
ctl -v
DEBUG >> Command:
DEBUG >> Options:
DEBUG >>     validate => false
DEBUG >>     autocomplete => false
DEBUG >>     sort_by => host,role,serviceName
DEBUG >>     by_service => false
DEBUG >>     hide_headers => false
DEBUG >> Execute `ps -eo args | grep "Tungsten Replicator"`
DEBUG >> RC: 0, Result: 
/home/tungsten/installs/cookbook/releases/tungsten-replicator-3.0.0-497_pid94836
/tungsten-replicator/bin/../../cluster-home/bin/wrapper-linux-x86-64 
/home/tungsten/installs/cookbook/releases/tungsten-replicator-3.0.0-497_pid94836
/tungsten-replicator/bin/../conf/wrapper.conf wrapper.syslog.ident=treplicator 
wrapper.pidfile=/home/tungsten/installs/cookbook/releases/tungsten-replicator-3.
0.0-497_pid94836/tungsten-replicator/bin/../var/treplicator.pid 
wrapper.name=treplicator wrapper.displayname=Tungsten Replicator Service 
wrapper.daemonize=TRUE 
wrapper.anchorfile=/home/tungsten/installs/cookbook/releases/tungsten-replicator
-3.0.0-497_pid94836/tungsten-replicator/bin/../var/treplicator.anchor 
wrapper.ignore_signals=TRUE 
wrapper.statusfile=/home/tungsten/installs/cookbook/releases/tungsten-replicator
-3.0.0-497_pid94836/tungsten-replicator/bin/../var/treplicator.status 
wrapper.java.statusfile=/home/tungsten/installs/cookbook/releases/tungsten-repli
cator-3.0.0-497_pid94836/tungsten-replicator/bin/../var/treplicator.java.status 
wrapper.script.version=3.5.17 --
sh -c export LANG=en_US; ps -eo args | grep "Tungsten Replicator"
grep Tungsten Replicator, No STDERR
DEBUG >> Execute `/home/tungsten/installs/cookbook/tungsten/tools/tpm query 
dataservices | awk -F ":" '{print $1}' | tr -d " "`
DEBUG >> RC: 0, Result: alpha, No STDERR
DEBUG >> Execute `/home/tungsten/installs/cookbook/tungsten/tools/tpm query 
values dataservices.alpha.dataservice_replication_members`
DEBUG >> RC: 0, Result: {
  "dataservices.alpha.dataservice_replication_members": "host1,host4"
}, No STDERR
DEBUG >> Search ifconfig for 192.168.194.129
DEBUG >> Search ifconfig for 192.168.194.133
DEBUG >> Execute `test -f 
/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/bin/trepctl`
DEBUG >> RC: 0, Result: , No STDERR
DEBUG >> Execute `test -f 
/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/bin/trepctl` on 
host4 as tungsten
DEBUG >> RC: 0, Result:
DEBUG >> Search ifconfig for 192.168.194.129
DEBUG >> Search ifconfig for 192.168.194.133
DEBUG >> Execute 
`/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/bin/trepctl  
services -full -json`
DEBUG >> Execute 
`/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/bin/trepctl  
services -full -json` on host4 as tungsten
DEBUG >> RC: 0, Result: [
{"masterListenUri":"thl:\/\/host1:2112\/","clusterName":"alpha","uptimeSeconds":
"75.476","pendingExceptionMessage":"NONE","autoRecoveryEnabled":"false","applied
LastEventId":"mysql-bin.000002:0000000000000418;3964","pendingError":"NONE","res
ourcePrecedence":"99","transitioningTo":"","offlineRequests":"NONE","state":"ONL
INE","simpleServiceName":"alpha","extensions":"","pendingErrorEventId":"NONE","s
erviceName":"alpha","sourceId":"host1","version":"Tungsten Replicator 3.0.0 
build 
497","role":"master","currentTimeMillis":"1414589980835","masterConnectUri":"thl
:\/\/localhost:\/","rmiPort":"10000","siteName":"default","pendingErrorSeqno":"-
1","pipelineSource":"\/var\/lib\/mysql","appliedLatency":"0.19","host":"host1","
pendingErrorCode":"NONE","latestEpochNumber":"0","maximumStoredSeqNo":"0","chann
els":"1","appliedLastSeqno":"0","serviceType":"local","autoRecoveryTotal":"0","s
eqnoType":"java.lang.Long","currentEventId":"mysql-bin.000002:0000000000000418",
"minimumStoredSeqNo":"0","relativeLatency":"74.835","timeInStateSeconds":"74.734
","started":"true","dataServerHost":"host1","useSSLConnection":"false"}

], No STDERR
DEBUG >> RC: 0, Result: [
{"masterListenUri":"thl:\/\/host4:2112\/","clusterName":"alpha","uptimeSeconds":
"67.214","pendingExceptionMessage":"NONE","autoRecoveryEnabled":"false","applied
LastEventId":"mysql-bin.000002:0000000000000418;3964","pendingError":"NONE","res
ourcePrecedence":"99","transitioningTo":"","offlineRequests":"NONE","state":"ONL
INE","simpleServiceName":"alpha","extensions":"","pendingErrorEventId":"NONE","s
erviceName":"alpha","sourceId":"host4","version":"Tungsten Replicator 3.0.0 
build 
497","role":"slave","currentTimeMillis":"1414589981020","masterConnectUri":"thl:
\/\/host1:2112\/","rmiPort":"10000","siteName":"default","pendingErrorSeqno":"-1
","pipelineSource":"thl:\/\/host1:2112\/","appliedLatency":"8.995","host":"host4
","pendingErrorCode":"NONE","latestEpochNumber":"0","maximumStoredSeqNo":"0","ch
annels":"1","appliedLastSeqno":"0","serviceType":"remote","autoRecoveryTotal":"0
","seqnoType":"java.lang.Long","currentEventId":"NONE","minimumStoredSeqNo":"0",
"relativeLatency":"75.02","timeInStateSeconds":"66.306","started":"true","dataSe
rverHost":"host4","useSSLConnection":"false"}
,
{"masterListenUri":"thl:\/\/host4:2112\/","clusterName":"bravo","uptimeSeconds":
"66.399","pendingExceptionMessage":"NONE","autoRecoveryEnabled":"false","applied
LastEventId":"mysql-bin.000002:0000000000000418;3857","pendingError":"NONE","res
ourcePrecedence":"99","transitioningTo":"","offlineRequests":"NONE","state":"ONL
INE","simpleServiceName":"bravo","extensions":"","pendingErrorEventId":"NONE","s
erviceName":"bravo","sourceId":"host4","version":"Tungsten Replicator 3.0.0 
build 
497","role":"slave","currentTimeMillis":"1414589981026","masterConnectUri":"thl:
\/\/host2:2112\/","rmiPort":"10000","siteName":"default","pendingErrorSeqno":"-1
","pipelineSource":"thl:\/\/host2:2112\/","appliedLatency":"1.483","host":"host4
","pendingErrorCode":"NONE","latestEpochNumber":"0","maximumStoredSeqNo":"0","ch
annels":"1","appliedLastSeqno":"0","serviceType":"remote","autoRecoveryTotal":"0
","seqnoType":"java.lang.Long","currentEventId":"NONE","minimumStoredSeqNo":"0",
"relativeLatency":"63.026","timeInStateSeconds":"61.616","started":"true","dataS
erverHost":"host4","useSSLConnection":"false"}
,
{"masterListenUri":"thl:\/\/host4:2112\/","clusterName":"charlie","uptimeSeconds
":"66.156","pendingExceptionMessage":"NONE","autoRecoveryEnabled":"false","appli
edLastEventId":"mysql-bin.000002:0000000000000424;4139","pendingError":"NONE","r
esourcePrecedence":"99","transitioningTo":"","offlineRequests":"NONE","state":"O
NLINE","simpleServiceName":"charlie","extensions":"","pendingErrorEventId":"NONE
","serviceName":"charlie","sourceId":"host4","version":"Tungsten Replicator 
3.0.0 build 
497","role":"slave","currentTimeMillis":"1414589981031","masterConnectUri":"thl:
\/\/host3:2112\/","rmiPort":"10000","siteName":"default","pendingErrorSeqno":"-1
","pipelineSource":"thl:\/\/host3:2112\/","appliedLatency":"5.21","host":"host4"
,"pendingErrorCode":"NONE","latestEpochNumber":"0","maximumStoredSeqNo":"0","cha
nnels":"1","appliedLastSeqno":"0","serviceType":"remote","autoRecoveryTotal":"0"
,"seqnoType":"java.lang.Long","currentEventId":"NONE","minimumStoredSeqNo":"0","
relativeLatency":"71.031","timeInStateSeconds":"65.896","started":"true","dataSe
rverHost":"host4","useSSLConnection":"false"}

]
| host  | servicename | role   | state  | appliedlastseqno | appliedlatency |
| host1 | alpha       | master | ONLINE |                0 |          0.190 |
| host4 | alpha       | slave  | ONLINE |                0 |          8.995 |
| host4 | bravo       | slave  | ONLINE |                0 |          1.483 |
| host4 | charlie     | slave  | ONLINE |                0 |          5.210 |
DEBUG >> Finish 
/home/tungsten/installs/cookbook/tungsten/tungsten-replicator/scripts/multi_trep
ctl -v
DEBUG >> RC: 0

Original comment by g.maxia on 29 Oct 2014 at 1:40

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 19 Dec 2014 at 7:03

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 19 Jan 2015 at 2:18