search

danielcheng007 / tungsten-replicator

Automatically exported from code.google.com/p/tungsten-replicator
0 stars 0 forks source link

in a star topology, heartbeat from a spoke causes random high latency #632

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?

1. install a star topology
2. in a spoke, run a 'heartbeat' in the master service
3. look at the hub service status

What is the expected output?

all services have a sub-second latency

What do you see instead?

The latency for the hub service goes to a random number, usually higher than 
1,000

What is the possible cause?

N/A

What is the proposed solution?

N/A

Original issue reported on code.google.com by g.maxia on 9 Jul 2013 at 11:08

GoogleCodeExporter commented 9 years ago 
Here's an example of the heartbeat as it is originated in a spoke (service 
bravo) and picked up in the hub (service charlie)
 ~/tsb2/db3/thl -service bravo list -seqno 6
SEQ# = 6 / FRAG# = 0 (last frag)
- TIME = 2013-07-09 12:54:30.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000001439;0
- SOURCEID = 127.0.0.1
- METADATA = 
[mysql_server_id=102;dbms_type=mysql;is_metadata=true;service=bravo;shard=tungst
en_bravo;heartbeat=NONE]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', 
character_set_client = 33, collation_connection = 33, collation_server = 8]
- SCHEMA = tungsten_bravo
- SQL(0) = UPDATE tungsten_bravo.heartbeat SET source_tstamp= '2013-07-09 
12:54:30', salt= 3, name= 'NONE'  WHERE id= 1 /* ___SERVICE___ = [bravo] */
________________________________________________________________________________
________________________________________________________________________________
_________________________________  ~/tsb2/db3/thl -service charlie list -seqno 
55
SEQ# = 55 / FRAG# = 0 (last frag)
- TIME = 2013-07-09 12:54:30.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000000019039;0
- SOURCEID = 127.0.0.1
- METADATA = 
[mysql_server_id=103;dbms_type=mysql;is_metadata=true;service=bravo;shard=tungst
en_bravo]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', 
character_set_client = 33, collation_connection = 33, collation_server = 8]
- SCHEMA = tungsten_bravo
- SQL(0) = UPDATE tungsten_bravo.heartbeat SET source_tstamp= '2013-07-09 
12:54:30', salt= 3, name= 'NONE'  WHERE id= 1 /* ___SERVICE___ = [bravo] */
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', 
character_set_client = 33, collation_connection = 33, collation_server = 8]
- SQL(1) = UPDATE tungsten_bravo.trep_commit_seqno SET seqno=6, fragno=0, 
last_frag=1, source_id='127.0.0.1', epoch_number=0, 
eventid='mysql-bin.000002:0000000000001439;0', applied_latency=1, 
update_timestamp='2013-07-09 12:54:31', shard_id='tungsten_bravo', 
extract_timestamp='2013-07-09 12:54:30' WHERE task_id=0
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', 
character_set_client = 33, collation_connection = 33, collation_server = 8]
- SQL(2) = UPDATE tungsten_bravo.heartbeat SET seqno= 6, eventid= 
'mysql-bin.000002:0000000000001439;0', target_tstamp= '2013-07-09 12:54:31', 
lag_millis= 1085  WHERE id= 1

Original comment by g.maxia on 9 Jul 2013 at 11:14

GoogleCodeExporter commented 9 years ago 
Can you add related `trepctl status` output where latency is visible?

Original comment by linas.vi...@continuent.com on 9 Jul 2013 at 11:32

GoogleCodeExporter commented 9 years ago 
Here's a full set:

trepctl -service delta status
Processing status command...
NAME                     VALUE
----                     -----
appliedLastEventId     : mysql-bin.000002:0000000012254995;0
appliedLastSeqno       : 45029
appliedLatency         : 0.187
channels               : 1
clusterName            : default
currentEventId         : mysql-bin.000002:0000000012254995
currentTimeMillis      : 1373371035465
dataServerHost         : qa.r4.continuent.com
extensions             :
latestEpochNumber      : 0
masterConnectUri       :
masterListenUri        : thl://qa.r4.continuent.com:12333/
maximumStoredSeqNo     : 45029
minimumStoredSeqNo     : 0
offlineRequests        : NONE
pendingError           : NONE
pendingErrorCode       : NONE
pendingErrorEventId    : NONE
pendingErrorSeqno      : -1
pendingExceptionMessage: NONE
pipelineSource         : /home/jenkins/sandboxes/repl_test/data
relativeLatency        : 6072.465
resourcePrecedence     : 99
rmiPort                : 10333
role                   : master
seqnoType              : java.lang.Long
serviceName            : delta
serviceType            : local
simpleServiceName      : delta
siteName               : default
sourceId               : qa.r4.continuent.com
state                  : ONLINE
timeInStateSeconds     : 12670.641
transitioningTo        :
uptimeSeconds          : 12670.864
version                : Tungsten Replicator 2.1.1 build 73
Finished status command...

trepctl -service charlie status
Processing status command...
NAME                     VALUE
----                     -----
appliedLastEventId     : mysql-bin.000002:0000000029976196;0
appliedLastSeqno       : 26252
appliedLatency         : 11794.469
channels               : 1
clusterName            : default
currentEventId         : NONE
currentTimeMillis      : 1373371042430
dataServerHost         : qa.r4.continuent.com
extensions             :
latestEpochNumber      : 0
masterConnectUri       : thl://qa.r3.continuent.com:12333/
masterListenUri        : thl://qa.r4.continuent.com:2112/
maximumStoredSeqNo     : 26252
minimumStoredSeqNo     : 0
offlineRequests        : NONE
pendingError           : NONE
pendingErrorCode       : NONE
pendingErrorEventId    : NONE
pendingErrorSeqno      : -1
pendingExceptionMessage: NONE
pipelineSource         : thl://qa.r3.continuent.com:12333/
relativeLatency        : 12313.43
resourcePrecedence     : 99
rmiPort                : 10333
role                   : slave
seqnoType              : java.lang.Long
serviceName            : charlie
serviceType            : remote
simpleServiceName      : charlie
siteName               : default
sourceId               : qa.r4.continuent.com
state                  : ONLINE
timeInStateSeconds     : 12677.511
transitioningTo        :
uptimeSeconds          : 12677.607
version                : Tungsten Replicator 2.1.1 build 73
Finished status command...

thl -service delta list -seqno 45029
SEQ# = 45029 / FRAG# = 0 (last frag)
- TIME = 2013-07-09 12:16:03.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000012254995;0
- SOURCEID = qa.r4.continuent.com
- METADATA = 
[mysql_server_id=40;dbms_type=mysql;is_metadata=true;service=delta;shard=tungste
n_delta;heartbeat=NONE]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- SQL(0) =
 - ACTION = UPDATE
 - SCHEMA = tungsten_delta
 - TABLE = heartbeat
 - ROW# = 0
  - COL(1: ) = 1
  - COL(2: ) = NULL
  - COL(3: ) = NULL
  - COL(4: ) = 2013-07-09 12:16:03.0
  - COL(5: ) = NULL
  - COL(6: ) = NULL
  - COL(7: ) = 30
  - COL(8: ) = [B@29c56c60
  - KEY(1: ) = 1

thl -service charlie list -seqno 26252
SEQ# = 26252 / FRAG# = 0 (last frag)
- TIME = 2013-07-09 10:32:09.0
- EPOCH# = 0
- EVENTID = mysql-bin.000002:0000000029976196;0
- SOURCEID = qa.r3.continuent.com
- METADATA = 
[mysql_server_id=30;dbms_type=mysql;is_metadata=true;service=alpha;shard=tungste
n_alpha]
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', 
character_set_client = 33, collation_connection = 33, collation_server = 8]
- SCHEMA =
- SQL(0) = UPDATE `tungsten_alpha`.`heartbeat` SET `id` =  1  , `seqno` =  NULL 
 , `eventid` =  NULL  , `source_tstamp` =  '2013-07-09 13:48:43'  , 
`target_tstamp` =  NULL  , `lag_millis` =  NULL  , `salt` =  32  , `name` =  
UNHEX( '4e4f4e45' )  WHERE `id` =  1 /* ___SERVICE___ = [alpha] */
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', 
character_set_client = 33, collation_connection = 33, collation_server = 8]
- SQL(1) = UPDATE tungsten_alpha.trep_commit_seqno SET seqno=26905, fragno=0, 
last_frag=1, source_id='qa.r1.continuent.com', epoch_number=0, 
eventid='mysql-bin.000002:0000000007131649;0', applied_latency=0, 
update_timestamp='2013-07-09 13:48:43', shard_id='tungsten_alpha', 
extract_timestamp='2013-07-09 13:48:43' WHERE task_id=0
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign_key_checks = 1, unique_checks = 1, sql_mode = 'IGNORE_SPACE', 
character_set_client = 33, collation_connection = 33, collation_server = 8]
- SQL(2) = UPDATE tungsten_alpha.heartbeat SET seqno= 26905, eventid= 
'mysql-bin.000002:0000000007131649;0', target_tstamp= '2013-07-09 13:48:43', 
lag_millis= 401  WHERE id= 1

Original comment by g.maxia on 9 Jul 2013 at 12:00

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 30 Jul 2013 at 1:51

GoogleCodeExporter commented 9 years ago 
Moving to 2.1.2

Original comment by linas.vi...@continuent.com on 21 Aug 2013 at 12:59

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 26 Aug 2013 at 1:54

GoogleCodeExporter commented 9 years ago 
There won't be a 2.1.3.

Original comment by linas.vi...@continuent.com on 17 Sep 2013 at 10:13

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 23 Dec 2013 at 9:51

GoogleCodeExporter commented 9 years ago 
postponing

Original comment by stephane...@continuent.com on 30 Jan 2014 at 9:19

GoogleCodeExporter commented 9 years ago

Original comment by stephane...@continuent.com on 5 Aug 2014 at 9:15

GoogleCodeExporter commented 9 years ago

Original comment by linas.vi...@continuent.com on 19 Jan 2015 at 2:18