search

HashDataInc / bireme

Bireme is an incremental synchronization tool for the Greenplum / HashData data warehouse
https://hashdatainc.github.io/bireme/
Apache License 2.0
137 stars 53 forks source link

当同步的表个数过多时,数据不能同步. #88

Open shubifeng opened 6 years ago

shubifeng commented 6 years ago

这些日志完全看不懂

[admin@bireme ~]$ tail -f app/bireme/logs/bireme.err 
18:04:56,062 DEBUG QueuedThreadPool - queue acceptor-0@73163d48
18:04:56,063 INFO  AbstractConnector - Started ServerConnector@1e6a3214{HTTP/1.1,[http/1.1]}{10.62.2.8:9090}
18:04:56,063 DEBUG AbstractLifeCycle - STARTED @9794ms ServerConnector@1e6a3214{HTTP/1.1,[http/1.1]}{10.62.2.8:9090}
18:04:56,063 INFO  Server - Started @9794ms
18:04:56,063 DEBUG AbstractLifeCycle - STARTED @9794ms org.eclipse.jetty.server.Server@49049a04
18:04:56,064 DEBUG QueuedThreadPool - run acceptor-0@73163d48
18:07:47,905 TRACE ConsumerCoordinator - Sending OffsetCommit request with {maxwell_topic-0=OffsetAndMetadata{offset=28341, metadata=''}} to coordinator localhost:9091 (id: 2147483647 rack: null) for group bireme
18:07:47,906 TRACE NetworkClient - Sending OFFSET_COMMIT {group_id=bireme,group_generation_id=-1,member_id=,retention_time=-1,topics=[{topic=maxwell_topic,partitions=[{partition=0,offset=28341,metadata=}]}]} to node 2147483647.
18:07:47,908 TRACE NetworkClient - Completed receive from node 2147483647, for key 8, received {throttle_time_ms=0,responses=[{topic=maxwell_topic,partition_responses=[{partition=0,error_code=0}]}]}
18:07:47,909 DEBUG ConsumerCoordinator - Group bireme committed offset 28341 for partition maxwell_topic-0

[admin@bireme ~]$ tail -f app/bireme/logs/bireme.out 
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_report.report_record_attr.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_campaign.campaign_instance_attr.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_cs.service_order_category.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_auth.privilege_resource.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_cs.device_sales_registration.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_layout.share_info.
18:04:54 INFO  TaskGenerator - TaskGenerator Start.
18:04:54 INFO  ChangeLoader - Loader Start, corresponding table vm_campaign.redpack_record.
18:04:54 INFO  Dispatcher - Dispatcher Start.
18:04:54 INFO  Provider maxwell1 - Provider maxwell1 Start.

===============================================

[admin@bireme ~]$ tail -f app/bireme/logs/bireme.err 
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@3401a114{/vm_store.mijia_order_sku,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@3401a114{/vm_store.mijia_order_sku,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@7e276594
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@7e276594
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@7e276594
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@655ef322{/vm_campaign.campaign_present,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@55562aa9
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@55562aa9
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@55562aa9
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@73173f63{/vm_channel.channel_delivery_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@35e5d0e5
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@35e5d0e5
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@35e5d0e5
18:14:17,050 INFO  ContextHandler - Stopped o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED o.e.j.s.h.ContextHandler@25e2ab5a{/vm_wares.unilife_prod_info_attr,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping o.e.j.s.h.ContextHandler@3d9f6567{/vm_channel.channel_purchase_after_service_info,null,SHUTDOWN}
18:14:17,050 DEBUG AbstractHandler - stopping o.e.j.s.h.ContextHandler@3d9f6567{/vm_channel.channel_purchase_after_service_info,null,UNAVAILABLE}
18:14:17,050 DEBUG AbstractLifeCycle - stopping cn.hashdata.bireme.StateServer$StateHandler@c055c54
18:14:17,050 DEBUG AbstractHandler - stopping cn.hashdata.bireme.StateServer$StateHandler@c055c54
18:14:17,050 DEBUG AbstractLifeCycle - STOPPED cn.hashdata.bireme.StateServer$StateHandler@c055c54

[admin@bireme ~]$ tail -f app/bireme/logs/bireme.err 
......................................
...............................
lue=199 bytes)), (record=DefaultRecord(offset=82670, timestamp=1520391951184, key=72 bytes, value=169 bytes)), (record=DefaultRecord(offset=82671, timestamp=1520391951184, key=72 bytes, value=186 bytes)), (record=DefaultRecord(offset=82672, timestamp=1520391951184, key=72 bytes, value=199 bytes)), (record=DefaultRecord(offset=82673, timestamp=1520391951184, key=72 bytes, value=186 bytes)), (record=DefaultRecord(offset=82674, timestamp=1520391951184, key=72 bytes, value=169 bytes)), (record=DefaultRecord(offset=82675, timestamp=1520391951184, key=72 bytes, value=169 bytes))]}]}]}
18:14:28,424 DEBUG Fetcher - Fetch READ_UNCOMMITTED at offset 78640 for partition maxwell_topic-0 returned fetch data (error=NONE, highWaterMark=572082, lastStableOffset = -1, logStartOffset = 0, abortedTransactions = null, recordsSizeInBytes=1048576)
18:14:28,424 TRACE Fetcher - Preparing to read 1048576 bytes of data for partition maxwell_topic-0 with offset 78640
18:14:28,424 TRACE Fetcher - Updating high watermark for partition maxwell_topic-0 to 572082
18:14:28,425 TRACE Fetcher - Returning fetched records at offset 78640 for assigned partition maxwell_topic-0 and update position to 79140
18:14:28,425 TRACE Fetcher - Returning fetched records at offset 79140 for assigned partition maxwell_topic-0 and update position to 79640
18:14:28,426 TRACE Fetcher - Returning fetched records at offset 79640 for assigned partition maxwell_topic-0 and update position to 80140
[admin@bireme ~]$ tail -f app/bireme/logs/bireme.out
18:15:46 INFO  ChangeLoader - Chang to passimistic mode.
18:16:16 INFO  ChangeLoader - Chang to passimistic mode.
18:16:31 INFO  ChangeLoader - Chang to passimistic mode.
18:16:37 INFO  ChangeLoader - Chang to passimistic mode.
18:16:46 INFO  ChangeLoader - Chang to passimistic mode.
18:16:53 INFO  ChangeLoader - Chang to passimistic mode.
18:16:55 INFO  ChangeLoader - Chang to passimistic mode.
18:17:20 INFO  ChangeLoader - Chang to passimistic mode.
18:17:51 INFO  ChangeLoader - Chang to passimistic mode.
18:17:52 INFO  ChangeLoader - Chang to passimistic mode.
18:19:23 INFO  ChangeLoader - Chang to optimistic mode.
wangzw commented 6 years ago

Several bugs has been identified in v1.0. Please use latest code from master branch. The master branch is currently under stress test and will release soon.

shubifeng commented 6 years ago

@wangzw 非常感谢解答, 2.0版本 计划日期什么时候?

wangzw commented 6 years ago

It is still under stress test and will not release until stable. We currently do not have enough hands and any contribution will be welcomed.

shubifeng commented 6 years ago

@wangzw master代码也还是会出现同样不能同步的问题. 运行一段时间后控制台消息就不动了, 像睡眠了一样. 但比1.0版本还是有所改进.

另外咨询下: 1.同一个表的数据在插入的时候怎么保证先后顺序? 如果插入失败的话, 后面的数据处理机制是怎样的? 2.触发目标库DML操作的阀值条件是什么?