alibaba / MongoShake

MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
GNU General Public License v3.0
1.72k stars 441 forks source link

阿里云sharding to sharding同步,数据全部进入目标sharding的primary shard #485

Closed cau991 closed 3 years ago

cau991 commented 3 years ago

一..说明 mongoshake版本:2.4.16 os:centos7 阿里云sharding版本:3.4 阿里云的sharding到sharding的数据同步,源和目标sharding均未8个shard的集群,源集群3个mongos、目标集群2个mongos。 同步前手动对目标sharding进行了库和部分表的分片,即enableSharding和shardCollection,其它小表未分片。 同步过程无报错信息,从控制台的监控查看,源sharding的数据最终全部都进入到了目标sharding的primary shard中,且同步前手动对目标sharding进行了shardCollection的表均变为未分片,即db.table.status()为"sharded" : false。 查看日志发现了

[INFO] replication from [sharding] to [replica]

麻烦帮忙看下

二..配置参数如下 对源集群已经在控制台申请过CS和Shard的地址。

conf.version = 6 id = mongoshake master_quorum = false full_sync.http_port = 9111 incr_sync.http_port = 9110 system_profile_port = 9210 log.level = info log.dir = /var/log/mongoshake log.file = collector.log log.flush = false sync_mode = all mongo_urls = mongodb://dba_sourcesync:+++@d-+++.mongodb.rds.aliyuncs.com:3717,d-+++.mongodb.rds.aliyuncs.com:3717;mongodb://dba_sourcesync:+++@d-+++.mongodb.rds.aliyuncs.com:3717,d-+++.mongodb.rds.aliyuncs.com:3717 等共计8个 mongo_cs_url = mongodb://dba_sourcesync:+++@dds-+++-cs380.mongodb.rds.aliyuncs.com:3717,dds-+++-cs968.mongodb.rds.aliyuncs.com:3717 mongo_s_url = mongodb://dba_sourcesync:+++@s-+++.mongodb.rds.aliyuncs.com:3717,s-+++.mongodb.rds.aliyuncs.com:3717,s-+++.mongodb.rds.aliyuncs.com:3717 mongo_connect_mode = secondaryPreferred tunnel = direct tunnel.address = mongodb://dba_destsync:+++@s-+++.mongodb.rds.aliyuncs.com:3717,s-+++.mongodb.rds.aliyuncs.com:3717 tunnel.message = raw filter.namespace.black = filter.namespace.white = filter.pass.special.db = filter.ddl_enable = false checkpoint.storage.url = checkpoint.storage.db = mongoshake checkpoint.storage.collection = ckpt_default checkpoint.start_position = 1970-01-01T00:00:00Z transform.namespace = full_sync.reader.collection_parallel = 6 full_sync.reader.write_document_parallel = 8 full_sync.reader.document_batch_size = 128 full_sync.reader.read_document_count = 200000 full_sync.collection_exist_drop = true full_sync.create_index = foreground full_sync.executor.insert_on_dup_update = false full_sync.executor.filter.orphan_document = false full_sync.executor.majority_enable = false incr_sync.mongo_fetch_method = oplog incr_sync.target_delay = 0 incr_sync.oplog.gids = incr_sync.shard_key = collection incr_sync.shard_by_object_id_whitelist = incr_sync.worker = 8 incr_sync.worker.oplog_compressor = none incr_sync.worker.batch_queue_size = 64 incr_sync.adaptive.batching_max_size = 1024 incr_sync.fetcher.buffer_capacity = 256 incr_sync.executor.upsert = false incr_sync.executor.insert_on_dup_update = false incr_sync.conflict_write_to = db incr_sync.executor.majority_enable = false incr_sync.change_stream.watch_full_document = false

cau991 commented 3 years ago

总结下,是目标sharding的账号权限问题,出了需要目标库的读写权限外,还需要clusterManager角色