alibaba / MongoShake

MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
GNU General Public License v3.0
1.7k stars 442 forks source link

配置白名单 applyOps类型无法过滤 #789

Open liuxinwang opened 1 year ago

liuxinwang commented 1 year ago

问题:配置了白名单,还是有其它库表数据发送到kafka,debug观察日志: [2023/02/12 20:00:45 CST] [DEBG] NamespaceFilter check oplog:[{applyOps [[{op u} {ns platform_core_order.order_after_sale} {ui......

版本:v2.8.2

配置文件:

conf.version = 10
id = mongoshake
master_quorum = true
full_sync.http_port = 9101
incr_sync.http_port = 9100
system_profile_port = 9200
log.level = debug
log.dir =
log.file = collector.log
log.flush = false
sync_mode = incr
mongo_urls = mongodb://root:xxxxxx@xxxxxx.mongodb.rds.aliyuncs.com:3717,xxxxxx.mongodb.rds.aliyuncs.com:3717/

mongo_cs_url =
mongo_s_url = 
mongo_ssl_root_ca_file = 

tunnel = kafka
tunnel.address = mongo_oplog_local_dev@xxxxxx:9092
tunnel.message = json
tunnel.kafka.partition_number = 12
tunnel.json.format =
tunnel.mongo_ssl_root_ca_file =

mongo_connect_mode = secondaryPreferred

filter.namespace.black =
filter.namespace.white = dba_test
filter.pass.special.db =
filter.ddl_enable = false
filter.oplog.gids = false

checkpoint.storage.url =
checkpoint.storage.db = mongoshake_dev
checkpoint.storage.collection = ckpt_default
checkpoint.storage.url.mongo_ssl_root_ca_file = 
checkpoint.start_position = 2023-02-08T00:03:51Z

transform.namespace =

full_sync.reader.collection_parallel = 6
full_sync.reader.write_document_parallel = 8
full_sync.reader.document_batch_size = 128
full_sync.reader.fetch_batch_size = 8192
full_sync.reader.parallel_thread = 1
full_sync.reader.parallel_index = _id
full_sync.collection_exist_drop = true
full_sync.create_index = none
full_sync.executor.insert_on_dup_update = false
full_sync.executor.filter.orphan_document = false
full_sync.executor.majority_enable = false

incr_sync.mongo_fetch_method = oplog
incr_sync.change_stream.watch_full_document = false
incr_sync.oplog.gids =
incr_sync.shard_key = auto
incr_sync.shard_by_object_id_whitelist = 
incr_sync.worker = 12
incr_sync.tunnel.write_thread = 12
incr_sync.target_delay = 0
incr_sync.worker.batch_queue_size = 64
incr_sync.adaptive.batching_max_size = 1024
incr_sync.fetcher.buffer_capacity = 256
incr_sync.reader.fetch_batch_size = 8192
incr_sync.executor.upsert = false
incr_sync.executor.insert_on_dup_update = false
incr_sync.conflict_write_to = none
incr_sync.executor.majority_enable = false
special.source.db.flag =
skip.nsshardkey.verify =
zhangst commented 1 year ago

applyOps是嵌套的,之后版本修正

datuxing commented 9 months ago

请问这个修复了吗,配置白名单,仍然会同步其他db