alibaba / MongoShake

MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具,可以满足迁移和同步的需求,进一步实现灾备和多活功能。
GNU General Public License v3.0
1.72k stars 441 forks source link

bug:update语句会报错Retryable writes are not supported #884

Open John-92 opened 2 months ago

John-92 commented 2 months ago

对insert操作是可以同步,但是对update的操作,会报错:db.test.findOneAndUpdate( { _id: ObjectId("66b256226945c1c4cc9f603a") }, { $set: vin: "TESTOTA16952187287"}})

mongoshake 版本2.8.4 souce: 开源 mongodb 4.2 dest AWS mongodb 4.0

John-92 commented 2 months ago

[13:27:42 UTC 2024/08/06] [CRIT] (github.com/alibaba/MongoShake/v2/executor.(*Executor).execute:124) Re player-0, executor-0, oplog for namespace[db_tsp_vehicle_data_a] op[u] failed. error type[mongo.CommandError] error[Retryable writes are not supported], logs number[1], firstLog: {" ts":{"T":1722931671,"I":2},"t":17,"h":0,"v":2,"op":"u","ns":"db_tsp_vehicle_data_integrate.otavehicle data","o":[{"Key":"$set","Value":[{"Key":"pushNum","Value":1},{"Key":"pushRet","Value":"DEVICE_CENTE "},{"Key":"responseResult","Value":""},{"Key":"updateBy","Value":null},{"Key":"updateTime", "Value":"2024-08-06T02Z"}]}],"o2":[{"Key":"_id","Value":""}],"lsid":" A","txnNumber":5,"PrevOpTime":"H","ui":{"Subtype":4,"Data":"BbcLqotD"}}

John-92 commented 2 months ago

一直以为是Retryable writes are not supported的原因,虽然说mongodb4.0不默认开启Retryable writes,然后在连接上增加retryWrites=true,发现于事无补;然后将目标库的对应的表重命名了,倒是会插入数据,但是checkpoint一直没变,大佬们,求助下

John-92 commented 2 months ago

mongoshake 版本2.8.4,配置 conf.version = 10 id = mongoshake master_quorum = false full_sync.http_port = 9101 incr_sync.http_port = 9100 system_profile_port = 9200 log.level = info log.dir = log.file = collector.log log.flush = false

sync_mode = all

mongo_urls = mongodb://xxxx mongo_cs_url = mongo_s_url = mongo_ssl_root_ca_file = tunnel = direct

tunnel.address = mongodb://zxxxxx tunnel.message = raw tunnel.kafka.partition_number = 1 tunnel.json.format = tunnel.mongo_ssl_root_ca_file = mongo_connect_mode = secondaryPreferred

filter.namespace.black =
filter.namespace.white =
filter.pass.special.db = filter.ddl_enable = true filter.oplog.gids = false checkpoint.storage.url = checkpoint.storage.db = mongoshake checkpoint.storage.collection = ckpt_default checkpoint.storage.url.mongo_ssl_root_ca_file = checkpoint.start_position = 1970-01-01T00:00:00Z

transform.namespace =
full_sync.reader.collection_parallel = 6 full_sync.reader.write_document_parallel = 8 full_sync.reader.document_batch_size = 128 full_sync.reader.fetch_batch_size = 8192 full_sync.reader.parallel_thread = 1 full_sync.reader.parallel_index = _id

full_sync.collection_exist_drop = true

full_sync.create_index = foreground full_sync.executor.insert_on_dup_update = false full_sync.executor.filter.orphan_document = false full_sync.executor.majority_enable = false

>= 4.0 change_stream

incr_sync.mongo_fetch_method = oplog incr_sync.change_stream.watch_full_document = false incr_sync.oplog.gids = incr_sync.shard_key = collection incr_sync.shard_by_object_id_whitelist = incr_sync.worker = 8 incr_sync.tunnel.write_thread = 8 incr_sync.target_delay = 0 incr_sync.worker.batch_queue_size = 64 incr_sync.adaptive.batching_max_size = 1024 incr_sync.fetcher.buffer_capacity = 256 incr_sync.reader.fetch_batch_size = 8192

incr_sync.executor.upsert = false

incr_sync.executor.insert_on_dup_update = false
incr_sync.conflict_write_to = none incr_sync.executor.majority_enable = false special.source.db.flag =

John-92 commented 2 months ago

一直以为是Retryable writes are not supported的原因,虽然说mongodb4.0不默认开启Retryable writes,然后在连接上增加retryWrites=true,发现于事无补;然后将目标库的对应的表重命名了,倒是会插入数据,但是checkpoint一直没变,大佬们,求助下

对insert操作是可以同步,但是对update的操作,会报错:db.test.findOneAndUpdate( { _id: ObjectId("66b256226945c1c4cc9f603a") }, { $set: { _id: ObjectId("66b256226945c1c4cc9f603a"), vin: "TESTOTA16952187287"}})