search

caosiyang / py-mongo-sync

Oplog-based data sync tool that synchronizes data from a replica set to another deployment, e.g.: standalone, replica set, and sharded cluster.
http://caosiyang.github.io/py-mongo-sync/
MIT License
108 stars 46 forks source link

sync is have error: ERROR no such item for Cursor instance #7

Closed supersujj closed 7 years ago

supersujj commented 7 years ago

rs1:PRIMARY> rs.status() { "set" : "rs1", "date" : ISODate("2017-09-05T04:48:36.289Z"), "myState" : 1, "term" : NumberLong(1), "heartbeatIntervalMillis" : NumberLong(2000), "optimes" : { "lastCommittedOpTime" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) }, "appliedOpTime" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) }, "durableOpTime" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) } }, "members" : [ { "_id" : 0, "name" : "172.16.10.115:27021", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 5482, "optime" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2017-09-05T04:48:26Z"), "electionTime" : Timestamp(1504582325, 1), "electionDate" : ISODate("2017-09-05T03:32:05Z"), "configVersion" : 1, "self" : true }, { "_id" : 1, "name" : "172.16.10.115:27022", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4603, "optime" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) }, "optimeDurable" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2017-09-05T04:48:26Z"), "optimeDurableDate" : ISODate("2017-09-05T04:48:26Z"), "lastHeartbeat" : ISODate("2017-09-05T04:48:35.206Z"), "lastHeartbeatRecv" : ISODate("2017-09-05T04:48:36.119Z"), "pingMs" : NumberLong(0), "syncingTo" : "172.16.10.115:27023", "configVersion" : 1 }, { "_id" : 2, "name" : "172.16.10.115:27023", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 4603, "optime" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) }, "optimeDurable" : { "ts" : Timestamp(1504586906, 1), "t" : NumberLong(1) }, "optimeDate" : ISODate("2017-09-05T04:48:26Z"), "optimeDurableDate" : ISODate("2017-09-05T04:48:26Z"), "lastHeartbeat" : ISODate("2017-09-05T04:48:35.206Z"), "lastHeartbeatRecv" : ISODate("2017-09-05T04:48:35.236Z"), "pingMs" : NumberLong(0), "syncingTo" : "172.16.10.115:27021", "configVersion" : 1 } ], "ok" : 1 }

but when i insert some recond,and use command: python sync.py --from 172.16.10.115:27021 --src-authdb admin --src-username admin --src-password xxx --to 172.16.10.116:27021 --dst-authdb admin --dst-username admin --dst-password xxx --dbs test --start-optime 1504586906 --log /root/test.log

2017-09-05 12:53:48,663 INFO src hostportstr : 172.16.10.115:27021 2017-09-05 12:53:48,663 INFO src engine : mongodb 2017-09-05 12:53:48,663 INFO src authdb : admin 2017-09-05 12:53:48,663 INFO src username : admin 2017-09-05 12:53:48,664 INFO src password : admin 2017-09-05 12:53:48,664 INFO dst hostportstr : 172.16.10.116:27021 2017-09-05 12:53:48,664 INFO dst authdb : admin 2017-09-05 12:53:48,665 INFO dst username : admin 2017-09-05 12:53:48,665 INFO dst password : admin 2017-09-05 12:53:48,665 INFO databases : ['test'] 2017-09-05 12:53:48,666 INFO collections : [] 2017-09-05 12:53:48,666 INFO src db :
2017-09-05 12:53:48,666 INFO dst db :
2017-09-05 12:53:48,667 INFO start optime : 1504586906 2017-09-05 12:53:48,667 INFO log filepath : /root/test.log 2017-09-05 12:53:48,667 INFO pymongo version : 3.5.1 2017-09-05 12:53:48,668 INFO ================================================ 2017-09-05 12:53:48,805 INFO locating oplog, it will take a while 2017-09-05 12:53:48,807 INFO start timestamp is Timestamp(1504586906, 1) actually 2017-09-05 12:53:48,808 INFO try to sync oplog from Timestamp(1504586906, 1) on 172.16.10.115:27021 2017-09-05 12:53:48,810 ERROR no such item for Cursor instance 2017-09-05 12:53:48,810 ERROR Timestamp(1504586906, 1) not found, terminate 2017-09-05 12:53:48,811 INFO exit

caosiyang commented 7 years ago

It's an incompatible problem of pymongo when access MongoDB v3.4 . The tailable cursor cannot get document through __getitem__(index), e.g. cursor[0]. The solution is replcace all of cursor[0] with cursor.next().

supersujj commented 7 years ago

OK,ths!