search

caosiyang / py-mongo-sync

Oplog-based data sync tool that synchronizes data from a replica set to another deployment, e.g.: standalone, replica set, and sharded cluster.
http://caosiyang.github.io/py-mongo-sync/
MIT License
108 stars 46 forks source link

数据同步一部分后 程序挂起了 感觉不再同步剩余数据 #29

Closed jren2019 closed 4 years ago

jren2019 commented 4 years ago

大神，麻烦指导一下，这个是什么原因？

caosiyang commented 4 years ago

能提供相关日志吗？是全量同步阶段还是增量同步阶段？

jren2019 commented 4 years ago

[2020-04-27 22:31:09,844] [INFO] [58483] [config:78] >> dst authdb : admin [2020-04-27 22:31:09,845] [INFO] [58483] [config:78] >> dst username : [2020-04-27 22:31:09,845] [INFO] [58483] [config:78] >> dst password : [2020-04-27 22:31:09,858] [INFO] [58483] [config:78] >> dst db version : 4.2.5 [2020-04-27 22:31:09,858] [INFO] [58483] [config:78] >> databases : TEST_DB [2020-04-27 22:31:09,859] [INFO] [58483] [config:78] >> collections : TEST_DB.weather [2020-04-27 22:31:09,859] [INFO] [58483] [config:78] >> db mapping : [2020-04-27 22:31:09,859] [INFO] [58483] [config:78] >> fileds : [2020-04-27 22:31:09,859] [INFO] [58483] [config:78] >> start optime : None [2020-04-27 22:31:09,859] [INFO] [58483] [config:78] >> optime logfile : [2020-04-27 22:31:09,859] [INFO] [58483] [config:78] >> log filepath : sync.log [2020-04-27 22:31:09,859] [INFO] [58483] [config:78] >> pymongo version : 3.10.0 [2020-04-27 22:31:09,860] [INFO] [58483] [config:78] >> ================================================ [2020-04-27 22:31:17,047] [INFO] [58483] [common_syncer:197] >> 236458 TEST_DB.weather [2020-04-27 22:31:17,215] [INFO] [58483] [common_syncer:204] >> large collections: [] [2020-04-27 22:31:17,215] [INFO] [58483] [common_syncer:205] >> small collections: [‘TEST_DB.weather’] [2020-04-27 22:31:41,723] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 10000/236458 [4.23%] [2020-04-27 22:31:42,543] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 20000/236458 [8.46%] [2020-04-27 22:31:43,325] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 30000/236458 [12.69%] [2020-04-27 22:31:44,137] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 40000/236458 [16.92%] [2020-04-27 22:31:44,933] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 50000/236458 [21.15%] [2020-04-27 22:31:45,693] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 60000/236458 [25.37%] [2020-04-27 22:31:46,480] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 70000/236458 [29.60%] [2020-04-27 22:31:47,241] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 80000/236458 [33.83%] [2020-04-27 22:31:48,036] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 90000/236458 [38.06%] [2020-04-27 22:31:48,780] [INFO] [58483] [progress_logger:60] >> TEST_DB.weather 100000/236458 [42.29%] [2020-04-27 22:33:46,448] [INFO] [58483] [common_syncer:82] >> keyboard interrupt [2020-04-27 22:33:46,448] [INFO] [58483] [sync:38] >> exit quitely ...

caosiyang commented 4 years ago

当前代码的find选项与MongoDB 4.2版本不兼容

https://github.com/caosiyang/py-mongo-sync/blob/master/mongosync/mongo/syncer.py#L93

需要改为

cursor = self._src.client()[src_dbname][src_collname].find(no_cursor_timeout=True)