search

caosiyang / py-mongo-sync

Oplog-based data sync tool that synchronizes data from a replica set to another deployment, e.g.: standalone, replica set, and sharded cluster.
http://caosiyang.github.io/py-mongo-sync/
MIT License
108 stars 46 forks source link

UserWarning: MongoClient opened before fork. #25

Closed willli666 closed 4 years ago

willli666 commented 5 years ago

复制replica set到最后出现这样一个warning，请问是否需要担心？

[ OK ]  [87/88] parse_apollobox._User   360894/360894   325.8s
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forkin
g. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
  "MongoClient opened before fork. Create MongoClient only "
[ OK ]  [88/88] parse_apollobox.UserProduct     1459494/1459494 150.6s
caosiyang commented 5 years ago

程序里将100w条以上的集合视为large_collection，采用多进程同步。 这个错误此前我没有遇到过，不知是否与pymongo的版本有关，有空我再看一下。 这样吧，临时改一行代码，把大集合的门槛调高，设置一个足够大的值，应该就不会报错了。 https://github.com/caosiyang/py-mongo-sync/blob/master/mongosync/common_syncer.py#L46

kamilkrzeminski commented 5 years ago

Same issue here, please let us know what would be the fix, not the workaround. Thanks

caosiyang commented 5 years ago

I always use pymongo 3.5.1 and Python 2.7, no such warnings found.