caosiyang / py-mongo-sync

Oplog-based data sync tool that synchronizes data from a replica set to another deployment, e.g.: standalone, replica set, and sharded cluster.
http://caosiyang.github.io/py-mongo-sync/
MIT License
108 stars 46 forks source link

pymongo.errors.OperationFailure: not allowed to create index on config.transactions #24

Closed willli666 closed 5 years ago

willli666 commented 5 years ago

Hello!

I'm trying to use your tool to clone data from one replica set to another one. Both set are running mongo 4.0. I got two pymongo errors at the beginning and then the program stuck after some time.

Here is the full log

================================================
src hostportstr :  
src authdb      :  admin
src username    :  admin
src password    : 
src db version  :  4.0.3
dst hostportstr :  10.128.0.2:27017
dst authdb      :  admin
dst username    :  root
dst password    : 
dst db version  :  4.0.3
databases       :  
collections     :  
db mapping      :  
fileds          :  
start optime    :  None
optime logfile  :  
log filepath    :  apollo.log
pymongo version :  3.7.2
================================================
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run
  File "/home/willapollobox/py-mongo-sync/mongosync/mongo/syncer.py", line 82, in _sync_collection
    self._create_index(namespace_tuple)
  File "/home/willapollobox/py-mongo-sync/mongosync/mongo/syncer.py", line 76, in _create_index
    self._dst.create_index(dst_dbname, dst_collname, format(keys), **options)
  File "/home/willapollobox/py-mongo-sync/mongosync/mongo/handler.py", line 69, in create_index
    self._mc[dbname][collname].create_index(keys, **options)

  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/collection.py", line 1958, in create_index
    self.__create_index(keys, kwargs, session, **cmd_options)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/collection.py", line 1860, in __create_index
    session=session)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/collection.py", line 244, in _command
    retryable_write=retryable_write)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/pool.py", line 579, in command
    unacknowledged=unacknowledged)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/network.py", line 150, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
OperationFailure: not allowed to create index on config.transactions
2019-02-19T18:34:00Z <Greenlet at 0x7fbb3b996ec0: <bound method MongoSyncer._sync_collection of <mongosync.mongo.syncer.MongoSyn
cer object at 0x7fbb403e78d0>>((u'config', u'transactions'))> failed with OperationFailure
Traceback (most recent call last):
  File "sync.py", line 28, in <module>
    syncer.run()
  File "/home/willapollobox/py-mongo-sync/mongosync/common_syncer.py", line 68, in run
    self._sync()
  File "/home/willapollobox/py-mongo-sync/mongosync/common_syncer.py", line 92, in _sync
    self._initial_sync()
  File "/home/willapollobox/py-mongo-sync/mongosync/common_syncer.py", line 190, in _initial_sync
    for res in pool.imap(self._sync_collection, small_colls):
  File "src/gevent/_imap.py", line 107, in gevent.__imap.IMapUnordered.__next__
  File "src/gevent/_imap.py", line 39, in gevent.__imap._raise_exc
  File "src/gevent/greenlet.py", line 317, in gevent._greenlet.Greenlet._raise_exception

  File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run

  File "/home/willapollobox/py-mongo-sync/mongosync/mongo/syncer.py", line 82, in _sync_collection
    self._create_index(namespace_tuple)
  File "/home/willapollobox/py-mongo-sync/mongosync/mongo/syncer.py", line 76, in _create_index
    self._dst.create_index(dst_dbname, dst_collname, format(keys), **options)
  File "/home/willapollobox/py-mongo-sync/mongosync/mongo/handler.py", line 69, in create_index
    self._mc[dbname][collname].create_index(keys, **options)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/collection.py", line 1958, in create_index
    self.__create_index(keys, kwargs, session, **cmd_options)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/collection.py", line 1860, in __create_index
    session=session)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/collection.py", line 244, in _command
    retryable_write=retryable_write)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/pool.py", line 579, in command
    unacknowledged=unacknowledged)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/network.py", line 150, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/home/willapollobox/.local/lib/python2.7/site-packages/pymongo/helpers.py", line 155, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: not allowed to create index on config.transactions
[ OK ]  [1/89]  parse_apollobox.JewelryBoxReview        7/7     0.1s
...
[ OK ]  [2/89]  parse_apollobox.fs.files        1/1     0.0s
[ OK ]  [87/89] parse_apollobox._User   360894/360894   316.9s
then the program just hangs here and I killed it after half an hour

程序卡在87/89这里几乎一小时不动,我就把程序关了。之后我查看程序的log,里面没有错误的log,最后一行就是

2019-02-19 18:39:18,127 INFO    parse_apollobox._User   360000/360894   [99.75%]
2019-02-19 18:39:18,331 INFO [ OK ]     parse_apollobox._User   360894/360894   [100.00%]

我有几个问题请教,新手,请见谅

  1. 最开始的not allowed to create index on config.transactions 这个error需要在乎吗?
  2. 我关程序的时候是不是程序还在正常的复制? log中有这么一句 large collections: [u'parse_apollobox.UserProduct'], 而这个collection并不在已经完成的那87/89之中。如果还在正常复制,为什么一点log都没有,我看到其他collection复制中都有百分比的更新。
  3. README中没有详细介绍如何使用增量同步,请问这个程序是会在复制collection完成后自动持续进行oplog增量同步吗?

非常感谢!

willli666 commented 5 years ago
  1. toml中指定database 就没有这个error了
  2. 重新跑了一次,这一次很快就全部完成了
  3. 是的,之后会自动持续增量同步

都是我自己的问题,关闭了,多谢作者