[ERROR] mongo_connector.oplog_manager:324

[ghost]

hi guys , I got a serious problem today.

the mongo-connector.log's content is here :

OperationFailed: TransportError(404, u'{"_index":"nicovideo","_type":"posts","_id":"27131240","found":false}')
2016-09-10 00:01:36,787 [ERROR] mongo_connector.oplog_manager:324 - Unable to process oplog document {u'h': -4790094769725122799L, u'ts': Timestamp(1473480246, 2), u'o': {u'$set': {u'view_count': 22, u'__v': 5, u'urls': [{u'cookie': u'sm29618346:1473480091:1473480091:0747757f18bebe4a:1', u'vip': False, u'type': u'mp4', u'get_at': datetime.datetime(2016, 9, 10, 4, 4, 6, 945000), u'value': u'http://smile-fnl11.nicovideo.jp/smile?m=29618346.70461'}]}}, u't': 2L, u'v': 2, u'ns': u'nicovideo.posts', u'o2': {u'_id': 29618346}, u'op': u'u'}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 310, in run
    ns, timestamp)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 43, in wrapped
    reraise(new_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 32, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/mongo_connector/doc_managers/elastic2_doc_manager.py", line 161, in update
    id=u(document_id))
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py", line 69, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py", line 330, in get
    doc_type, id), params=params)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py", line 307, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py", line 93, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py", line 105, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
OperationFailed: TransportError(404, u'{"_index":"nicovideo","_type":"posts","_id":"29618346","found":false}')

it seems like the mongo-connector cannot find the documents from the oplog

I need some help , thanks a lot!

[kubage88]

Maybe we have the same problem, Official developer has marked it as a bug. # Resumable initial collection dump

[ghost]

@kubage88 谢谢你的回复

错误大概的意思可能是在oplog中无法找到对应的数据吧

那请问最后你们怎么解决这个mongodb和es的数据同步问题了呢？

谢谢

[kubage88]

@audeSt 暂时还没有解决方案，官方给我的答复是将在接下来的新版本中修复。我现在遇到这种问题都是删除索引从头同步。

[ghost]

@kubage88 谢谢，打算自己实现了

[ShaneHarvey]

@audeSt Can you provide the steps to reproduce this problem?