search

yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Apache License 2.0
1.88k stars 479 forks source link

Mongo-connector is stops unexpectedly. #856

Open krunalpatel155 opened 5 years ago

krunalpatel155 commented 5 years ago

Guys,

I'm new to Mongo-Connector and Elastic Search. I want to sync all the documents from MongoDB to elasticsearch and therefore I've used Mongo-Connector and doc2_elastic_search. After the first successful attempt to start mongo-connector for the second time it's getting failed and below is the error log. Can you please help me to find out the root and resolution.

2019-02-11 10:40:14,624 [ALWAYS] mongo_connector.connector:50 - Starting mongo-connector version: 3.1.1 2019-02-11 10:40:14,625 [ALWAYS] mongo_connector.connector:50 - Python version: 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609] 2019-02-11 10:40:14,625 [ALWAYS] mongo_connector.connector:50 - Platform: Linux-4.4.0-1075-aws-x86_64-with-Ubuntu-16.04-xenial 2019-02-11 10:40:14,625 [ALWAYS] mongo_connector.connector:50 - pymongo version: 3.7.2 2019-02-11 10:40:14,629 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 4.0.6 2019-02-11 10:40:14,629 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.elastic2_doc_manager version: 1.0.0 2019-02-11 10:40:14,635 [INFO] mongo_connector.oplog_manager:137 - OplogThread: Initializing oplog thread 2019-02-11 10:40:14,636 [INFO] mongo_connector.connector:402 - MongoConnector: Starting connection thread MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset=u'rs0') 2019-02-11 10:40:14,637 [DEBUG] mongo_connector.oplog_manager:214 - OplogThread: Run thread started 2019-02-11 10:40:14,637 [DEBUG] mongo_connector.oplog_manager:216 - OplogThread: Getting cursor 2019-02-11 10:40:14,637 [DEBUG] mongo_connector.oplog_manager:881 - OplogThread: reading last checkpoint as Timestamp(1549877201, 2) 2019-02-11 10:40:14,647 [DEBUG] mongo_connector.oplog_manager:752 - OplogThread: Oldest oplog entry has timestamp Timestamp(1549274474, 2). 2019-02-11 10:40:14,647 [DEBUG] mongo_connector.oplog_manager:239 - OplogThread: about to process new oplog entries 2019-02-11 10:40:14,647 [DEBUG] mongo_connector.oplog_manager:242 - OplogThread: Cursor is still alive and thread is still running. 2019-02-11 10:40:14,647 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 0 2019-02-11 10:40:14,647 [DEBUG] mongo_connector.oplog_manager:271 - OplogThread: Operation for this entry is c 2019-02-11 10:40:14,647 [DEBUG] urllib3.util.retry:210 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-02-11 10:40:14,648 [DEBUG] urllib3.connectionpool:205 - Starting new HTTP connection (1): localhost:9200 2019-02-11 10:40:14,662 [DEBUG] urllib3.connectionpool:393 - http://localhost:9200 "POST /_refresh HTTP/1.1" 200 49 2019-02-11 10:40:14,663 [INFO] elasticsearch:83 - POST http://localhost:9200/_refresh [status:200 request:0.015s] 2019-02-11 10:40:14,663 [DEBUG] elasticsearch:85 - > None 2019-02-11 10:40:14,663 [DEBUG] elasticsearch:86 - < {"_shards":{"total":0,"successful":0,"failed":0}} 2019-02-11 10:40:14,663 [DEBUG] mongo_connector.oplog_manager:322 - OplogThread: Documents removed: 0, inserted: 0, updated: 0 so far 2019-02-11 10:40:14,663 [DEBUG] mongo_connector.oplog_manager:325 - OplogThread: Doc is processed. 2019-02-11 10:40:14,663 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 1 2019-02-11 10:40:14,663 [DEBUG] mongo_connector.oplog_manager:271 - OplogThread: Operation for this entry is c 2019-02-11 10:40:14,663 [DEBUG] urllib3.util.retry:210 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-02-11 10:40:14,666 [DEBUG] urllib3.connectionpool:393 - http://localhost:9200 "POST /_refresh HTTP/1.1" 200 49 2019-02-11 10:40:14,666 [INFO] elasticsearch:83 - POST http://localhost:9200/_refresh [status:200 request:0.003s] 2019-02-11 10:40:14,666 [DEBUG] elasticsearch:85 - > None 2019-02-11 10:40:14,666 [DEBUG] elasticsearch:86 - < {"_shards":{"total":0,"successful":0,"failed":0}} 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:322 - OplogThread: Documents removed: 0, inserted: 0, updated: 0 so far 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:325 - OplogThread: Doc is processed. 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 2 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:192 - OplogThread: Skipping oplog entry: 'FullText.FTC' is not in the namespace configuration. 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 3 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 4 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 5 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 6 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 7 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 6 2019-02-11 10:40:14,666 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 7 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 8 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 9 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 10 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 11 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 12 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 13 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 14 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 15 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 16 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:252 - OplogThread: Iterating through cursor, document number in this cursor is 17 2019-02-11 10:40:14,667 [DEBUG] mongo_connector.oplog_manager:271 - OplogThread: Operation for this entry is c 2019-02-11 10:40:14,667 [DEBUG] urllib3.util.retry:210 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-02-11 10:40:14,669 [DEBUG] urllib3.connectionpool:393 - http://localhost:9200 "POST /_refresh HTTP/1.1" 200 49 2019-02-11 10:40:14,669 [INFO] elasticsearch:83 - POST http://localhost:9200/_refresh [status:200 request:0.002s] 2019-02-11 10:40:14,669 [DEBUG] elasticsearch:85 - > None 2019-02-11 10:40:14,669 [DEBUG] elasticsearch:86 - < {"_shards":{"total":0,"successful":0,"failed":0}} 2019-02-11 10:40:14,669 [DEBUG] urllib3.util.retry:210 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-02-11 10:40:14,676 [DEBUG] urllib3.connectionpool:393 - http://localhost:9200 "PUT /fulltext/_mapping/Test HTTP/1.1" 404 355 2019-02-11 10:40:14,676 [WARNING] elasticsearch:97 - PUT http://localhost:9200/fulltext/_mapping/Test [status:404 request:0.007s] 2019-02-11 10:40:14,676 [DEBUG] elasticsearch:105 - > {"_source": {"enabled": true}} 2019-02-11 10:40:14,676 [DEBUG] elasticsearch:110 - < {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"fulltext","index_uuid":"na","index":"fulltext"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"fulltext","index_uuid":"na","index":"fulltext"},"status":404} 2019-02-11 10:40:14,676 [ERROR] mongo_connector.util:109 - Fatal Exception Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 107, in wrapped func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/mongo_connector/oplog_manager.py", line 306, in run docman.handle_command(doc, entry["ns"], timestamp) File "/usr/local/lib/python2.7/dist-packages/mongo_connector/util.py", line 44, in wrapped raise new_type(str(exc_value)).with_traceback(exc_tb) AttributeError: 'OperationFailed' object has no attribute 'with_traceback' 2019-02-11 10:40:15,638 [ERROR] mongo_connector.connector:411 - MongoConnector: OplogThread <OplogThread(Thread-3, stopped 140456587400960)> unexpectedly stopped! Shutting down 2019-02-11 10:40:15,638 [INFO] mongo_connector.connector:478 - MongoConnector: Stopping all OplogThreads 2019-02-11 10:40:15,638 [DEBUG] mongo_connector.oplog_manager:371 - OplogThread: exiting due to join call. 2019-02-11 10:40:16,626 [DEBUG] urllib3.util.retry:210 - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None) 2019-02-11 10:40:16,628 [DEBUG] urllib3.connectionpool:393 - http://localhost:9200 "POST /_refresh HTTP/1.1" 200 49 2019-02-11 10:40:16,628 [INFO] elasticsearch:83 - POST http://localhost:9200/_refresh [status:200 request:0.002s] 2019-02-11 10:40:16,629 [DEBUG] elasticsearch:85 - > None

leleTino commented 5 years ago

Hi guy, news in order to solve that issue? I'm trying to connect my mongodb database to ELK stack but have this problem:

" [ERROR] mongo_connector.connector:411 "

horysk commented 4 years ago

me too

johnda98 commented 4 years ago

what solved this ? having a VERY similar deal here

leleTino commented 4 years ago

I don't know, I'm not using it in this periodIl 22 ott 2020 21:37, John D Allen notifications@github.com ha scritto: what solved this ? having a VERY similar deal here

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

johnda98 commented 4 years ago

thanks for kind note.. I think there are 3 aspects.. the id for mongo local, the id for neo4j(app) and the id for the neo4jdatabase

I am not sure what mongo-connector is complaining about .. any thoughts ? much appreciate my friend.

odd thing is I set eno4j conf file to 'no dbms authentication to true' .. so that solved that .. I dont get asked for a neo4j app logon user/password .. so presume they are null OR 'neo4j' 'neofj' by default user/password ... and mongo localhost is wide open

most odd.. a puzzle

you suggest any tools I can use to dig deeper ? I used curl on the mongo url and the neo4j target .. it seems to 'see them'

John D Allen.

*T +1 417 849 0705 * CST | john.allen@leveridgesystems.com john.allen@leveridgesystems.com | https://www.leveridgesystems.com https://www.leveridgesystems.com | linkedin.com/in/johndallen1 http://linkedin.com/in/johndallen1 | skype john.allen_67 https://www.yourhashgraphtoken.com https://www.yourhashgraphtoken.com

Leveridge Systems INC. Reg Missouri Charter #01082460. The information contained in this email and any attached files is confidential and intended solely for the addressee(s). The email may be legally privileged or prohibited from disclosure and unauthorized use. If you are not the named addressee you may not use, copy, or disclose this information to any other person. If you received this message in error please notify the sender immediately and permanently delete it from your system.

On Fri, Oct 23, 2020 at 3:42 AM leleTino notifications@github.com wrote:

I don't know, I'm not using it in this periodIl 22 ott 2020 21:37, John D Allen notifications@github.com ha scritto: what solved this ? having a VERY similar deal here

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yougov/mongo-connector/issues/856#issuecomment-715194573, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGDUXKWQUJJBLGIA2B3XNLSME6VRANCNFSM4GWPWYIQ .