search

yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Apache License 2.0
1.88k stars 479 forks source link

Mongo-connector seems not to work with transactions - doesn't sync with elasticsearch #928

Open AmrAdelKhalil opened 3 years ago

AmrAdelKhalil commented 3 years ago

Hello, I have mongo with two replica-sets and I'm using mongo-connector to sync the data to elasticsearch, It was working fine till I started to use Transactions in my code. When I logged the mongo-connector service it was showing regular output like:

2021-07-13 19:35:30,625 [INFO] mongo_connector.oplog_manager:76 - OplogThread for replica set 'rs0' is up to date with the oplog.

and when I run the script which include the transaction, the logs showed this:

2021-07-13 19:35:41,166 [INFO] elasticsearch:81 - POST http://host:9200/_refresh [status:200 request:0.004s]

I tried to test this by inserting another document without the transaction and it showed:

2021-07-13 19:21:59,465 [INFO] elasticsearch:81 - POST http://marketplace2-elasticsearch-client:9200/_bulk [status:200 request:0.012s]

Now, what I can see is that the transaction is going somehow wrong with mongo-connector. Can anyone help me with that?

mongodb version: 4.0 mongo-connector version: latest & using: mongo-connector[elastic5]

AmrAdelKhalil commented 3 years ago

Check these out: 1- https://pypi.org/project/mongo-connector-transaction/ 2- https://pypi.org/project/elastic2-doc-manager-transaction/1.2.0/