search

yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Apache License 2.0
1.88k stars 479 forks source link

Not able to change the cluster #924

Open singhmahendrapal opened 3 years ago

singhmahendrapal commented 3 years ago

Currently we are using a Mongodb 3.6 version cluster and to Elasticsearch syncing we are using mongo-connector. I want to upgrade it to 4.0. So we are following the below steps - 1) Taken a snapshot of 3.6 cluster 2) Created a new cluster with 4.0 version and restore the 3.6 sanpsot in it successfully. 3) Updated the new 4.0 replicaset name into existing oplog.ts file (with existing value)

Now my syncing has been stopped. Can someone provide any help on it so that i can start syncing again?

Note: Since we have the same data into new 4.0 cluster so i don't want the syncing since starting. Syning should work when i insert a new record into new cluser 4.0

Thanks in advance. As it's very urgent for me.

singhmahendrapal commented 3 years ago

Hi @jaraco, @AayushU and @ShaneHarvey, Since you guys are the most active contributor of mongo-connector. Actually, I need to change the MongoDB cluster in our application. So I created a new cluster with the updated version and dump all the data into it from old cluster. But after doing this my mongo-connector syncing stopped and if i remove my oplog.ts file it starts the syncing beginging.

Current in our DB we have around 2 CRs data collection so sync them into Elastic Search via mongo-connector since starting it will take time. So do you guys any suggestion and idea so that we can point out the new cluter and start the syncing where our old cluster left? I have no issue to pay you guys for this efforts. Pleaes let us know so that we can proceed accordinngly.

It's very urgent so please have a look and do the needful.

ShaneHarvey commented 3 years ago

Hi @singhmahendrapal, apologies for the situation. It does sound frustrating. Unfortunately, I do not work on mongo-connector anymore. Taking a look through the open issues shows that mongo-connector only has limited support for MongoDB 3.6 (https://github.com/yougov/mongo-connector/issues/767) and does not support MongoDB 4.0 (https://github.com/yougov/mongo-connector/issues/865). It seems like you would need to wait for those projects to be completed before using mongo-connector with MongoDB 4.0.

Note that if you're using MongoDB Atlas than you might be interested in using Atlas Search which provides full text search natively in the MongoDB cluster itself. With Atlas Search there's no need to manually sync data to other systems.

jaraco commented 3 years ago

Hello Singh. I unfortunately have little to add. I also have moved to a new role where I spend little time with MongoDB and mongo-connector. I wish I could have help, but I have neither context nor bandwidth to offer. I wish you luck. Feel free to ping me on PRs that would aid in supporting your needs. I'm happy to merge and release contributions.