search

yougov / mongo-connector

MongoDB data stream pipeline tools by YouGov (adopted from MongoDB)
Apache License 2.0
1.88k stars 478 forks source link

Mongo connector stops working if primary mongo replica goes down #472

Open NehaPateliya25 opened 8 years ago

NehaPateliya25 commented 8 years ago

I have setup mongo replica structure with 1 primary and 2 secondary.

I am using mongo connector to sync data from mongo to solr using below cmd

mongo-connector -m 10.0.0.1:27017 -t http://10.0.0.5:8983/solr/user -d solr_doc_manager.py -n mydb.user_cl --auto-commit-interval=0

Its working fine when mongo on host 10.0.0.1 is on.

But when server with host 10.0.0.1 is down mongo connector stop working.

Can it automatically switch to current primary mongo replica or else is there any option to manage it?

llvtt commented 8 years ago

mongo-connector should be tolerant of replica set failover. If that's not happening, then something is wrong.

Can you please provide:

  1. Any output from stdout/stderr from mongo-connector
  2. The contents of mongo-connector.log
  3. The versions of mongo-connector and MongoDB

Thanks!

panqiushi commented 7 years ago

I also have this problem. My mongodb cluster is sharded cluster and each shard has an replica set. When a shard master and slave switch, the mongo-connector will down.

There not much log when this happened, here is the log: 2017-08-14 15:01:14,595 [ERROR] mongo_connector.connector:324 - MongoConnector: OplogThread <OplogThread(Thread-4, stopped 139664618944256)> unexpectedly stopped! Shutting down

I think when mongodb cluster working in my situation, mongo-connector is working on sharded mode. And when switch between master and slave, mongo-connector cannnot find the new master to connect.