search

DataReply / kafka-connect-mongodb

Apache License 2.0
129 stars 61 forks source link

Cluster aware #5

Open jlpedrosa opened 8 years ago

jlpedrosa commented 8 years ago

Hi I was wondering if the connector should not:

1) be aware that there are more than one node of the replica set (so host,port should be a list not an entry), so if the primary goes down we still can connect to the new primary (that was a secondary) and still read the oplog. and/or 2) it should be aware of the fact that it's a shared cluster and spawn one job per shard/replica set? In other ways, first determine that it's connected to a mongos, also process should be able to connect to multiple mongos (as the driver) to be fault tolerant.

Makes sense?

patelliandrea commented 8 years ago

Hi, as soon as I have time I'll update the connector to work with the new confluent platform and Kafka 0.10. Since I'm updating it I can change the connector so it can connect to multiple hosts instead of only one. Regarding the second point I have to look into it and maybe implement it, my biggest problem right now is that I don't have that much time for this project so I can't promise anything.