uber / uReplicator

Improvement of Apache Kafka Mirrormaker
Apache License 2.0
914 stars 198 forks source link

How to add more than one kafka source or destionation? #229

Open cemakpolat opened 5 years ago

cemakpolat commented 5 years ago

Hi,

After executing the uReplicator source code, I wanted to add more than two kafka sources for the same topic. I couldn't achieve this purpose. I did some change in the source code where the cluster of kafka source and kafka destination are added. Basically I added another kafka-source and prepare the related config file like server3.properties. kafka-source-3 is started as a worker, however, the messages that I sent from kafka-source-3 to the kafka-destination with the same-topic didn't work.

If I summarize the question: Does uReplicator support the data and topic synchronization between the multiple kafka sources to multiple kafka-destination? If yes, what should be the change that I should do? Thanks in advance!

Cheers Cem

yangy0000 commented 5 years ago

to approach this goal, we would suggest you set up two ureplicator clusters.

but, yes, we support data replication between multiple sources to multiple destinations in the federated mode (We don't have any documentation on how to setup ureplicator on federated mode yet. there is a brief instruction on youtube )

our implementation is different from what your thought, basically, each worker still replicate from one source cluster to on destination cluster. but there is another layer which allows you to manage multiple sources to destinations even for the same topic.