search

uber / uReplicator

Improvement of Apache Kafka Mirrormaker
Apache License 2.0
917 stars 199 forks source link

uReplicator vs. Kafka MirrorMaker w/New Consumer (not high-level consumer) #127

Closed ghost closed 6 years ago

ghost commented 6 years ago

Hi.

The documentation on the web for uReplicator seems to compare uReplicator to Kafka MirrorMaker before the arrival of "new consumer" support for it.

https://docs.confluent.io/current/clients/consumer.html

Does this upstream change obsolete uReplicator in any way? There are features of uReplicator that I enjoy like the REST API -- but are the high-level consumer concerns about MirrorMaker on the wiki and blog pages still relevant with the new consumer? Thanks!

xhl1988 commented 6 years ago

No. MirrorMaker worker doesn't use those features in high-level consumer like auto partition assignment/balance. It depends on Controller to assign partitions to worker, in which the consumer simply consume the partition it got assigned. So it doesn't make much different to use SimpleConsumer or NewConsumer in Worker. Besides the REST API, the biggest benefit of uReplicator Controller is that it can assign partitions in a smart way: based on number of partitions and traffic, and can move lagging partitions to a dedicated worker, which can significantly improve the replication performance.

ghost commented 6 years ago

Thanks!