search

uber / uReplicator

Improvement of Apache Kafka Mirrormaker
Apache License 2.0
917 stars 199 forks source link

Reset worker offset #139

Closed ntviet18 closed 6 years ago

ntviet18 commented 6 years ago

Hi @xhl1988, @fx19880617

I want to reset worker offsets for some cases like 1) target topic was deleted, I want to restart from the beginning 2) I want to skip some messages How can do that?

xhl1988 commented 6 years ago

@ntviet18

  1. The easiest way to reset offset is to delete the committed offset node in source cluster zk, like /<your cluster>/consumers/<your consumer group>/offsets/<your topic>, and set https://github.com/uber/uReplicator/blob/master/config/consumer.properties#L38 to smallest, then restart the workers and it will consume from the beginning
  2. You can refer to this pull request: https://github.com/uber/uReplicator/pull/132/files, you need to a: provide filterProps and set enabled to true; b. implement your own needToSend logic to do the filtering.
ntviet18 commented 6 years ago

@xhl1988 , thank you very much your solutions are very elegant and helpful