search

streamnative / kop

Kafka-on-Pulsar - A protocol handler that brings native Kafka protocol to Apache Pulsar
https://streamnative.io/docs/kop
Apache License 2.0
449 stars 133 forks source link

[FEATURE] Live migration tooling #1475

Open ZiyaoWei opened 1 year ago

ZiyaoWei commented 1 year ago

Is your feature request related to a problem? Please describe. A clear solution for migration from Kafka to KoP currently doesn't exist.

Describe the solution you'd like Add a live migration system so users could migrate from Kafka to KoP.

Describe alternatives you've considered We can provide offline migration tooling, which wouldn't be as user friendly.

Additional context None

eolivelli commented 1 year ago

I think that it depends on your application.

For instance if you are not keeping data for much time it is easy to attach your producers to the new cluster (Pulsar) and add new consumers on the new cluster. When the data from the old cluster is consumed you can drop it.

But if you have higher requirements then you have to design something specific

ZiyaoWei commented 1 year ago

Thanks @eolivelli! I have a design doc that we've gone through internally, but I'd love your (and anybody else in the community's) take on it as well! It's my oversight to not have shared it earlier here.

Link to the doc: https://docs.google.com/document/d/1MCpJEJEkIuMW-0IK7ubNOvE1x2hjATv52KmiAQdhgrE