search

confluentinc / kafka-tutorials

Tutorials and Recipes for Apache Kafka
https://developer.confluent.io/tutorials
Apache License 2.0
15 stars 90 forks source link

Changing the number of partitions and replicas of a Kafka topic #446

Closed ybyzek closed 4 years ago

ybyzek commented 4 years ago

Changing number of partitions using ksqlDB Show the recommended steps to change the number of partitions.

This is one possible example in case the producers can be stopped (open for discussion):

Stop producers Run the ksqlDB query: 

CREATE STREAM old WITH (KAFKA_TOPIC='old-topic',VALUE_FORMAT='JSON');
CREATE STREAM new WITH (KAFKA_TOPIC='new-topic',VALUE_FORMAT=’JSON',PARTITIONS=12)  AS SELECT * FROM old EMIT CHANGES;

Check all data has been migrated Increase number of consumers in the consumer group Change topic subscription of the consumer group to new-topic (auto.offset.reset=’latest’)

cc: @SrWaves

ybyzek commented 4 years ago

@rmoff @bbejeck if we created this KT, and included changing both partition count and replica count, would it be ok to retire https://www.confluent.io/stream-processing-cookbook/ksql-recipes/repartitioning-changing-replication ?

rmoff commented 4 years ago

Yes, sounds good.

minor nit: You don't need to respecify VALUE_FORMAT in the output if you're not changing it from the input AFAIK

ybyzek commented 4 years ago

KT added via https://github.com/confluentinc/kafka-tutorials/pull/470