search

nodefluent / kafka-streams

equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:
https://nodefluent.github.io/kafka-streams/
MIT License
830 stars 111 forks source link

Creating new topic with multiple partitions #76

Closed mangvlad closed 6 years ago

mangvlad commented 6 years ago

I am trying to produce messages to a new topic using the "to" method:

stream.to({"topic":options.topic, "outputPartitionsCount":5});

This works fine, except the new topic always created with 1 partition.

Is there a way to create a topic with more than one partition?

Thank you!

krystianity commented 6 years ago

Hi @mangvlad the outputPartitionsCount or second argument of to(), is only used internally to spread the messages across partitions accordingly. However it cannot create a new topic with a certain amount of partitions, if the topic does not exist already it is created by the broker (depending on your broker configuration). It seems like you have enabled auto topic creation for the broker, but set the default partition count to 1.

My suggestion is to either pull that broker setting up or to create topic manually via Kafka cli tools beforehand. E.g. we have a repo to configure topics (there are also additional settings alongside partition count, like retention for example..) that is rolled out via Kafka cli tools automatically and creates topics, we have also disabled auto topic creation to ensure everyone sticks to this policy.

But in the end this is always up to what suits you best..

mangvlad commented 6 years ago

Got it! Thank you!