search

nodefluent / kafka-streams

equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:
https://nodefluent.github.io/kafka-streams/
MIT License
830 stars 111 forks source link

Can I access the high level producer through this library? #96

Closed boriskogan81 closed 5 years ago

boriskogan81 commented 5 years ago

I have an issue-when I write to a topic, I would like to automatically distribute across its partitions, however many there may be. At the moment, I have to configure the number of partitions manually with stream.to([stream name], [number of partitions]). I would like to use the HighlevelProducer available in Kafka for this. Is this possible?

krystianity commented 5 years ago

Hi @boriskogan81 yes you can access the producer. However for what you want to achieve designing the final stream events in a key value structure would be the best way, as you can pass your desired partition and topic right with them.

https://github.com/nodefluent/kafka-streams/blob/master/lib/messageProduceHandle.js#L92

if you map your stream events to look like so: {key: "..", value: "..", partition: 0, topic: "my-topic" } before calling to, you can control where they are produced to on a per-event-basis.