search

wintoncode / winton-kafka-streams

A Python implementation of Apache Kafka Streams
Apache License 2.0
313 stars 55 forks source link

Another partition problem #51

Open exception13 opened 6 years ago

exception13 commented 6 years ago

if use multiple process/threads with partitioning topic current realization StoreChangeLogger try write with specified partition without checking and repartition this topic.

Any ideas?

ah- commented 6 years ago

Hi, you mean it should ensure that the changelog topic has the same number of partitions as the input topic?

Agree that it should check, state management is still not finished. Would probably make most sense around the builder somewhere.

For creating more partitions we'd need https://github.com/edenhill/librdkafka/issues/986 first.