search

deephaven / deephaven-core

Deephaven Community Core
Other
249 stars 79 forks source link

FEATURE REQUEST: Specify the partition for a Kafka record on publish #4206

Open jjbrosnan opened 1 year ago

jjbrosnan commented 1 year ago

As a user, I'd like to specify the partition for a Kafka record on publish. There's a constructor for ProducerRecord (see here) that could be used.

supernomad commented 1 year ago

Hey @jjbrosnan a concrete usecase of this, is that we have a large volume of events, and we want to have DH instances that can consume entire subsets of our data. We do this currently by partitioning data logically for our data across the various kafka topic partitions. This allows a single DH instance to handle the entire set of data and correctly aggregate the data.

We want to take this and have it applied to output topics from DH so that downstream consumers can also rely on partitioning properly. Otherwise we can't split up our data sets across instances to spread load.