Kafka exactly once consumer - vertical scaling

[ichrist-gr]

Hello folks,

we have a case where we need to scale (vertical) our exactly once consumer-producer chain. For example we have 10 partitions and we would like to read the kafka events from 1 pod in parallel.

But unfortunatelly after an investigation we found out the below piece of code that restricts us from scaling our kafka consumer.

List<KafkaConsumer<Object, Object>> consumers = clientService.getConsumers(channel);
if (consumers.isEmpty()) {
    throw KafkaExceptions.ex.unableToFindConsumerForChannel(channel);
} else if (consumers.size() > 1) {
    throw KafkaExceptions.ex.exactlyOnceProcessingNotSupported(channel);
}

Code exists here in line 85: KafkaTransactionsImpl

Are you planning to enable the vertical scaling feature for the kafka consumer for exactly-once feature?

[ozangunalp]

Sorry, I forgot to write back to this. The issue here is that a single producer can only enter a single transaction at a time. We'd need a multi-producer client that'll create a producer on-the-fly per-partition to enable this.

I won't have time to look at this now, if you'd like to give it a go I can help.