Yannic92
commented
3 years ago Hi @ennru, We have a similar usecase where we want to have the benefit of a shared consumer to save resources, but also make use of what you already achieved in this project regarding subscribing, rebalancing, offset tracking..
To my understanding this would just require to pass an ActorRef to commitableSource and have a SingleSourceLogic which is similar to the existing but with the only difference to return this given ActorRef in createConsumerActor instead of creating a new one like it's done here: https://github.com/akka/alpakka-kafka/blob/master/core/src/main/scala/akka/kafka/internal/SingleSourceLogic.scala#L36
What do you think? Would this be an option, or are there any reasons to not allow this?
Similar to Consumer.committableExternalSource(), Consumer.committableSource() method should also share the KafkaConsumerActor. For reasons, if multiple streams are created in a single consumer application, each stream creates a new Kafka Consumer instance which creates new heartbeat threads. Sharing a consumer would limit such heartbeat threads and resources can be optimized.
Using committableExternalSource() instead of committableSource() brings more administrative activities to the application like rebalancing, offset tracking/assignment etc. even if the offsets are stored in kafka.