Describe the bug
Quoted from a Gitter conversation:
V.Zaytsev @unit7
Hi everyone! Can anyone suggest me why for Flink-streamlets kafka consumer-group-id is specified as the name of the input topic? In fact, this means that only one Flink-streamlet can read from a particular topic, that is all Flink-streamlets will be combined in one consumer group. So what was the design idea? For akka-streamlets, consumer-group-id is formed taking into account the streamlet and inlet from which the reading occurs.
Thanks in advance.
The related sources:
Debasish Ghosh @debasishg 11
@unit7-0 We can surely align the consumer group id to make it similar to akka. But Flink Kafka connectors don’t use the consumer group management functionality because they use lower-level APIs on each parallel instance for more control on individual partition consumption. So, essentially, the “group.id” setting in the Flink Kafka connector is only used for committing offsets back to ZK / Kafka brokers.
V.Zaytsev @unit7
@debasishg, Did I understand correctly that since each flink-streaming-job tracks the offsets individually and saves it by the internal checkpoint mechanism, is there no need to specify a different groupd.id for each streamlet? Accordingly in the case when there are two streamlets reading the same topic but has different business logic will work correctly although the consumers will be in the same consumer-group?
I think they are right and we should adjust our consumer group id to be the same as the one we use for the Akka streamlet runner.
debasishg
commented
4 years ago
Describe the bug Quoted from a Gitter conversation:
I think they are right and we should adjust our consumer group id to be the same as the one we use for the Akka streamlet runner.