kafka consumer group operator

[carlobongiovanni]

Is your feature request related to a problem? Please describe. In a controlled environment where all resources are managed, having a way to manage the lifecycle of a consumer group could be an useful feature.

Describe the solution you'd like when enabled, the entity operator would deploy a new consumergoup-operator that would manage creation and deletion of consumergroups

Describe alternatives you've considered

Additional context

[scholzj]

Can you describe more details about how would you expect it to work? I do not think there is any real process for creation of consumer groups. It is created when the first client connects with it. There is also AFIAK no way how to not allow the client to create new consumer groups. So the KafkaConsumerGroup custom resources cannot be really used as the single source of truth (and I think we learned our lesson on topics that the bi-directional sync is very hard and should not use it for yet another operator). Also other operations like resetting consumer offsets or deletion would be complicated since they IIRC require the consumer group to be not used ... so that would not work well either. So TBH I'm not sure I'm convinced about how this would work, how it would be implemented and what would be the value.

[carlobongiovanni]

In a cluster with ACLs enabled only allowed users would be able to create them, so there should be no interference by new kafka consumers. I would expect the possibility to create the CG (consumer group), and the possibility to also set its offset (to latest, earliest or to a given value). There would be also a possibility to ignore specific consumer groups, filtering them by regex.

I know this can be a strange request, but normally many developers that use Kafka don't know almost nothing about CGs. Therefore a process to create or delete a CG would benefit the life of an SRE. Also it could become helpful to improve failover processes.

[scholzj]

In a cluster with ACLs enabled only allowed users would be able to create them, so there should be no interference by new kafka consumers.

I think there is just one ACL for both using and creating a consumer group. So you cannot protect the creation using ACLs.

I would expect the possibility to create the CG (consumer group), and the possibility to also set its offset (to latest, earliest or to a given value).

Can you share more details how would the operator create them? Are there really some Admin API calls for it in Kafka?

I know this can be a strange request, but normally many developers that use Kafka don't know almost nothing about CGs. Therefore a process to create or delete a CG would benefit the life of an SRE. Also it could become helpful to improve failover processes.

I can understand why it would be useful if it worked as topics or users. But I do not think the current lifecycle of Consumer Groups in Kafka is the same. So I do not think you can really operate it.

[carlobongiovanni]

I see, thanks anyway for looking into it ;)