Allow for more control over consumer group used

[donovanmuller]

Clusters that have authorization configured need more options around the consumer group Id (consumerIdPrefix), as ACL's are often based on the consumer group used.

For example, as it is now, the consumerIdPrefix is not sufficient because ACL's are linked to a specific/static consumer group Id and therefore a varying consumerIdPrefix + suffix combination does not work in this scenario.

Besides supporting a static consumer group Id, it would be great if a Cluster default consumer group Id could be provided when creating a new Cluster and potentially also override that consumer group per View.

[Crim]

Ah that's interesting. So in your ideal scenario you would be able to define a fixed consumer group id per view?

I think the tricky part is figuring how that works when you have multiple users in kafka-webview all consuming the same topic using the same consumer group id. I feel like the consumers stored state would over write each other no?

I'll do some investigating tomorrow.

[Crim]

@donovanmuller Which Authorizer are you using in your cluster? https://docs.confluent.io/current/kafka/authorization.html seems to indicate that the username for the ACL is generated from the client's certificate when using SSL. For SASL it's based on the supplied username.

I haven't found documentation that shows examples using the consumer group id as the username

[donovanmuller]

@Crim We are using SASL for authentication and unfortunately the Kafka cluster is operated by another team, so I'm making some assumptions here but I believe they have used this form of specifying the --group consumer group name included in the ACL together with the principal/username for specific topics: https://docs.confluent.io/current/kafka/authorization.html#adding-or-removing-a-principal-as-producer-or-consumer

bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 \
 --add --allow-principal User:Bob \
 --consumer --topic test-topic --group Group-1

Using the example above, I would like to configure Group-1 in Kafka Webview at a cluster level as the default consumer group to use for all Topics or if possible, specify a consumer group at a View level as well, which would override the cluster level consumer group for that view.

[Crim]

I need to do some testing to see if we were to setup static consumer group ids for a view (or cluster), will Kafka-WebView run into issues when multiple users attempt to consume using the same view by only receiving a portion of the data back from Kafka. I feel like how we're currently explicitly subscribing to specific partitions during the consume, this wouldn't be a problem.

Along the same lines, I need to verify how consumer state gets stored. If it's tied to the consumer group, or consumer id specifically.

[Crim]

So I've put some more thought into this and pretty sure what you've outlined makes perfect sense.

I like the idea of:

• Optionally set a default consumer group id at the cluster level.
• At the view level you can then optionally override it with something more specific.
• If no consumer group ids have been explicitly set at any level, fall back to using a generated consumer group id (similar to what is done today)

I believe doing the above, and still using consumer id's tied to the current user in Kafka-WebView allows things to work how one would expect in regards to consuming all data from a topic even if there are simultaneous subscribes/consumers in the same consumer group. I've started putting together a branch that incorporates these changes. 👍

[donovanmuller]

Awesome. Let me know if/when you'd like me to test on my side.