Provide Ability to DELETE consumer groups

[JohnOmernik]

I would like the ability to remove complete consumer groups via the kafka rest API. Right you can (based on documentation and testing) only remove instances. The Groups persists. This is difficult in that I have a "attach to the fire hose and see what's live" tool. Basically I just want to attach to a topic, and all messages from the time I attached forward, I don't care about previous messages, nor do I care where I stop. When I start the script, drop the needle and start pulling... the problem is, even if I unregister the previous consumer instance, the consumer group persists, and I get back logged, i.e. I start getting messaged where the other consumer stopped. While this is a good normal behavior, the API should have a way to destroy consumer groups (like kafka/bin/kafka-consumer-groups.sh --zookeeper node:2181 --group mygroup --delete but in API form.

[ewencp]

@WalterDalton Are you actually using the consumer group functionality here, i.e. balancing across multiple consumers? While I agree it would be nice to be able to remove consumer groups via the API (along with a variety of other admin functionality), it's not clear to me that's the best solution in this case. Your use case sounds like it might be a better fit for simple consumer style reads. I think the issue you'd face right now trying to implement that is that there won't be an easy mechanism to get the current offset to know where to start reading -- if we had a new consumer implementation we could use it in simple-consumer mode but still have it reset the offset to the latest offset when it is created.

[JohnOmernik]

In my case, I am not "using" it. My reader just wants all records, from all partitions in a single thread, and I set it to start at the end of the offsets (using Largest) and basically start from there. When I am finished, I don't care where I ended up. If there is a way to do that with the rest api that I am missing (in registering a consumer group and just listening) let me know. I may have some updates/suggestions for documentation in that case ;) That said, My work around is I have a small script running on the command line that I can run that uses curl to delete the consumer group instance, then uses the script in the kafka-9 bin directory to remove the consumer group, prior to each run of my "take a sampling" script. To me, the biggest issue is that I just want that consumer group gone every time my script is run, and ideally I'd have the capability in the rest-api itself, without having to use the shell.

[suhas22]

Hi, I agree on John's requirement, as I am also currently in need to this feature. My scenario is : I stream a lot of data from spark into Kafka, in spark we have the ability to point to a specific consumer group while streaming, if the consumer group is not available, it would create one, same as topics. However while testing stuff out before moving it onto prod, we had created a lot of consumer groups which is not required anymore, so need a way to delete them and clean up the cluster.

Any help on this is greatly appreciated, as I don't find a option to delete/remove the consumer group right now. I am on 0.8.2.1 kafka version.

[JohnOmernik]

@ewencp Hey Ewen, it's been a while on this issue, I am curious, now that I am using Confluent 3 and Kafka 10, are there any options for what I am trying to do? Ideally, providing a rest request for me to delete a consumer group would be ideal (once again, so I don't need to "shell" out side of my script) however, if another method of getting data via a simple consumer in the Rest API is available, that would be great as well. Thanks!

[ewencp]

@JohnOmernik Agree that this would be useful to expose, it's part of what I would consider admin APIs but has not yet been implemented. A lot of these things will be nicer to add/maintain with the KIP-4 admin APIs since currently implementing them requires using internal Kafka APIs that don't guarantee compatibility.