Export number of members and assigned partitions for each topic in a consumer group

[amuraru]

Currently, for each consumer group the total number of members (consumers) in the group is exported. For a consumer group subscribed to multiple topics, it would be also useful to have a metric to capture the number of consumer group members for each topic and also the number of partitions assigned for each topic.

Here is an example:

bash-5.1# /opt/kafka/bin/kafka-consumer-groups.sh  --bootstrap-server kafka:29092 --group my-group --describe  --members --verbose

GROUP             CONSUMER-ID                                 HOST            CLIENT-ID       #PARTITIONS     ASSIGNMENT
my-group my-consumer-2fcbe3db-48d5-459f-8317-2c71d077c0d8 /172.16.100.90  my-consumer          3               topic_1(1,2,3)
my-group my-consumer-2f128b7e-59c0-405a-ab23-ece7572c6b74 /172.16.96.154  my-consumer          0               -
my-group my-consumer-568d7ec8-db98-49f3-bd5a-2b93983e9f70 /172.16.109.218 my-consumer          3               topic_2(1,2,3)
my-group my-consumer-0ca436c7-ee5b-4626-9fdb-b1563423fd3a /172.16.150.15  my-consumer          1               topic_3(1),topic_2(4)
my-group my-consumer-5e99fcca-6108-4381-8f84-c8eca6438e2f /172.16.109.23  my-consumer          1               topic_4(1)
my-group my-consumer-afca0542-e05f-4389-a537-fe56ccbac2a8 /172.16.150.15  my-consumer          1               topic_1(4),topic_3(2)
my-group my-consumer-30539235-7a47-4b4b-b0a3-86ac7c6081d0 /172.16.96.154  my-consumer          0               -
my-group my-consumer-75b3aa7b-7ee8-4600-8640-41be291423cc /172.16.109.218 my-consumer          1               topic_2(5)

While the consumer group has 8 members in total, one can see that the group is not very well balanced from individual topic perspective, i.e. two consumers are not assigned any partition

It would be useful to have the following metrics:

1/ consumer_group_members -> 8 (already available)

2/ consumer_group_topic_members

• {topic_name="topic_1"} -> 2
• {topic_name="topic_2"} -> 3
• {topic_name="topic_3"} -> 2
• {topic_name="topic_4"} -> 1 When filtering a topic one can see the level of consume parallelism

3/ consumer_group_topic_assigned_partitions

• {topic_name="topic_1"} -> 4
• {topic_name="topic_2"} -> 5
• {topic_name="topic_3"} -> 2
• {topic_name="topic_4"} -> 1 This can be useful to check if all partitions of a topic are assigned in a consumer group