Add details for Topics and Consumer Groups into the Details Drawer

[christiemolloy]

This screenshot is from the user-testing URL. I believe we can replace these mock totals with actual data from the endpoint.

This is a function Pete wrote to Fetch Instant Metrics. We should also link to the Topics page and the Consumer Groups page in the Data Plane.

  const fetchInstantMetrics = async () => {
    const accessToken = await authContext?.getToken();
    if (accessToken !== undefined && accessToken !== '') {
      try {
        const apisService = new DefaultApi({
          accessToken,
          basePath
        });
        if (!instanceDetail || !instanceDetail.id) {
          return;
        }
        await apisService.getMetricsByKafkaId(instanceDetail.id, 1, 5, ['kafka_controller_kafkacontroller_offline_partitions_count', 'kafka_controller_kafkacontroller_global_partition_count']).then((res) => {
          const data = res.data;
          let globalPartitionCounter = 0, offlinePartitionCounter = 0;
          data.items?.forEach(item => {
            const labels = item.metric;
            if (labels === undefined) {
              throw new Error('item.metric cannot be undefined');
            }
            if (item.values === undefined) {
              throw new Error('item.values cannot be undefined');
            }
            if (labels['__name__'] === 'kafka_controller_kafkacontroller_offline_partitions_count') {
              offlinePartitionCounter += item.values[item.values.length - 1].Value;
            }
            if (labels['__name__'] === 'kafka_controller_kafkacontroller_global_partition_count') {
              globalPartitionCounter += item.values[item.values.length - 1].Value;
            }
          });
          setGlobalPartitionCount(globalPartitionCounter);
          setOfflinePartitionCount(offlinePartitionCounter);
        });
      } catch (error) {
        let reason: string | undefined;
        if (isServiceApiError(error)) {
          reason = error.response?.data.reason;
        }
        /**
         * Todo: show user friendly message according to server code
         * and translation for specific language
         *
         */
        addAlert(t('something_went_wrong'), AlertVariant.danger, reason);
      }
    }
  };

[christiemolloy]

@jgiardino anything else you want to add ?

[jgiardino]

This is great! Probably just worth pointing out that these would display above the other metadata in the Details tab.

[ajaypratap003]

@pmuir Can you please confirm the above API function fetchInstantMetrics is correct to get the total Topic and Consumer. Which one is the topic and consumer value in function.

[pmuir]

I'm not sure this is the right way to do it anymore. Since I wrote that We added the instant and range queries. You should just be able to use the instant query directly. Also those metrics aren't the ones for topic count of consumer group count, they are for partitions.

[ajaypratap003]

I'm not sure this is the right way to do it anymore. Since I wrote that We added the instant and range queries. You should just be able to use the instant query directly. Also those metrics aren't the ones for topic count of consumer group count, they are for partitions.

@pmuir Can you please give me some idea how can i get the total topic and consumers from API getMetricsByInstantQuery. What I need to pass in filter and what I need to do for getting total count.

[pmuir]

I'm not sure this is exposed at the moment. @redmikhail @redhatHameed do we have a list of metrics available published somewhere?

[redmikhail]

@pmuir I think that metrics marked for "UI" in this document https://docs.google.com/spreadsheets/d/17hLZkcyD1_ujzB0uZQnmdQdDSLDgkmieXgN6Sx9i5Aw/edit#gid=56854475 and listed here should be still more or less accurate https://docs.google.com/spreadsheets/d/1IumrFWNDSxxq58LEeQ0Hh-QPjpclNHrOMKYd3Exf-qA/edit#gid=1739826719

[pmuir]

And I think @david-martin has a guide for how to request new metrics?

[redmikhail]

And I believe this would be references in the code of fleet manager https://github.com/redhatHameed/kas-fleet-manager/blob/cd0c5004100b67c8237a63e1cfed71402461f94b/pkg/client/observatorium/api.go#L45

[david-martin]

And I think @david-martin has a guide for how to request new metrics?

https://github.com/bf2fc6cc711aee1a0c2a/observability-resources-mk/tree/main/resources/prometheus#prometheus-remote-write-to-observatorium

[redhatHameed]

And I believe this would be references in the code of fleet manager https://github.com/redhatHameed/kas-fleet-manager/blob/cd0c5004100b67c8237a63e1cfed71402461f94b/pkg/client/observatorium/api.go#L45

thanks, @redmikhail that's correct, There are two steps to expose any new metrics on UI. 1- add it remote write configuration https://github.com/bf2fc6cc711aee1a0c2a/observability-resources-mk/blob/main/resources/prometheus/remote-write.yaml

2- update fleet manager code to that https://github.com/redhatHameed/kas-fleet-manager/blob/cd0c5004100b67c8237a63e1cfed71402461f94b/pkg/client/observatorium/api.go#L45

[redmikhail]

Just as an additional step to the guide mentioned by @david-martin , please also update the document https://docs.google.com/spreadsheets/d/17hLZkcyD1_ujzB0uZQnmdQdDSLDgkmieXgN6Sx9i5Aw/edit#gid=56854475 until we have better all encompassing document

[christiemolloy]

Spoke to David Martin about getting values for Topics / Consumer Groups and he gave me these formulas to use to calculate Topics / Consumer Groups:

Number of Topics:

count(kafka_topic_partitions{namespace="cmolloy-kafka-devexp-1sljc628mgipfw9vmttpar01hdw"})

{}  2

Number of partitions in each topic

sum(kafka_topic_partitions{namespace="cmolloy-kafka-devexp-1sljc628mgipfw9vmttpar01hdw"}) by(topic)

{topic="__consumer_offsets"}    50
{topic="__strimzi_canary"}  3

Number of consumer groups

count(count(kafka_consumergroup_current_offset{namespace="cmolloy-kafka-devexp-1sljc628mgipfw9vmttpar01hdw"}) by(consumergroup))

{}  1

I'll be giving these a try shortly.