search

Dabz / ccloudexporter

Prometheus exporter for Confluent Cloud API metric
https://docs.confluent.io/current/cloud/metrics-api.html
87 stars 53 forks source link

Gather consumer lag #12

Open Dabz opened 4 years ago

Dabz commented 4 years ago

Currently, the Metric API does not expose the consumer lag. But we could retrieve it in multiple ways, e.g. the exporter could rely on the Admin API to expose it.

alonisser commented 4 years ago

Note that there is lightbend/kafka-lag-exporter project that get the lag

danksim commented 3 years ago

@Dabz could you elaborate? is this planned work for the future? or is it already possible? would love to get Confluent Kafka broker lag metrics via this exporter.

Dabz commented 3 years ago

Hi @danksim , at the moment, you can not retrieve the lag via the ccloudexporter as this metric is not exposed through the Metrics API. The lag might be exposed through the Metrics API in the future, but I do not have any ETA to communicate.

In the meantime, if you need to retrieve the lag, the easiest is to rely on the lag exporter: lightbend/kafka-lag-exporter . I did a POC a while ago to gather the lag inside ccloudexporter, it introduces a fair amount of changes (e.g. need to be bundled with librdkafka), so I put it aside for the moment.

danksim commented 3 years ago

@Dabz I did end up using lightbend/kafka-lag-exporter to export our hosted (paid) Confluent Kafka broker's lag metrics yesterday. Was just wondering if there was a way to do it using Dabz/ccloudexporter without introducing another dependency. That is all. Thank you!