Closed tsuz closed 4 months ago
@tsuz I did some tests of the local branch attached in this issue. Now, we have the consumer lag per Global Consumer Thread
stream_librdkafka_consumer_metrics_consumer_lag{thread_id="test-app-146c8352-0c9c-46d6-a0af-3fa8323dca68-GlobalStreamThread",librdkafka_client_id="test-app-146c8352-0c9c-46d6-a0af-3fa8323dca68-GlobalStreamThread",application_id="test-app",broker_id="2",topic="Input2",partition_id="0"} -1
But the lag is still to -1, because the GlobalKTable never commit the offset.
We will probably use the consumer_lag_stored
stats in this case :
https://github.com/confluentinc/librdkafka/blob/master/STATISTICS.md
@tsuz Can you test the PR with your use case and let me know if it's good for you ?
@LGouellec
Fantastic. Thank you for the quick turnaround.
I see this topic has 1 partition but has two consumer lag where one is -1? Is this a limitation on the kafka dotnet side?
stream_librdkafka_consumer_metrics_consumer_lag{thread_id="test-app-rocksdbconcurrency2-004fb3c0-3c28-42ca-ab2f-79c67a009c42-GlobalStreamThread",librdkafka_client_id="test-app-rocksdbconcurrency2-004fb3c0-3c28-42ca-ab2f-79c67a009c42-GlobalStreamThread",application_id="test-app-rocksdbconcurrency2",broker_id="11",topic="mytopic",partition_id="0"} 249
stream_librdkafka_consumer_metrics_consumer_lag{thread_id="test-app-rocksdbconcurrency2-004fb3c0-3c28-42ca-ab2f-79c67a009c42-GlobalStreamThread",librdkafka_client_id="test-app-rocksdbconcurrency2-004fb3c0-3c28-42ca-ab2f-79c67a009c42-GlobalStreamThread",application_id="test-app-rocksdbconcurrency2",broker_id="-1",topic="mytopic",partition_id="-1"} -1
@tsuz
Not a limitation more a classic behavior.
Librdkafka expose a metric with a partition -1 for internal assigned partition : https://github.com/confluentinc/librdkafka/blob/master/STATISTICS.md#partitions
But GlobalKtable use assign(..), so the -1 partition is never used, that's why you always have the partition -1 metric lag at -1
@LGouellec Thanks, that makes sense. So far it looks good on my end of testing. 👍
Description
Java version of Kafka Streams emits a consumer lag for GlobalKTable. For example,
However, this is not emitted in dotnet.
How to reproduce
curl localhost:9098
Checklist
Please provide the following information: