Stats callback return zeros in all rolling window stats counters

[geraldus]

Hi! I'm trying to monitor my Kafka producer by setting stats_cb with statisticsInterval ... <> setCallback $ statsCallback .... I can see an increase in such metrics as txmsgs, tx_bytes, but all the rolling window stats are filled with zeros, including int_latency and outbuf_latency, throttle, rtt, etc.

For example

 "outbuf_latency": {
        "avg": 0,
        "cnt": 0,
        "hdrsize": 11376,
        "max": 0,
        "min": 0,
        "outofrange": 0,
        "p50": 0,
        "p75": 0,
        "p90": 0,
        "p95": 0,
        "p99": 0,
        "p99_99": 0,
        "stddev": 0,
        "sum": 0
      },

I'm facing the same behaviour with a local Kafka server hosted in a Docker container and with my corporate stage environment. Am I missing something? Is it related to librdkafka version or some configuration parameters?

[AlexeyRaga]

I need to check how we actually parse it, but it may be librdkafka

[AlexeyRaga]

@geraldus looks like we don't do anything special with the stats callback and the data it receives

https://github.com/haskell-works/hw-kafka-client/blob/17451f5d020518190401cf34720ac780376a284e/src/Kafka/Internal/RdKafka.chs#L428

Maybe librdkafka needs to be instructed to compute these values... Not sure..

[geraldus]

Hi @AlexeyRaga! Sorry, I think I closed this thread and posted my answer the day I opened it, but that's not true. Yes, I've looked into the sources myself and found out that nothing special happens with the statistics coming from librdkafka. It turned out that the statistics will only be non-zero if there is active producing and consuming happening, while I tested this on a kind of dead topic with no consumers attached. I was able to achieve my goals and sample the needed metrics further with some other active topics. Thanks and sorry for the late reply.

[AlexeyRaga]

@geraldus This is something I didn't know, thanks for sharing this info and finding time to do it! Really appreciated!