No lag reported in metrics

[TonyMcTony]

We are using Burrow, v1.3.4 against Amazon MSK (Apache Kafka), v2.5.1. Trying to setup burrow/prometheus/grafana monitoring, we faced following issue:

On stage env

/metrics endpoint of burrow shows everything that is expected regarding our custom topics, including:

• burrow_kafka_consumer_current_offset
• burrow_kafka_consumer_partition_lag
• burrow_kafka_topic_partition_offset

REST endpoints show the same, everything looks fine.

On live env

/metrics endpoint shows values of:

• burrow_kafka_consumer_current_offset and burrow_kafka_consumer_partition_lag for ONLY some internal Amazon MSK topic (__amazon_msk_canary), BUT NOT FOR ANY CUSTOM TOPIC OF OURS
• burrow_kafka_topic_partition_offset for our custom topics as well as for ___amazon_mskcanary

What's even more strange, REST endpoints show all as expected for all topics.

After spending a whole day on research (kafka upgrade from v2.2 to v2.5.1, restarting the Burrow, artificially creating the lag by turning off the consumer group) we've totally stuck.

Any help would be appreciated.

[TonyMcTony]

After enabling more producers, the lag metric appeared for every consumer.

[TarasGTI]

Hey Tony, Could you please share burrow config? Thanks