Reporting old metrics if topic is re-created within poll interval

[seruman]

Describe the bug If a topic is deleted then re-created within poll interval, exporter reports kafka_partition_latest_offset as the value before deletion. This also causes lag metrics to be wrong since they're derived from it.

To Reproduce Logs: https://gist.github.com/seruman/6f9a6718e29760af41683a4b45bd5ef2 Config:

kafka-lag-exporter {
  port = 8000
  client-group-id = "kafkalagexporter"
  kafka-client-timeout = 30 seconds
  clusters = [
    {
     name = "local"
     bootstrap-brokers = "kafka:9092"
    }
  ]
}

• Create topic my-topic
• Start consumer my-group
• Produce 100 messages
• Wait until kafka-lag-exporter polls and Prometheus scrapes
• Within poll interval, before kafka-lag-exporter polls;
  • Stop consumer
  • Delete topic
  • Re-create topic
  • Start consumer
  • Produce 20 messages

Environment

• Version: 0.6.2
• Version of Apache Kafka cluster: 2.3.1, 2.3.1 AWS MSK
• Run with Helm or Standalone: Standalone

Additional context

[seglo]

Interesting. Will the latest offset ever reset on its own (perhaps after 2 intervals?) or does it remain in this state indefinitely?

[seruman]

It stays in that state. Previous logs I've shared only had a single poll before shutting the exporter down. It still reports old state even after multiple intervals;

• Logs
• Metrics screenshot

Lately we experienced this again, it was still reporting the old latest offset even after an hour the topic recreated;

• Metrics screenshot

[seruman]

Hi @seglo, Since there were new releases, I tried to re-produce the issue with the 0.7.1 and I can confirm it is still re-producable even after multiple poll intervals.