search

provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management
Apache License 2.0
9.81k stars 1.19k forks source link

Topics: Message count in overview incorrect when using cleanup policy COMPACT #3535

Open joschi opened 1 year ago

joschi commented 1 year ago

Describe the bug (Actual behavior)

When using topics with clean-up policy COMPACT (log.cleanup.policy=compact) the Message Count metric on the topics overview page is incorrect.

https://github.com/provectus/kafka-ui/blob/v0.6.0/kafka-ui-react-app/src/components/Topics/Topic/Overview/Overview.tsx#L19-L25

The code to calculate the message count (SUM(partition.offsetMax - partition.offsetMin)) only works on non-compacted topics (with cleanup policy DELETE).

Expected behavior

The Message Count metric on the topics overview page should either be correct or not shown at all for topics with cleanup policy

Set up

Kafka UI 0.6.0 using the official Docker image deployed as a container in Kubernetes (EKS).

Steps to Reproduce

  1. Create topic with log.cleanup.policy=compact.
  2. Produce messages into the topic, some of which have the same partition key.
  3. Trigger or wait for log compaction to be performed on the Kafka broker(s) for the topic.
  4. Compare the actual number of messages (for example with the topics statistics view in Kafka UI) with the Message Count shown on the topics overview page.

Screenshots

Topics overview Topics statistics
joschi commented 1 year ago

https://github.com/redpanda-data/console/issues/83 describes a way for an approximate number of messages per partition by using the average message size (calculated by reading just some messages from the topic) and the partition sizes.

nmichaud2 commented 12 months ago

I think having it in the statistics is fine. I'm not sure it would be a good idea to approximate unless it is clearly indicated it is an approximation !