search

Morningstar / kafka-offset-monitor

A small web app to monitor the progress of kafka consumers and their lag wrt the log.
Apache License 2.0
281 stars 109 forks source link

Web page stops listing details for consumer groups after a period of time #37

Open williamla opened 6 years ago

williamla commented 6 years ago

Hi there,

I have this app (kom) running across three different environments, and have noticed that within 24 hours in my prod environment, the interface stops updating. I can see a static list of consumers, but clicking into any of the consumers will show no further details outside of the templated page.

image

The prod environment has a lot more traffic than the others, but also has enough CPU and RAM allocated to the service. Based on the console logs, kom is still running and checking/reading committed offsets as well. Any idea on what this could be?

williamla commented 6 years ago

Taking a look at the console logs on a fresh restart, it appears that kom gets stuck processing/updating the committed offsets for a high-frequency committing consumer and isn't able to update the offsets for the other topics as often or for long periods of time.

Eventually, it does get around to the other consumers. After some time, the committed offset in the web interface will remain static while the LogEndOffset and Lag will increase. The Last Seen calculation will also be thrown off like below:

image

image