Kafka: Emit production rate

[arunramani]

Description

Add metric to track Kafka message production rate which can be useful for correlating with Kafka lag. For each collection period, the latest Kafka broker offset is compared with the previous minute to calculate the production rate.

One interesting use case is being able to use this metric to roughly calculate lag as time. This can be done by calculating the time t such that sum(time = now to t) of productionRate = lag i.e. how many minutes of production equals the current lag.

Release note

Added a new streaming ingest metric ingest/kafka/partitionProduction

---

Key changed/added classes in this PR

• SeekableStreamingSupervisor
• KafkaSupervisor

---

This PR has:

• [x] been self-reviewed.
• [x] added documentation for new or modified features or behaviors.
• [ ] a release note entry in the PR description.
• [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• [ ] added integration tests.
• [ ] been tested in a test Druid cluster.

[kfaraz]

@arunramani , could you please share some details as to how this metric would provide additional insight that is not already covered by metrics like Kafka lag, partition lag and/or message gap?