search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
37 stars 449 forks source link

Support Producer and Consumer datastreams in the Kafka Integration #11485

Open ishleenk17 opened 1 month ago

ishleenk17 commented 1 month ago

Add support for the consumer and producer datastream in the Kafka Integration. The corresponding metricsets are present in the Beats Kafa module.

ishleenk17 commented 1 month ago

I cross-checked the beats module. The producer metrics are not present in any of the datastreams in Integration. We should add them. cc: @lalit-satapathy

lalit-satapathy commented 1 month ago

The kafka consumer and producer metrics are based on Jolokia JMX. They monitor the Java app. These Jolokia metrics can be fetched by jolokia input also. We can add this, if needed as a priority.

But, it is slightly different from actual Kafka topics monitoring done by consumergroup and producer.