search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
112 stars 4.93k forks source link

Metricbeat not able to collect metrics from Kerberised Kafka #11420

Closed mayank-mahajan-guavus closed 1 year ago

mayank-mahajan-guavus commented 5 years ago

Issue: Metric beat is not to collect metrics from Kerberized Kafka. It works well with non-kerberized Kafka

logs snapshot: Logs when metric beat is running on kerberised setup:

} 2019-03-25T05:51:14.506Z INFO kafka/log.go:53 Closed connection to broker testautomation-slv-01.gvs.ggn:6667

2019-03-25T05:51:14.507Z INFO kafka/log.go:53 Connected to broker at testautomation-slv-01.gvs.ggn:6667 (unregistered)

2019-03-25T05:51:14.757Z INFO kafka/log.go:53 Closed connection to broker testautomation-slv-01.gvs.ggn:6667

2019-03-25T05:51:14.758Z DEBUG [publish] pipeline/processor.go:308 Publish event: { "@timestamp": "2019-03-25T05:51:14.001Z", "@metadata": { "beat": "metricbeat", "type": "doc", "version": "6.6.2" }, "error": { "message": "EOF" }, "metricset": { "name": "consumergroup", "module": "kafka", "host": "testautomation-slv-01.gvs.ggn:6667", "rtt": 756169 }, "host": { "name": "testautomation-slv-01.gvs.ggn" }, "beat": { "name": "testautomation-slv-01.gvs.ggn", "hostname": "testautomation-slv-01.gvs.ggn", "version": "6.6.2" }, "event": { "dataset": "kafka.consumergroup", "duration": 756169315 } } 2019-03-25T05:51:15.310Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://192.168.194.143:9200)) 2019-03-25T05:51:15.311Z DEBUG [elasticsearch] elasticsearch/client.go:697 ES Ping(url=http://192.168.194.143:9200) 2019-03-25T05:51:15.314Z DEBUG [elasticsearch] elasticsearch/client.go:720 Ping status code: 200 2019-03-25T05:51:15.314Z INFO elasticsearch/client.go:721 Connected to Elasticsearch version 6.2.4 2019-03-25T05:51:15.315Z DEBUG [elasticsearch] elasticsearch/client.go:739 HEAD http://192.168.194.143:9200/_template/jyoti-miqproddc-new_kerb1 2019-03-25T05:51:15.323Z INFO template/load.go:130 Template already exists and will not be overwritten. 2019-03-25T05:51:15.323Z INFO instance/beat.go:894 Template successfully loaded. 2019-03-25T05:51:15.323Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://192.168.194.143:9200)) established 2019-03-25T05:51:15.330Z DEBUG [elasticsearch] elasticsearch/client.go:323 PublishEvents: 2 events have been published to elasticsearch in 7.39981ms. 2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/ackloop.go:160 ackloop: receive ack [0: 0, 2] 2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=2, start-seq=1, end-seq=2

2019-03-25T05:51:15.330Z DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:2

jsoriano commented 5 years ago

Hi @mayank-mahajan-guavus, thanks for opening this issue, I think there would be two things to do here:

botelastic[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

iatalay commented 4 years ago

Does metricbeat currently support input from kerberised kafka?

wangqinghuan commented 3 years ago

Could anyone help to implement this pr?

botelastic[bot] commented 2 years ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

DaveyDevOps commented 2 years ago

In my limited experience kerberised kafka is usually part of a larger vendor supported product, those product typically ship with their own set of monitoring which reduces the needed for metricbeat. However I think it would be to good to enable this feature for completeness of coverage.

botelastic[bot] commented 1 year ago

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!