search

braedon / prometheus-kafka-consumer-group-exporter

Prometheus Kafka Consumer Group Exporter
MIT License
73 stars 39 forks source link

scrape kafka 2.6.0 error occurred #44

Closed ChaoHsupin closed 2 years ago

ChaoHsupin commented 2 years ago

My guess from the log is that current Exporter does not support the new version of Kafka

key to log:

kafka.errors.UnsupportedCodecError: UnsupportedCodecError: Libraries for lz4 compression codec not found

exporter logs：

[2022-06-16 03:05:16,311] root.INFO MainThread Requesting topics and partition assignments
[2022-06-16 03:05:16,312] root.INFO MainThread Requesting high-water marks
[2022-06-16 03:05:16,312] root.INFO MainThread Requesting low-water marks
[2022-06-16 03:05:16,710] root.INFO MainThread Received topics and partition assignments
[2022-06-16 03:05:25,832] root.INFO MainThread Requesting low-water marks
[2022-06-16 03:05:26,219] root.INFO MainThread Received high-water marks from node 1
[2022-06-16 03:05:26,228] root.INFO MainThread Received high-water marks from node 3
[2022-06-16 03:05:26,331] root.INFO MainThread Received high-water marks from node 2
[2022-06-16 03:05:30,856] root.INFO MainThread Requesting topics and partition assignments
[2022-06-16 03:05:30,857] root.INFO MainThread Requesting high-water marks
[2022-06-16 03:05:31,271] root.INFO MainThread Received high-water marks from node 2
[2022-06-16 03:05:31,337] root.INFO MainThread Received topics and partition assignments
[2022-06-16 03:05:31,341] root.INFO MainThread Received high-water marks from node 3
[2022-06-16 03:05:31,342] root.INFO MainThread Received high-water marks from node 1
Traceback (most recent call last):
  File "/usr/bin/prometheus-kafka-consumer-group-exporter", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/prometheus_kafka_consumer_group_exporter/__init__.py", line 143, in main
    for message in consumer:
  File "/usr/lib/python3.6/site-packages/kafka/consumer/group.py", line 1126, in __next__
    return next(self._iterator)
  File "/usr/lib/python3.6/site-packages/kafka/consumer/group.py", line 1096, in _message_generator
    for msg in self._fetcher:
  File "/usr/lib/python3.6/site-packages/kafka/consumer/fetcher.py", line 482, in __next__
    return next(self._iterator)
  File "/usr/lib/python3.6/site-packages/kafka/consumer/fetcher.py", line 396, in _message_generator
    self._next_partition_records = self._parse_fetched_data(completion)
  File "/usr/lib/python3.6/site-packages/kafka/consumer/fetcher.py", line 788, in _parse_fetched_data
    unpacked = list(self._unpack_message_set(tp, records))
  File "/usr/lib/python3.6/site-packages/kafka/consumer/fetcher.py", line 450, in _unpack_message_set
    for record in batch:
  File "/usr/lib/python3.6/site-packages/kafka/record/default_records.py", line 262, in __iter__
    self._maybe_uncompress()
  File "/usr/lib/python3.6/site-packages/kafka/record/default_records.py", line 171, in _maybe_uncompress
    self._assert_has_codec(compression_type)
  File "/usr/lib/python3.6/site-packages/kafka/record/default_records.py", line 114, in _assert_has_codec
    "Libraries for {} compression codec not found".format(name))
kafka.errors.UnsupportedCodecError: UnsupportedCodecError: Libraries for lz4 compression codec not found
[2022-06-16 03:05:59,913] kafka.client.INFO MainThread Bootstrapping cluster metadata from [('10.18.145.152', 9092, <AddressFamily.AF_INET: 2>)]
[2022-06-16 03:05:59,917] kafka.conn.INFO MainThread <BrokerConnection node_id=bootstrap host=10.18.145.152:9092 <connecting> [IPv4 ('10.12.145.152', 9092)]>: connecting to 10.12.145.152:9092 [('10.18.145.152', 9092) IPv4]
[2022-06-16 03:05:59,918] kafka.conn.INFO MainThread <BrokerConnection
braedon commented 2 years ago

@ChaoHsupin You probably just need to install the lz4 python library. See the kafka-python docs: https://kafka-python.readthedocs.io/en/master/install.html#optional-lz4-install

If you can confirm that adding the lz4 package fixes your issue, I'll look at updating the Docker image to install the various optional packages.

ChaoHsupin commented 2 years ago

@ChaoHsupin You probably just need to install the lz4 python library. See the kafka-python docs: https://kafka-python.readthedocs.io/en/master/install.html#optional-lz4-install

If you can confirm that adding the lz4 package fixes your issue, I'll look at updating the Docker image to install the various optional packages.

@braedon Thanks，i had solved it !