Exporter crashes with error: struct.error: unpack_from requires a buffer of at least 2 bytes

[jcanamero]

Hi there,

I am running version 0.4.0 vía Docker image on several systems. The docker container enters in a re-starting loop, returning the following output:

_Traceback (most recent call last): File "/usr/local/bin/prometheus-kafka-consumer-group-exporter", line 11, in load_entry_point('prometheus-kafka-consumer-group-exporter', 'console_scripts', 'prometheus-kafka-consumer-group-exporter')() File "/usr/src/app/prometheus_kafka_consumer_group_exporter/init.py", line 155, in main key = parse_key(message.key) File "/usr/src/app/prometheus_kafka_consumer_group_exporter/parsing.py", line 33, in parse_key (topic, remaining_key) = read_string(remaining_key) File "/usr/src/app/prometheus_kafka_consumer_group_exporter/parsing.py", line 23, in read_string length, remaining = read_short(bytes) File "/usr/src/app/prometheus_kafka_consumer_group_exporter/parsing.py", line 5, in read_short num = unpack_from('>h', bytes)[0] struct.error: unpackfrom requires a buffer of at least 2 bytes

It seems there is some issue within the code (parsing.py file, read_short function), my python is not good enough to get things working. I hope someone could find a solution.

Thanks in advance

[akunszt]

We also had this error. The code assumes that the message.key will be at least 8 bytes but in case it gets only 4 bytes: b'\x00\x00\x00\x01' I don't know if it's a legit Kafka message key or not. As a quick fix we applied this patch to ignore this kind of messages:

--- a/prometheus_kafka_consumer_group_exporter/__init__.py
+++ b/prometheus_kafka_consumer_group_exporter/__init__.py
@@ -2,6 +2,7 @@ import argparse
 import javaproperties
 import logging
 import signal
+import struct
 import sys

 from jog import JogFormatter
@@ -152,7 +153,11 @@ def main():
                 collectors.set_exporter_offsets(exporter_offsets)

                 if message.key and message.value:
-                    key = parse_key(message.key)
+                    try:
+                       key = parse_key(message.key)
+                    except struct.error:
+                       key = None
+
                     if key:
                         value = parse_value(message.value)

As you can see I'm not an expert in Python either. I think there is a nicer way to handle this and I still don't know what was the root cause. It just fixes the crashes and let the exporter run.

[stefan-hudelmaier]

We are also running into this problem. The key fails to parse (parse_key fails when trying to read the topic as remaining_key has length 0).

The fix by @jutley resolves the issue for us.

[braedon]

@jcanamero fixed in 0.5.0