Parsely / pykafka

Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance.
http://pykafka.readthedocs.org/
Apache License 2.0
1.12k stars 232 forks source link

`decode_messages` crashes when payload ends in a header #80

Closed kbourgoin closed 11 years ago

kbourgoin commented 11 years ago

This checks for the payload ending in the middle of a message: https://github.com/Parsely/samsa/blob/master/samsa/client.py#L288

However, if the payload ends a couple bytes after the message, the header struct decoding fails with:

Exception in thread Thread-28:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/data/virtualenvs/ptrack-nextgen/local/lib/python2.7/site-packages/samsa/consumer/partitions.py", line 131, in _fetch
    for message in messages:
  File "/data/virtualenvs/ptrack-nextgen/local/lib/python2.7/site-packages/samsa/client.py", line 279, in decode_messages
    header = Message.Header.unpack_from(payload, offset)
  File "/data/virtualenvs/ptrack-nextgen/local/lib/python2.7/site-packages/samsa/utils/namedstruct.py", line 41, in unpack_from
    unpacked = super(NamedStruct, self).unpack_from(*args, **kwargs)
error: unpack_from requires a buffer of at least 5 bytes
kbourgoin commented 11 years ago

See the attached commit for a fix.