search

steeve / python-lz4

LZ4 bindings for python
http://github.com/steeve/python-lz4
106 stars 31 forks source link

big-endianness for size #25

Open facboy opened 9 years ago

facboy commented 9 years ago

it would be great if you could specify in the method calls (loads and dumps) whether the size header was little endian or big endian.

steeve commented 9 years ago

The size is always stored in little endian, to guarantee interop between archs. See https://github.com/steeve/python-lz4/blob/master/src/python-lz4.c#L44

facboy commented 9 years ago

actually i'm reading data from a different library (hadoop as it happens), and its block compression uses a completely different scheme. basically it has the original size as a big-endian int32, followed by the compressed size as a big-endian int32, and then the compressed data. so a simple option wouldn't have worked in any case. maybe the ability to specify the size and then treat the data as merely the compressed stream?