search

steeve / python-lz4

LZ4 bindings for python
http://github.com/steeve/python-lz4
105 stars 31 forks source link

big-endianness for size #25

Open facboy opened 10 years ago

facboy commented 10 years ago

it would be great if you could specify in the method calls (loads and dumps) whether the size header was little endian or big endian.

steeve commented 10 years ago

The size is always stored in little endian, to guarantee interop between archs. See https://github.com/steeve/python-lz4/blob/master/src/python-lz4.c#L44

facboy commented 10 years ago

actually i'm reading data from a different library (hadoop as it happens), and its block compression uses a completely different scheme. basically it has the original size as a big-endian int32, followed by the compressed size as a big-endian int32, and then the compressed data. so a simple option wouldn't have worked in any case. maybe the ability to specify the size and then treat the data as merely the compressed stream?