search

bkaradzic / go-lz4

Port of LZ4 lossless compression algorithm to Go
BSD 2-Clause "Simplified" License
211 stars 23 forks source link

LZ4 Framing #16

Open rektide opened 9 years ago

rektide commented 9 years ago

I'd like to see LZ4 framing supported.

The frame-descriptor includes an optional content-length, which may help to eliminate go-lz4's unique means of storing such:

go-lz4 saves a uint32 with the original uncompressed length at the beginning of the encoded buffer

dgryski commented 9 years ago

This method is common to most of the libraries that wrap lz4 (python, ruby, etc.)

I agree proper framing would be nice to have.

calmh commented 8 years ago

Would you accept a PR to add a function lz4.DecodeLength(dst, src []byte, length int) ([]byte, error)? This function would operate on data that doesn't have the four byte preamble.

The reasoning is that it's easy to get the desired format from Encode() when you don't want the length preamble - just chop off the first four bytes of the returned buffer. The other direction is more annoying if you happen to already have a buffer with data without length preamble - at minimum it requires constructing a new buffer with the length at front and copying the compressed data.

By extracting the decode function for a known length, the existing Decode() can trivially delegate to this new function after extracting the length from the head of the buffer.

(The existing format is also unuseful to us because the length is encoded in little endian byte order rather than network byte order...)