search

bkaradzic / go-lz4

Port of LZ4 lossless compression algorithm to Go
BSD 2-Clause "Simplified" License
211 stars 23 forks source link

encode with io.Writer / decode with io.Reader #14

Open lachie opened 10 years ago

lachie commented 10 years ago

I'd love to use this code, but my project involves compressing files up to gigabytes in size.

For that use case I really need a streaming version of the algorithm (i.e. using the io.Writer/io.Reader interfaces). I see there's one as part of the reference C implementation, but I really have no idea where to start with porting it to Go.

dgryski commented 10 years ago

The io.Reader/io.Writer interface was removed in https://github.com/bkaradzic/go-lz4/commit/5298eaf858a62a1a1195bebb8e280ffca025237d due to bugs and slowness.

I agree having a streaming interface would be useful. IIRC, the old C library faked streaming by buffering in (~65k?) blocks and compressing in batches. You could do something similar and construct your own framing, assuming you don't need compatibility with any other implementation.

I see there is now a more complete spec for an LZ4 streaming format: https://docs.google.com/document/d/1gZbUoLw5hRzJ5Q71oPRN6TO4cRMTZur60qip-TE7BhQ/edit . We should probably support this format.

dgryski commented 10 years ago

Looks like the streaming format isn't completely implementing in the C lz4 implementation: https://code.google.com/p/lz4/issues/detail?id=42

lachie commented 10 years ago

ok no worries, that sounds like a good solution

dgryski commented 10 years ago

Leaving this open as a reminder to put back the streaming support.

dgryski commented 10 years ago

http://fastcompression.blogspot.fr/2014/05/streaming-api-for-lz4.html

vtolstov commented 10 years ago

Any progress about steaming support ?

dgryski commented 10 years ago

Spec has been finalised. I'll add it nearer the top of my todo list.

vtolstov commented 9 years ago

@dgryski hello, do you have any news about streaming support?

vtolstov commented 9 years ago

ping...

dgryski commented 9 years ago

I'm unlikely to get to this anytime soon. I will glady review any merge any pull requests for this feature.

Sorry :(

vtolstov commented 9 years ago

2015-01-20 13:39 GMT+03:00 Damian Gryski notifications@github.com:

I'm unlikely to get to this anytime soon. I will glady review any merge any pull requests for this feature.

Sorry :(

Ok. May be i'm try.

Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru jabber: vase@selfip.ru

xxorde commented 7 years ago

ping...

Any progress so far?

Or does anyone has good working example for using blocks and compressing in batches?

dgryski commented 7 years ago

Some of the other Go lz4 ports have implemented the streaming interface: https://github.com/pierrec/lz4 for example

xxorde commented 7 years ago

Thank you for this hint. It was way harder to implement this myself than I thought.