Open lachie opened 10 years ago
The io.Reader/io.Writer interface was removed in https://github.com/bkaradzic/go-lz4/commit/5298eaf858a62a1a1195bebb8e280ffca025237d due to bugs and slowness.
I agree having a streaming interface would be useful. IIRC, the old C library faked streaming by buffering in (~65k?) blocks and compressing in batches. You could do something similar and construct your own framing, assuming you don't need compatibility with any other implementation.
I see there is now a more complete spec for an LZ4 streaming format: https://docs.google.com/document/d/1gZbUoLw5hRzJ5Q71oPRN6TO4cRMTZur60qip-TE7BhQ/edit . We should probably support this format.
Looks like the streaming format isn't completely implementing in the C lz4 implementation: https://code.google.com/p/lz4/issues/detail?id=42
ok no worries, that sounds like a good solution
Leaving this open as a reminder to put back the streaming support.
Any progress about steaming support ?
Spec has been finalised. I'll add it nearer the top of my todo list.
@dgryski hello, do you have any news about streaming support?
ping...
I'm unlikely to get to this anytime soon. I will glady review any merge any pull requests for this feature.
Sorry :(
2015-01-20 13:39 GMT+03:00 Damian Gryski notifications@github.com:
I'm unlikely to get to this anytime soon. I will glady review any merge any pull requests for this feature.
Sorry :(
Ok. May be i'm try.
Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru jabber: vase@selfip.ru
ping...
Any progress so far?
Or does anyone has good working example for using blocks and compressing in batches?
Some of the other Go lz4 ports have implemented the streaming interface: https://github.com/pierrec/lz4 for example
Thank you for this hint. It was way harder to implement this myself than I thought.
I'd love to use this code, but my project involves compressing files up to gigabytes in size.
For that use case I really need a streaming version of the algorithm (i.e. using the io.Writer/io.Reader interfaces). I see there's one as part of the reference C implementation, but I really have no idea where to start with porting it to Go.