search

mwotton / lz4hs

Haskell bindings to lz4
BSD 3-Clause "New" or "Revised" License
21 stars 7 forks source link

plans to add streaming API? #12

Open charles-cooper opened 9 years ago

charles-cooper commented 9 years ago

Are there any plans to add support for the new LZ4 Streaming API? This would be useful for programs which don't want to strictly decompress an entire stream in memory.

http://fastcompression.blogspot.com/2014/05/streaming-api-for-lz4.html

jjl commented 8 years ago

I've started integrating this in my 'frame' branch https://github.com/jjl/lz4hs/commit/65c982368fd3f1e027bf6e659730969a9b5b28ea

mwotton commented 8 years ago

That's great to hear, please PR when you're done :)

On Sun, 10 Apr 2016 12:19 pm James Laver notifications@github.com wrote:

I've started integrating this in my 'frame' branch jjl@65c9823 https://github.com/jjl/lz4hs/commit/65c982368fd3f1e027bf6e659730969a9b5b28ea

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/mwotton/lz4hs/issues/12#issuecomment-208008075

charles-cooper commented 8 years ago

@jjl you may be interested in some work I did in https://github.com/charles-cooper/lz4hs/blob/master/Codec/Compression/LZ4/Decompress.hs

jjl commented 8 years ago

[@charles-cooper thanks for that. My haskell FFI knowledge is a bit limited, so I'm looking forward to cribbing from your attempt :)

Just to manage expectations, I'm anticipating closing this off sometime in April depending on how my other work goes.

Some of my notes so far:

My use case is a special snowflake. I'm aiming to have 64kb max blocks whereby I'd like to aim for 64kb compressed blocks, but rather than overflow the remaining data, i'd like to detect that the last compressed piece would overflow the buffer and keep it behind for the next block so that data isn't split. I'm intending to use it in conjunction with a Rope of ByteStrings.

What are other people looking to use this for? I'd like to bear this in mind when I finish the work.

Cheers

nh2 commented 7 years ago

Note I have picked up this task of writing an LZ4 frame implementation as part of a conduit (looking into what you guys have in your branches of course). Not quite done yet (compression works and is compatible with the lz4 command line utility, but I still have to do decompression), but I thought I'd leave that here in order to avoid duplicate work.