search

MiloszKrajewski / K4os.Compression.LZ4

LZ4/LH4HC compression for .NET Standard 1.6/2.0 (formerly known as lz4net)
MIT License
675 stars 77 forks source link

Add K4os.Compression.LZ4.Buffers prototype #74

Open rmja opened 2 years ago

rmja commented 2 years ago

This is a working initial prototype of LZ4FrameEncoder and LZ4FrameDecoder that works using PipeReader and PipeWriter. It is put in a separate K4os.Compression.LZ4.Buffers project as I do not have the correct csproj skillset to get this working in K4os.Compression.LZ4.Streams because of the very old target frameworks. This currently targets .NET 6 - it probably needs to be relaxed, but I need help to accomplish this.

If the target frameworks can be the same as K4os.Compression.LZ4.Streams then I see three proposals:

  1. Add this code directly in K4os.Compression.LZ4.Streams and use it as the fundamentals for the currently exported stream classes.
  2. Let this be in a different assembly/nuget which can be referenced by K4os.Compression.LZ4.Streams to accomplish the same task as 1.
  3. Do nothing with K4os.Compression.LZ4.Streams and have the two different implementations.

This implementation supports almost all features in a LZ4 frame, including ContentSize, block and content checksums, etc. except the dictionary field.

Feel free to make any comments and commit to this pr.

This pr replaces the previous gist https://gist.github.com/rmja/98dc7e0576c933faa0a75629b46af71c.

This pr fixes #14.

cc @MiloszKrajewski @aarnott

AArnott commented 2 years ago

I love it. You share nearly all the code between the two modes, and it seems to be truly streaming. Wonderful job, IMO. Thank you.

I left you a present. Happy to talk about it.

rmja commented 2 years ago

@MiloszKrajewski We now have a reasonably good proposal for a pipe streaming based encoder and decoder, and we now need your inputs to proceed from here. Let us know what you think ! cc @AArnott

MiloszKrajewski commented 2 years ago

I am not ignoring you, believe me. I just have my own branch with similar goal (see: abstracted-streams) and I need to wrap my head around it.

MiloszKrajewski commented 1 year ago

This PR has a lot of good code and feel bad about not using it. At the same time, it no longer applies in a sense, and baseline already has this functionality. So I decided to keep it a pipelines-prototype and look at this later, while deleting the PR because it won't be merged in this form probably ever.