search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.2k stars 4.72k forks source link

Add support for LZ4 to System.Compression #17714

Closed redknightlois closed 4 years ago

redknightlois commented 8 years ago

As per requested in https://github.com/dotnet/corefx/issues/9657 I am adding a new one.

LZ4 is by far the fastest compression algorithm out there (at the expense of compression ratio that-is). It makes sense to have such an algorithm available on the baseline release.

We can also provide our port of our managed LZ4 implementation of the r131 (which includes acceleration) if that makes the official inclusion of an algorithm we heavily rely upon easier. We are still working on the perf front but it is currently faster than the one that we had https://github.com/Corvalius/ravendb/blob/lz4-131/src/Sparrow/Compression/LZ4.cs

GSPP commented 8 years ago

From what I could quickly find on the web it seems that indeed LZ4 is the fastest and has the best compression in it's speed category.

redknightlois commented 8 years ago

@GSPP yes and the latest build also allows you to tune that with the acceleration parameter. You can try the port we did of the latest one from this code: https://github.com/Corvalius/ravendb/blob/v4.0/src/Sparrow/Compression/LZ4.cs

It is perf tuned but not that much... On Gen2 i5 and 1333Mhz we are memory bound. I still have a pass to be done in a few months on an Skylake DDR4, 3200Mhz memory to see if there is place for improvement. But as I mentioned before, we would gladly donate the implementation if anyone is interested into do the work to include that here.

ianhays commented 8 years ago

Next step here is to provide an API suggestion to get discussion moving. LZ4 support is a reasonable suggestion and something we would benefit from having.

karelz commented 4 years ago

Triage: This is sitting here for a long time, but no interest. It seems it is not worth tracking it at this moment. Looks like larger work anyway.