search

MiloszKrajewski / K4os.Compression.LZ4

LZ4/LH4HC compression for .NET Standard 1.6/2.0 (formerly known as lz4net)
MIT License
675 stars 77 forks source link

Implement frame format with K4os.Compression.LZ4 #79

Closed eerhardt closed 1 year ago

eerhardt commented 1 year ago

Description Apache Arrow IPC uses the frame format (https://github.com/apache/arrow/blob/apache-arrow-11.0.0/format/Message.fbs#L46-L48). We'd like to use the K4os.Compression.LZ4 nuget package to decompress arrow messages because we have Span input and outputs. However, according to https://github.com/apache/arrow/pull/33893#discussion_r1118148887, only the K4os.Compression.LZ4.Streams library implements the LZ4 frame format.

To reproduce Steps to reproduce the behavior: Add the K4os.Compression.LZ4 package to your library. Attempt to use it to decompress LZ4 frame input.

        public int Decompress(ReadOnlyMemory<byte> source, Memory<byte> destination)
        {
            return LZ4Codec.Decode(source.Span, destination.Span);
        }

Expected behavior The data should be decoded successfully.

Actual behavior -1 is returned and no data is decoded.

Environment

Additional context Add any other context about the problem here.

adamreeve commented 1 year ago

I'm the author of the referenced PR. After looking into this some more I realised I'd missed the existence of the LZ4Frame class in the Streams library, which supports the use of the frame format with a range of different data types so we don't have to work with a Stream specifically (https://github.com/MiloszKrajewski/K4os.Compression.LZ4#other-stream-like-data-structures).

So apologies for the noise and I believe this can be closed.

eerhardt commented 1 year ago

Closing per @adamreeve's comment above. Thanks, @adamreeve.