search

t-mat / lz4mt

Platform independent, multi-threading implementation of lz4 stream in C++11
Other
199 stars 47 forks source link

lz4mt c++ library for decompression #41

Closed roopa01 closed 2 years ago

roopa01 commented 3 years ago

Hi,

I am trying to decompress a data block with lz4mt library.Data is compressed with lz4 compression algorithm. Data is 0000008DA20A1102030000000120000100120407003000003C05000F02000F13022300134408000610001110120004100031740080100081FF0001000F070A810D000F020000131A1400132E0800F510878065AD8F0000005A1A8EA5A24A00007C169306B4E3150052B54CA0D7FEB8AF000F0200FFE61F01F901FFE60F0200FFFFFFFFFFFFFFFFFFFFFF3A50000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

This is 512 bytes block. I have written this data in a text file. I tried LZ4_decompress_safe() function for decompression but it is returns negative value -15. please tell me what is the problm with this.Is this library is compitable to decompress the data block.

t-mat commented 3 years ago

Hi @roopa01, thanks for the report.

First of all, I suppose this issue is caused by confusion.

I must emphasis that this confusion is not your failure. But this confusion and complexity are nature of compression algorithm and its applications.

Please let me explain about some details and terminology in LZ4.

Now, please note that lz4mt always uses LZ4 frame format.

Since first 4 bytes of LZ4 frame format is 0x04, 0x22, 0x4d, 0x18, it seems you're using (low level) LZ4 block format. You also stated that you've used LZ4_decompress_safe() as a compression function. Since it outputs LZ4 block format, it can't be compatible with LZ4 frame format which lz4mt supports.

please tell me what is the problm with this.

If you need to use lz4mt, you must use LZ4 Frame Format.

Is this library is compitable to decompress the data block.

lz4mt is fully compatible with LZ4 Frame Format. But it isn't compatible with LZ4 Block format.

roopa01 commented 3 years ago

First of all thanks a lot for quick response. Actually i am working ZFS( Zetta byte file system) on linux and windows .In which whole file system metadata blocks are compressed with lz4 algorithm. So i want to decompress these data blocks.Size of metadata blocks can vary.I am able to decompress these blocks on linux using zdb a command line utility but on windows there are no any option for this. If you are able to understand my problem please help me.

Thanks roopa

t-mat commented 3 years ago

If you are able to understand my problem please help me.

I have no idea. I'm really sorry. But it's out of scope of lz4mt and lz4. Since your problem is strictly related to internal of ZFS, please ask the ZFS team directly.