Uncompress lz4ultra files?

[richarddd]

Hey,

Thanks for an awesome crate!

I'm facing issues when trying to decompress files that have been compressed with https://github.com/emmanuel-marty/lz4ultra I'm using block::decompress_size_prepended and ofc prepended the size of the uncompressed data into my compressed byte array as LE.

Command::new("lz4ultra")
    .arg("--favor-decSpeed")
    .arg(&tmp_filename)
    .arg(&filename)
    .output()?;

let bytes = fs::read(&filename)?;

let mut compressed = Vec::with_capacity(4);

compressed.extend_from_slice(&uncompressed_file_size.to_le_bytes());
compressed.extend_from_slice(&bytes);
compressed.truncate(4 + compressed.len());

fs::write(&filename, compressed)?;

Failes with: thread 'main' panicked at 'assertion failed: start_ptr >= output_base

In here decompress.rs:300:13

let start_ptr = unsafe { output_ptr.sub(offset) };
debug_assert!(start_ptr >= output_base);

[PSeitz]

Does lz4ultra use the frame or the block format? Probably the frame format

the truncate line doesn't make sense

compressed.truncate(4 + compressed.len());

[richarddd]

You're right, I think it's the frame format. https://github.com/emmanuel-marty/lz4ultra/blob/master/src/frame.c

So then i'm probably using the wrong format. Do you have any examples how this could be done? (reading from compressed byte array, or similar)

[PSeitz]

Yes in the docs: https://docs.rs/lz4_flex/latest/lz4_flex/

Or the tests: https://github.com/PSeitz/lz4_flex/blob/main/tests/tests.rs#L68

[richarddd]

Yes in the docs: https://docs.rs/lz4_flex/latest/lz4_flex/

Or the tests: https://github.com/PSeitz/lz4_flex/blob/main/tests/tests.rs#L68

Thank you. I tired that but getting: "read after the end of the buffer"

Maybe lz4ultra uses the hc format which i know isn't supported yet 🤔

[PSeitz]

there is no hc format, just a hc algorithm which outputs the same format

Can you try if it decompresses with the official lz4 cli ? If not, then it's just incompatible with lz4

[richarddd]

there is no hc format, just a hc algorithm which outputs the same format

Can you try if it decompresses with the official lz4 cli ? If not, then it's just incompatible with lz4

Yeah that works.

Tested with simple data and it works: https://www.rustexplorer.com/b#%2F*%0A%5Bdependencies%5D%0Alz4_flex%20%3D%20%7B%20version%20%3D%20%220.10%22%2C%20default-features%20%3D%20false%2C%20features%3D%5B%22frame%22%5D%20%7D%0A%0A*%2F%0A%0Afn%20main()%20%7B%0A%0A%20%20%20%20let%20ultra%3AVec%3Cu8%3E%20%3D%20vec!%5B0x04%2C0x22%2C0x4d%2C0x18%2C0x40%2C0x40%2C0xc0%2C0x0c%2C0x00%2C0x00%2C0x80%2C0x48%2C0x65%2C0x6c%2C0x6c%2C0x6f%2C0x20%2C0x57%2C0x6f%2C0x72%2C0x6c%2C0x64%2C0x21%2C0x00%2C0x00%2C0x00%2C0x00%5D%3B%0A%20%20%20%20let%20std%3AVec%3Cu8%3E%20%3D%20%20%20vec!%5B0x04%2C0x22%2C0x4d%2C0x18%2C0x64%2C0x40%2C0xa7%2C0x0c%2C0x00%2C0x00%2C0x80%2C0x48%2C0x65%2C0x6c%2C0x6c%2C0x6f%2C0x20%2C0x57%2C0x6f%2C0x72%2C0x6c%2C0x64%2C0x21%2C0x00%2C0x00%2C0x00%2C0x00%2C0x88%2C0x97%2C0xd6%2C0x0b%5D%3B%0A%20%20%20%20let%20ultra%3A%20%26%5Bu8%5D%20%3D%20%26ultra%3B%0A%20%20%20%20let%20std%3A%20%26%5Bu8%5D%20%3D%20%26std%3B%0A%20%20%20%20let%20mut%20de%20%3D%20lz4_flex%3A%3Aframe%3A%3AFrameDecoder%3A%3Anew(ultra)%3B%0A%20%20%20%20let%20mut%20out%20%3D%20Vec%3A%3Anew()%3B%0A%20%20%20%20std%3A%3Aio%3A%3ARead%3A%3Aread_to_end(%26mut%20de%2C%20%26mut%20out).unwrap()%3B%0A%20%20%20%20%0A%20%20%20%20%0A%0A%20%20%20%20println!(%22%7B%7D%22%2CString%3A%3Afrom_utf8(out).unwrap())%3B%0A%0A%0A%20%20%20%20let%20mut%20de%20%3D%20lz4_flex%3A%3Aframe%3A%3AFrameDecoder%3A%3Anew(std)%3B%0A%20%20%20%20let%20mut%20out%20%3D%20Vec%3A%3Anew()%3B%0A%20%20%20%20std%3A%3Aio%3A%3ARead%3A%3Aread_to_end(%26mut%20de%2C%20%26mut%20out).unwrap()%3B%0A%0A%20%20%20%20println!(%22%7B%7D%22%2CString%3A%3Afrom_utf8(out).unwrap())%3B%0A%7D

[richarddd]

Doh, it was my misstake. It works fine!