PSeitz / lz4_flex

Fastest pure Rust implementation of LZ4 compression/decompression.
MIT License
423 stars 29 forks source link

provide size of uncompressed data #159

Open jtmoon79 opened 1 month ago

jtmoon79 commented 1 month ago

Can lz4_flex provide an API for getting the uncompressed size of data before decompressing that data? I assume the uncompressed size is in some kind of block header or something like that? (I don't know the LZ4 binary format; just guessing)

This would be particularly helpful for lz4 compressed files.

PSeitz commented 1 month ago

There's the content_size flag in the frame format header, but it's existence is optional. Also there may be multiple frames. So it would be the sum of the content_size.

This would be particularly helpful for lz4 compressed files.

Why would that be helpful?

jtmoon79 commented 1 month ago

There's the content_size flag in the frame format header, but it's existence is optional.

How do I access the content_size for some file? Can you complete this code example:

use std::fs::File;
use std::fs::OpenOptions;
use std::io::prelude::Read;
use std::io::BufReader;
use std::path::Path;
use ::lz4_flex;

fn main() {
    let mut open_options = OpenOptions::new();
    let path = String::from("/tmp/file.lz4");
    let path_std = Path::new(&path);
    let file_lz: File = match open_options.read(true).open(path_std) {
        Ok(val) => val,
        Err(err) => panic!("{}", err),
    };
    let bufreader = BufReader::<File>::new(file_lz);
    let mut lz4_decoder = lz4_flex::frame::FrameDecoder::new(bufreader);
    // ... how to access content_size ?
}

I created /tmp/file.lz4 with

$ lz4c --version
*** LZ4 command line interface 64-bits v1.9.3, by Yann Collet ***
$ lz4c -9kv /var/log/syslog -c > /tmp/file.lz4
PSeitz commented 1 month ago

current_frame_info in FrameDecoder is private and depends on the current frame

jtmoon79 commented 1 month ago

There's the content_size flag in the frame format header, but it's existence is optional... So it would be the sum of the content_size. current_frame_info in FrameDecoder is private and depends on the current frame

I'm confused. Could you please provide a code example?

PSeitz commented 1 month ago

It's not accessible currently. It's existence is optional, so not sure what it can be used for externally