PSeitz / lz4_flex

Fastest pure Rust implementation of LZ4 compression/decompression.
MIT License
441 stars 28 forks source link

lz4 writer does not close properly #94

Closed rongcuid closed 1 year ago

rongcuid commented 1 year ago

When I use FrameEncoder, the last part of the file is not actually written properly:

let output = Box::new(lz4_flex::frame::FrameEncoder::new(File::create(output)?));
// many write calls
output.flush()?;

Using lz4cat on resulting file gives Error 68: unfinished stream.

I need this boxing because I am generalizing multiple output backends with Box<dyn Write>. I am not sure if this is causing the issue here, but flate2 with its gzip decoder closes files properly.

rongcuid commented 1 year ago

As a note, since I am using Box<dyn Writer>, I cannot actually call .finish(). In my opinion, this should really be called on drop().

edit:

zstd provides a type called AutoFinishEncoder. Maybe you can add something like this?

PSeitz commented 1 year ago

Thanks for the suggestion, I added auto_finish for FrameEncoder on the main branch. Can you test if that works for you?

rongcuid commented 1 year ago

I think it works.

PSeitz commented 1 year ago

Okay cool, I'll try to make a release next week

rongcuid commented 1 year ago

I'd just like to check on the status, thanks

PSeitz commented 1 year ago

I'm preparing a release. I did a lot of changes before auto_finish and want to make sure everything works properly (fuzzing, benchmarks etc.).

PSeitz commented 1 year ago

Related issue https://github.com/ouch-org/ouch/pull/163#discussion_r1108965151

PSeitz commented 1 year ago

Released with 0.11