entire `.lz4` files must be read before processing

jtmoon79 / super-speedy-syslog-searcher

Speedily search and merge log messages by datetime

MIT License

31 stars 1 forks source link

entire `.lz4` files must be read before processing #293

Open jtmoon79 opened 1 month ago

jtmoon79 commented 1 month ago

Summary

The entire .lz4 file is uncompressed to get it's uncompressed size before processing.

Current behavior

Crate lz4_flex does not provide an API for getting the .lz4 file uncompressed size. See https://github.com/PSeitz/lz4_flex/issues/159 So the entire file is uncompressed to get the uncompressed file size.

Suggested behavior

One or both

Submit a PR to lz4_flex.
refactor blockreader (and all readers) to not need a file size (difficult)

jtmoon79 commented 1 month ago

Follow on from #291

jtmoon79 commented 1 month ago

By default, .lz4 files compressed by lz4c do not include a content_size.

$ lz4c --help
*** LZ4 command line interface 64-bits v1.9.3, by Yann Collet ***
Usage :
      lz4c [arg] [input] [output]

...
--content-size : compressed frame includes original size (default:not present)

So even if I could access content_size in the FrameHeader, it would most often not be valid. So the current implementation of first reading the entire file would still be required.