search

logstash-plugins / logstash-output-file

Apache License 2.0
23 stars 53 forks source link

Provide additional compression options to output to zip, tar.gz, etc.. #52

Open ppf2 opened 7 years ago

ppf2 commented 7 years ago

Currently, this plugin supports output to gzip, but not tarball gzip or zip or bzip2. It will be nice to provide additional compression options. Example use case here is use LS to get subset of documents from ES, scrub and transform the data, and output the documents to a file to feed into Rally for testing. Rally's documents setting currently supports only tar.gz, bz2 or zip files. Certainly, the end user can do post processing to convert the file themselves, so this is an ER for convenience.

ppf2 commented 7 years ago

Looking at the Rally code, it looks like we have since implemented support for decompressing gz files so this specific use case no longer applies. Will leave this issue open in case users out there can benefit from additional compression options in this input.

mutt13y commented 4 years ago

bzip2 is around 27% better compression of log files in my testing. Testing Logstash output I also see a 3.5% improvement using gzip --best bzip2 is MUCH slower of course which might be an issue. lzma is 45% better

it is possible that I am seeing the improvement due to the external utility's ability to chose different block size compared to the stream encoding.

I would like to see bz2 and lzma as a codec option also.