logstash-input-gunzip - Contribution Logstash Filter Plugin - reading from gz file input

[PascalNoisette]

Hi,

I wrote an input plugin to read gziped file. I use this plugin to parse log file that has already been logrotated, and also to parse huge files that could not be decompressed easily.

Indeed a codec concerning gziped data already exist but it does not have the same purpuse (a codec is not an input). This kind of plugin has been discussed on the forum several time here or requested here and here. So I followed the guidelines up to the point you suggest to create an issue in the elastic/logstash repo to introduice my plugin to you.

Source code can be found on github and a first release can be downloaded via rubygems. It can be installed and tested quickly.

bin/logstash-plugin install logstash-input-gunzip
bin/logstash -e 'input { gunzip { path => ["/var/log/syslog.2.gz", "/var/log/syslog.3.gz"] } } output {stdout { codec => rubydebug }}'

Regards, Pascal Noisette

[andrewvc]

Thanks for the submission! We’re happy to hear about your new plugin for gzipped log files. We are, however, redesigning the plugin discovery and on-boarding process. I hope you don’t mind, but we’re going to be quiet on this for a little while as we design that process. We’ll make sure to respond in this issue once we have a new process in place however!

[guyboertje]

@PascalNoisette As of May 1, 2018 the Logstash file input plugin has the gz file read feature rolled into version 4.1.1. v4.1.2 is a bug fix version released since 4.1.1. Give it a try as it additionally supports a "read" mode with correct "end of content" handling and supports "identity" (path) based multiline.