Logstash 6.x compatibility

magnusbaeck / logstash-filter-verifier

Apache License 2.0

192 stars 27 forks source link

Logstash 6.x compatibility #43

Closed breml closed 6 years ago

breml commented 7 years ago

For Logstash 6.x there is a breaking change announced, which makes the flags -e and -f mutually exclusive (see https://github.com/elastic/logstash/issues/6926 and https://github.com/elastic/logstash/pull/6976/files). LFV at the moment uses the current behavior (concatenation of -e and -f) to add the necessary input and output to the filter "under test" (see https://github.com/magnusbaeck/logstash-filter-verifier/blob/master/logstash/process.go#L77-L83 and https://github.com/magnusbaeck/logstash-filter-verifier/blob/master/logstash/parallel_process.go#L183-L189). This needs to be changed in order to become compatible with Logstash 6.x.

magnusbaeck commented 7 years ago

This should just be a matter of generating a file in the config file tempdir with the correct -e argument as its contents, which is backwards compatible.

Etharin commented 6 years ago

Hi Magnus, can you explain a bit more what's needed to generate the file? Or is this a code change that's needed to support 6 now it's released?

magnusbaeck commented 6 years ago

A (simple) code change is required.

Etharin commented 6 years ago

Cool, will continue to use Logstash 5 for now. Thanks.

jstoja commented 6 years ago

Hey @magnusbaeck any news on the status of this issue?

NetAppBlueDevil commented 6 years ago

Upvote on this as well now that 6.0 is GA from elastic

magnusbaeck commented 6 years ago

Most of what's needed is available from the master branch and you're welcome to try it out. AFAIK the following things are still outstanding:

The logstash.yml from the Logstash distribution is used instead of a clean copy.
The same data directory as the installed Logstash is used, resulting in Logstash refusing to start if the lock file is present.
Generate a pipeline configuration file containing the stuff we currently pass via the -e option.

The first item is just lacking a few tests, the second is quite trivial, and the third shouldn't take long either. I think I'll be able to wrap things up the coming weekend.

magnusbaeck commented 6 years ago

Pushed a few more commits now and the only remaining issue is the last bullet from the list above. I think I have working code but tests remain to be written.