search

elastic / logstash-devutils

An assortment of tooling/libraries to make Logstash core and plugin development and releasing a bit easier.
Apache License 2.0
18 stars 27 forks source link

resurrect devutils with Java pipeline #79

Closed kares closed 4 years ago

kares commented 4 years ago

While the gem is used less-and-less for unit testing plugins (which is a good thing), helpers to setup a pipeline still come handy on more functional or performance tests. And since some plugins still have legacy specs around (which take a lot of effort to rewrite) the best solution is to get the helpers working using the preferred Java pipeline. Unfortunately, the new pipeline design did not have plugin testing in mind so we require a few noisy things to get it working every possible way the Ruby pipeline worked in tests.

Things I am unsure about is why devutils did not specify logstash-core gem dependency it supports. Seems like the choice has been to just leave things to be handled by logstash-core-plugin-api, now its explicit as we're dropping LS < 6.0 and its hard to make that distinction using the plugin API.

Plugins adopting the newer version might need to do some changes. These are mentioned in the changelog and include stuff such as specifying the gem 'insist' dependency (if its too much effort to get rid of the insist syntax).

All in all, I have tried these on 3-4 plugins so far, as well as adapting it for performance tests, which I am still not 100% happy about, all working after expected changes.

p.s. managed to leave a few TODOs (with explanation notes) around as I did not want to dive in even more to get it "exactly" as I anticipated.

kares commented 4 years ago

should be updated now - one last thing I am thinking about, as we move this forward, would be to incorporate forcing plugins to drop the logstash-core-plugin-api dependency by doing : 

-  spec.add_runtime_dependency "logstash-core-plugin-api", "<= 2.99", ">= 2.1.16"
+  spec.add_runtime_dependency "logstash-core", ">= 6", "< 9.0"

... since logstash-core-plugin-api has a fixed (concrete version) logstash-core dependency the plugin would need to remove it -> might be getting ahead of myself as I am not sure about consensus here :notebook:

kares commented 4 years ago

you're right, was a bit eager to post that here - gained the impression that plugin-api is going away. ... hasn't been released for long + by examining its contents I did not find any actual code :open_mouth:

that gem has been a source of confusion but in theory I understood what it brings and why it exists. just somehow thought it died or is going to soon ... thanks for explaining - appreciated very much!

anyhow, I tested this manually on a few plugins - works all the way back to 6.3 LS 6.2 does not work since org.logstash.execution.QueueReadClient was introduced in 6.3.