Plugins Manager in 5.X and beyond.

[ph]

We have implemented the plugin manager in logstash in 1.5 more than 2 years ago. With the java changes in the core and our future plan I think its a good time to discuss the state of plugin in logstash and how we handle the dependencies.

I want to see this as an open discussion, package manager / dependency are complex beast but it shouldn't stop us from talking.

First lets summarize the state of the plugins handling.

• We are using Bundler/rubygems as the control of the dependencies tree.
• We are using the gem as the data format
• Rubygems is hosting our plugins
• Plugins have cross dependencies (plugin A depends on plugin B)
• Some plugin are hybrid and have dependencies on java jars.

  Known issues

• Offline installing in a secure environment is still cumbersome, it requires a logstash instance to generate a "pack" of plugin
• xpack suffer from the same issues as the offline installation.
• Validating if its really a logtash plugin is fragile and slow. (require a rest call per plugin)
• Plugins requiring jars wont necessary mean they will use the exact version they want. require_jar doesn't offer a guarantee.
• conflict between cross dependencies in plugin still happen. (see Resolver class)

  Possible changes to bundlers and gemspec

• Checking all plugins to mark the dependencies correctly (runtime vs development)
• package signing would be nice to have and could remove the need to make rest call to check if its actually a logstash plugin?
• Implement a new bundler resolver to relax contrains checks.
• Implement our plugins manager as a bundler plugin?

  Open questions

• As we move forward with more java are gem and bundler still the right tools to handle dependencies?
• It is possible to Shadow gem dependencies into a single self contains artifact? It is worth it to help with xpack/offline?
• Is writing a simple installer that takes self contains tarballs a possible solution?

I understand the question are really open ended.

This was creating following a discussion @talevy and I had.

[purbon]

another example of odd behaviour is https://github.com/elastic/logstash/issues/5580, we should try harder to narrow down the sematic of upates as might be hard to grasp somehow. what do you think?

[Paladin]

If I can toss a few "blue sky" features, as the OP of 5580:

Necessary: whatever the system grows into, it should be able to provide an explanation of why it didn't do what was expected.

Useful: The system should be able to answer the question "What needs updating" at least as well as "gem outdated" does.

Ideal: feedback on what's getting installed and what's left to install. It's the kind of nicety a good UI provides for operations that can potentially take more than a few seconds, just to reassure the user that something is, in fact, getting done. It's not vital, which is why it's down here.

As for the open questions: From what I remember about java, there is no native dependency management system for an end user (all require dev tools like ant or maven to be installed in order to be useful) so the question of "what will it take to create one?" will probably be the major player in the first open question.

As for the second open question, it should be possible to parse the .gemspec file(s -- recursion for dependencies off the dependencies) to create a first pass at the shadow. My initial thought, however, is it's probably not worth it unless it's part of a move towards writing a completely new approach to installing/updating plugins from gems without bundler. IOW, possibly worth it if the future is Gems=yes and Bundler=no, but if either Gems=no or Bundler=yes, then not worth the candle.