internal: Gemify plugin boilerplate

jordansissel commented 9 years ago

The new plugin ecosystem aims to allow anyone to create and publish their own logstash plugins.

At present, all the plugins we ship have a bunch of boilerplate in gemspecs, rake, etc. I'd like to reduce boilerplate so we can reduce maintenance costs and reduce copy-paste problems.

Things to do, based on reviewing a few plugins:

[x] rakelib/ is all boilerplate. Move it to a single library and publish as a gem.
[x] Gemfile is some boilerplate. We can move the dependencies into the gem in the above point.\
[x] Include any test harness/helpers, like spec_helper.rb from Logstash.
[x] Update all plugins to remove the boilerplate and use the library the above two points implement.
[x] Update all plugins gemspecs to put the correct dev/runtime deps

jordansissel commented 9 years ago

spec_helper.rb: removed Ruby >=1.9.2 check; it's 2014, this is safe.

jordansissel commented 9 years ago

rake vendor almost ready and now attempts to support jar dependencies, but it dies -

⓿ oh-my(~/projects/logstash-output-elasticsearch) pull/18 !1!
% rake vendor
Fetching jar dependencies for jar-dependencies
Fetching jar dependencies for logstash-output-elasticsearch
[ERROR] org.apache.maven.project.artifact.InvalidDependencyVersionException: Invalid version: [0,0] found for: Dependency: rubygems:ftw in project: rubygems:logstash-output-elasticsearch:gem:0.1.7. Reason: Range cannot have identical boundaries: [0,0] for project rubygems:logstash-output-elasticsearch:gem:0.1.7 at /home/jls/projects/logstash-output-elasticsearch/logstash-output-elasticsearch.gemspec -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/InvalidDependencyVersionException
rake aborted!
Errno::ENOENT: No such file or directory @ rb_sysopen - /home/jls/projects/logstash-output-elasticsearch/deps.lst
/home/jls/.rvm/gems/ruby-2.1.2/gems/jar-dependencies-0.1.4/lib/jar_installer.rb:69:in `read'
/home/jls/.rvm/gems/ruby-2.1.2/gems/jar-dependencies-0.1.4/lib/jar_installer.rb:69:in `load_from_maven'
/home/jls/.rvm/gems/ruby-2.1.2/gems/jar-dependencies-0.1.4/lib/jar_installer.rb:261:in `install_dependencies'
/home/jls/.rvm/gems/ruby-2.1.2/gems/jar-dependencies-0.1.4/lib/jar_installer.rb:191:in `do_install'
/home/jls/.rvm/gems/ruby-2.1.2/gems/jar-dependencies-0.1.4/lib/jar_installer.rb:161:in `vendor_jars'
/home/jls/.rvm/gems/ruby-2.1.2/gems/logstash-devutils-0.0.1/lib/logstash/devutils/rake/vendor.rake:185:in `block (3 levels) in <top (required)>'
/home/jls/.rvm/gems/ruby-2.1.2/gems/logstash-devutils-0.0.1/lib/logstash/devutils/rake/vendor.rake:183:in `each'
/home/jls/.rvm/gems/ruby-2.1.2/gems/logstash-devutils-0.0.1/lib/logstash/devutils/rake/vendor.rake:183:in `block (2 levels) in <top (required)>'
/home/jls/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `eval'
/home/jls/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => vendor => vendor:jars
(See full trace by running task with --trace)

jsvd commented 9 years ago

Can I help w/ this? There's still some code that hasn't been pushed into the https://github.com/elasticsearch/logstash-devutils/pull/1 pr, right?

ph commented 9 years ago

@jsvd want to peer on this?

jsvd commented 9 years ago

Under jruby works:

% rake vendor
Fetching jar dependencies for jar-dependencies
Fetching jar dependencies for logstash-output-elasticsearch
rake vendor  32.17s user 1.31s system 177% cpu 18.846 total
% ruby -v
jruby 1.7.16 (1.9.3p392) 2014-09-25 575b395 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_20-b26 +jit [darwin-x86_64]

I get the same error when using mri

electrical commented 9 years ago

@jsvd can you confirm that the jars are downloaded and placedin the right structure? and that the right ruby file is created with the jar files in there?

electrical commented 9 years ago

MRI obviously won't work with jar dependencies btw. so we need to make sure that the vendor:jar task only is run when we are on jruby

jsvd commented 9 years ago

% pwd
/Users/joaoduarte/projects/logstash-plugins/logstash-output-elasticsearch
% find . -name '*.jar'
./lib/com/spatial4j/spatial4j/0.4.1/spatial4j-0.4.1.jar
./lib/org/antlr/antlr-runtime/3.5/antlr-runtime-3.5.jar
./lib/org/apache/lucene/lucene-analyzers-common/4.10.2/lucene-analyzers-common-4.10.2.jar
./lib/org/apache/lucene/lucene-core/4.10.2/lucene-core-4.10.2.jar
./lib/org/apache/lucene/lucene-grouping/4.10.2/lucene-grouping-4.10.2.jar
./lib/org/apache/lucene/lucene-highlighter/4.10.2/lucene-highlighter-4.10.2.jar
./lib/org/apache/lucene/lucene-join/4.10.2/lucene-join-4.10.2.jar
./lib/org/apache/lucene/lucene-memory/4.10.2/lucene-memory-4.10.2.jar
./lib/org/apache/lucene/lucene-misc/4.10.2/lucene-misc-4.10.2.jar
./lib/org/apache/lucene/lucene-queries/4.10.2/lucene-queries-4.10.2.jar
./lib/org/apache/lucene/lucene-queryparser/4.10.2/lucene-queryparser-4.10.2.jar
./lib/org/apache/lucene/lucene-sandbox/4.10.2/lucene-sandbox-4.10.2.jar
./lib/org/apache/lucene/lucene-spatial/4.10.2/lucene-spatial-4.10.2.jar
./lib/org/apache/lucene/lucene-suggest/4.10.2/lucene-suggest-4.10.2.jar
./lib/org/elasticsearch/elasticsearch/1.4.0/elasticsearch-1.4.0.jar
./lib/org/ow2/asm/asm/4.1/asm-4.1.jar
./lib/org/ow2/asm/asm-commons/4.1/asm-commons-4.1.jar

So, the jars should probably be put into the vendor/jar dir, right?

electrical commented 9 years ago

No, they should all be in the lib dir. that's correct. There also should be a 'logstash-output-elasticsearch_jars.rb' file in the lib dir. can you check that?

jsvd commented 9 years ago

% find . -name 'logstash-output-elasticsearch_jars.rb'
./logstash-output-elasticsearch_jars.rb

electrical commented 9 years ago

Okay, looks like the process worked.

jordansissel commented 9 years ago

Still needs work:

jar installation fails if duplicate gems are installed (I had two jar-dependencies installed)

jordansissel commented 9 years ago

Forced JRuby-only now:

% rvm default do gem build *gemspec
Invalid gemspec in [logstash-devutils.gemspec]: Only JRuby is supported

Also put the same "Abort unless we're using JRuby" on every file.

jordansissel commented 9 years ago

Successfully registered gem: logstash-devutils (0.0.1-java)

jordansissel commented 9 years ago

Changes required for all plugins: https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/18#issuecomment-63520712

electrical commented 9 years ago

I think this has all been done?

suyograo commented 9 years ago

Closing this since it has been implemented via https://github.com/elasticsearch/logstash-devutils

elastic / logstash

internal: Gemify plugin boilerplate #2077