andrewvc commented 6 years ago

This tracks the phases of the Native support for Java plugins Project.

Experimental phase:

[x] Merge Initial Java API Branch: https://github.com/elastic/logstash/pull/10216
- [x] Java plugins fully functional in Java execution engine
[x] Implement PoCs for stdin, line codec, and stdout
[x] Provide templates for packaging and deploying Java plugins as gems
- [x] Java filter template
- [x] Java input template
- [x] Java output template

Beta phase (https://github.com/elastic/logstash/pull/10232):

[x] Plugin config validation
[x] Java codecs
- [x] API consistency between encode and decode methods?
[x] Support flush method for Java filters
[x] Create Logstash API package with interfaces ('logstash-api') under co.elastic.logstash.api
- [x] Extract Java interface from Event class
- [x] Ensure that implementation is dependent on API interfaces, not concrete classes

GA (https://github.com/elastic/logstash/pull/10620):

[x] Developer guide for Java plugins
[x] Gradle task wrappers
[x] Implement classloader isolation (beta): https://github.com/elastic/logstash/issues/9521
[x] Support for publishing Java plugins packaged as gems via Jarvis

Beyond GA?

[ ] Implement classloader isolation (GA): https://github.com/elastic/logstash/issues/9521
[ ] Port or implement more complex filters: Beats Input, ES Filter, ES Output

zivziv77 commented 5 years ago

Hi, Sorry for the lake of knowledge. I need to write a Java input and output for logstash. I wonder if writing it directly with the new java plugin api.

I will need it to production in 4-5 months.

Where can I find documentation about this new java plugin?

When it should be in production ready stage?

Thanks a lot!

In addition I would to contribute from my knowledge to convert some of the plugin from ruby to java.

Thanks

Ziv

danhermann commented 5 years ago

@zivziv77, the current plan for this work is to release the Java Plugin API in a series of phases. The first phase is the experimental phase in which we'll solicit feedback on its basic functionality. After that, it would progress to the beta phase where we'd evaluate its stability and then it would be released as GA. I would hope that we would have something within your target timeframe, but I cannot commit to any specific release date.

Right now, that work is happening in the java-api branch of the Logstash repository. You can see the current state of the Java Plugin API in the org.logstash.plugins.api package here.

It may also be worth mentioning that we do already have a number of hybrid Ruby/Java plugins in which the bulk of the code is Java using Ruby shims to interface with the Ruby plugin API in Logstash. The TCP, date, and DLQ plugins are examples of hybrid Ruby/Java plugins if you'd like some examples of how those work.

danhermann commented 5 years ago

@zivziv77, I should also add that we'd be happy to have any contributions you might make in converting plugins from Ruby to Java!

zivziv77 commented 5 years ago

Hi

Thanks for your detailed answer. The main plugin that I need is Input Redis with cluster support with one of the java Redis client. I thought writing it myself and give it to you also.

Where can I find documentation about how the java api works?

I just think it doesn’t make sense writing hybrid plugin java and ruby if there’s going to be support for full java plugin without ruby.

Thanks

danhermann commented 5 years ago

@zivziv77, we don't have docs for the Java API, yet. It's pretty simple, so perhaps a brief explanation will provide what you need. (Note that these APIs are subject to change, though it is unlikely that they will change significantly.)

Your Redis input will need to implement the Input interface. Within the start method, you should create a loop that retrieves messages from Redis and writes them to the supplied QueueWriter instance as a Map<String, Object> that will be later converted to an Event within the Logstash event processing pipeline. In your stop method, you should disconnect from Redis and perform whatever cleanup actions might be required. You can see a very simple example of this basic pattern in the StreamInput class in that same file.

You will also need to provide a constructor that takes two parameters of type Configuration and Context. The Configuration instance should provide your input with configuration values such as the connection parameters for your Redis cluster. You probably won't need to do anything with the provided Context instance.

Let me know if that helps or if you have additional questions.

zivziv77 commented 5 years ago

Thank you! That sounds pretty straight forward. One question I didn’t notice where should I register my plugin to be loaded ? How can I work with logstash on the java api nad mark to load for example Redis-input from java and not from ruby?

danhermann commented 5 years ago

Oh, right. You'll need to add the @LogstashPlugin(name = "myRedisPluginName") annotation to your class. If you then add input { myRedisPluginName {} } (or whatever you call it) to your Logstash config, it will run your Java plugin. Note that you will need to do this with Logstash built from the java-api branch as that code has not yet been merged into the master branch.

nielsbasjes commented 5 years ago

I wanted to try to create a filter for two of my (Java) projects (Yauaa and Logparser) so I decided to start by simply trying to build the java filter example. Apparently the current documentation (yes I know, experimental) assumes more inside knowledge than I currently have. So the first attempt failed miserably:

* What went wrong:
A problem occurred evaluating root project 'logstash-filter-java_filter_example'.
> Could not get unknown property 'LOGSTASH_CORE_PATH' for object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

So I download logstash 6.6.0 and hardcoded the path into the build.gradle file. Now the build runs a bit better but fails with lots of errors indicating it cannot find anything it needs

error: package co.elastic.logstash.api does not exist
error: cannot find symbol
import org.logstash.Event;

It would really help me if you can point me a bit more into the right direction (preferably by updating the readme in the example project). Note that I do not have any experience in Ruby and/or Gradle yet.

Thanks.

danhermann commented 5 years ago

@nielsbasjes, thank you for experimenting with the Java plugin API. You have noted some deficiencies in the existing documentation which have also been reported here: https://github.com/logstash-plugins/logstash-filter-java_filter_example/issues/1

I am working on improving that documentation. Let me know if my explanation in that issue doesn't get you back up and running.

nielsbasjes commented 5 years ago

Ok, So as a first step I tried to simply "get the example running". Important note: I'm pretty experienced in Java, my Ruby/Gradle skills are close to none. In my mind this is exactly the group you are trying to enable with this Java api. This is as accurately as I could the description what I have tried.

In an empty working directory:

First get a really clean logstash

git clone --branch 6.6 --single-branch https://github.com/elastic/logstash.git
cd  logstash
./gradlew assemble
rake bootstrap
cd ..

Then get the filter example

git clone https://github.com/logstash-plugins/logstash-filter-java_filter_example
cd logstash-filter-java_filter_example
echo LOGSTASH_CORE_PATH=../logstash/logstash-core/ > gradle.properties

I had to create this (documented but missing) file: lib/logstash-filter-java_filter_example_jars.rb

require 'jar_dependencies'
require_jar('org.logstash.javaapi', 'logstash-filter-java_filter_example', '0.0.1')

Then build the filter gem

./gradlew vendor
gem build *.gemspec
cd ..

Install the filter

./logstash/bin/logstash-plugin install ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem

Validating ./logstash-filter-java_filter_example/logstash-filter-java_filter_example-0.0.1.gem Installing logstash-filter-java_filter_example Installation successful

Create test input file

echo "Hello World" > /tmp/helloworld.txt

Test config file run-test.conf

input {
  file {
    path => "/tmp/helloworld.txt"
    start_position => "beginning"
  }
}

filter {
  java_filter_example {
    source => "message"
  }
}

output {
  stdout {}
}

When I run this using

./logstash/bin/logstash -f run-test.conf

I get

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties [2019-02-09T10:45:46,337][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/queue"} [2019-02-09T10:45:46,343][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/dead_letter_queue"} [2019-02-09T10:45:46,618][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2019-02-09T10:45:46,627][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"} [2019-02-09T10:45:46,653][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"af5f5856-a671-49b8-bc88-de53df75ef7e", :path=>"/home/nbasjes/workspace/LogStash/logstash/data/uuid"} [2019-02-09T10:45:51,414][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"java.lang.reflect.InvocationTargetException", :backtrace=>["org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:284)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(PluginFactoryExt.java:181)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$plugin$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:71)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:428)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:355)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_EVAL(Interpreter.java:122)", "org.jruby.ir.interpreter.Interpreter.evalCommon(Interpreter.java:176)", "org.jruby.ir.interpreter.Interpreter.evalWithBinding(Interpreter.java:200)", "org.jruby.RubyKernel.evalCommon(RubyKernel.java:1027)", "org.jruby.RubyKernel.eval19(RubyKernel.java:994)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval19.call(RubyKernel$INVOKER$s$0$3$eval19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:983)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:974)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$initialize$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)", "org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:79)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:83)", "org.jruby.RubyClass.newInstance(RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$block$execute$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call19(RubyProc.java:273)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$exclusive$1(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)", "org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:80)", "org.jruby.runtime.Block.yieldSpecific(Block.java:134)", "org.jruby.ext.thread.Mutex.synchronize(Mutex.java:148)", "org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:498)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:94)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$method$exclusive$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "home.nbasjes.workspace.LogStash.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]} [2019-02-09T10:45:51,467][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:103:in create'", "org/logstash/execution/ConvergeResultExt.java:34:inadd'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:340:in `block in converge_state'"]} [2019-02-09T10:45:51,531][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

danhermann commented 5 years ago

@nielsbasjes, thank you for trying out the Java plugin API and leaving feedback on your experience. The problem you encountered when attempting to run Logstash with the example filter is that you were not running Logstash with the Java execution engine (with the --java-execution command-line flag) which is required for Java plugins. That requirement is not well-documented and I will fix that.

The rest of the steps you took look good. In the beta version of the Java plugin API that will be released with Logstash 6.7.0, many of the manual steps and all of the boilerplate Ruby source files will be automated.

nielsbasjes commented 5 years ago

Yes, that does the trick!!

./logstash/bin/logstash --java-execution -f run-test.conf

Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties [2019-02-11T21:03:04,121][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2019-02-11T21:03:04,145][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"} [2019-02-11T21:03:08,826][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#"} [2019-02-11T21:03:09,069][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/home/nbasjes/workspace/LogStash/logstash/data/plugins/inputs/file/.sincedb_e2e76eb1ecc8816fd37224075296c425", :path=>["/tmp/helloworld.txt"]} [2019-02-11T21:03:09,097][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"} [2019-02-11T21:03:09,161][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections [2019-02-11T21:03:09,164][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2019-02-11T21:03:09,461][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} { "@timestamp" => 2019-02-11T20:03:09.692Z, "path" => "/tmp/helloworld.txt", "host" => "lw7-06766", "@version" => "1", "message" => "dlroW olleH" }

nielsbasjes commented 5 years ago

FYI: I have been playing around with the example filter and now I have it working being built fully by Maven https://github.com/nielsbasjes/logstash-filter-java_filter_example/tree/BuildUsingMaven

nielsbasjes commented 5 years ago

I wanted to create a first 'real' filter and ran into something I do not understand. So if you guys can help I really appreciate it.

My starting point (This tag in my git repo) works on my machine. I can do mvn clean package and I then do the steps mentioned before to install and run this filter.

using this filter config

filter {
  reverse_string {
    source => "message"
  }
}

The output I get on a file containing "Niels Basjes" is

{
      "@version" => "1",
    "@timestamp" => 2019-02-13T09:05:40.911Z,
          "host" => "niels-laptop",
       "message" => "sejsaB sleiN",
          "path" => "/tmp/helloworld.txt"
}

which is what I expected.

Now I change this code in only 1 single aspect: The javacode no longer lives under org.example but under nl.example instead.

That is the only change! (Tag in my git repo)

When I repeat the same steps I now get this output:

$ date; ./reload.sh ; date ; ./run.sh wo 13 feb 2019 10:10:15 CET Successfully removed logstash-filter-reverse_string Validating /home/nbasjes/workspace/Prive/logstash/logstash-filter-java_filter_example/target/logstash-filter-reverse_string-0.0.1.gem Installing logstash-filter-reverse_string Installation successful wo 13 feb 2019 10:10:50 CET Sending Logstash logs to /home/nbasjes/workspace/LogStash/logstash/logs which is now configured via log4j2.properties [2019-02-13T10:11:05,597][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2019-02-13T10:11:05,611][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.1"} [2019-02-13T10:11:09,685][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"filter", :name=>"reverse_string", :path=>"logstash/filters/reverse_string", :error_message=>"undefined local variable or method nl' for LogStash::Filters::ReverseString:Class", :error_class=>NameError, :error_backtrace=>["org/jruby/RubyBasicObject.java:1657:inmethod_missing'", "/home/nbasjes/workspace/LogStash/logstash/vendor/local_gems/30a7a257/logstash-filter-reverse_string-0.0.1/lib/logstash/filters/reverse_string.rb:13:in javaClass'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:271:inadd_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:216:in lazy_add'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:189:inlegacy_lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in block in lookup'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:152:in lookup'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:206:inlookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:222:inplugin'", "org/logstash/plugins/PluginFactoryExt.java:149:in buildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:ininitialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:inexecute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]} [2019-02-13T10:11:09,704][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any filter plugin named 'reverse_string'. Are you sure this is correct? Trying to load the reverse_string filter plugin resulted in this error: Problems loading the requested plugin named reverse_string of type filter. Error: NameError undefined local variable or methodnl' for LogStash::Filters::ReverseString:Class", :backtrace=>["/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugins/registry.rb:211:in lookup_pipeline_plugin'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/plugin.rb:137:inlookup'", "org/logstash/plugins/PluginFactoryExt.java:222:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:149:inbuildFilter'", "org/logstash/execution/JavaBasePipelineExt.java:47:in initialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:ininitialize'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:37:in execute'", "/home/nbasjes/workspace/LogStash/logstash/logstash-core/lib/logstash/agent.rb:327:inblock in converge_state'"]} [2019-02-13T10:11:10,002][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

So to me the underlying error seems to be this:

:error_message=>"undefined local variable or method `nl' for LogStash::Filters::ReverseString:Class",

The place where I find this is in the just about the only piece of Ruby code:

$ cat ./bad/lib/logstash/filters/reverse_string.rb


#
# AUTOGENERATED DURING BUILD. DO NOT EDIT.
#
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "logstash-filter-reverse_string_jars"
require "java"

class LogStash::Filters::ReverseString < LogStash::Filters::Base config_name "reverse_string"

def self.javaClass() nl.example.logstash.ReverseString.java_class; end end

Where the working variant only has one line different: The one near the end

> $ diff ./{bad,good}/lib/logstash/filters/reverse_string.rb

13c13 < def self.javaClass() nl.example.logstash.ReverseString.java_class; end

def self.javaClass() org.example.logstash.ReverseString.java_class; end

nielsbasjes commented 5 years ago

For now I'm using an ugly workaround. First working third-party filter? https://yauaa.basjes.nl/UDF-Logstash.html

danhermann commented 5 years ago

@nielsbasjes, congrats on the first working third-party Java filter!

As for the change to the package of the filter, that does have to be updated in some of the Ruby source files as you noted above. I wasn't sure if that resolved your problem or if you needed something else.

Also, in the beta release of this API which will land in Logstash 6.7.0, all the Ruby source files will be auto-generated so no manual editing of them will be required.

nielsbasjes commented 5 years ago

The example filter clearly states (as expected):

constructors should validate configuration options

The question I have: What is the proper way of failing in case of bad configuration so that a descriptive error message is passed on to the user?

I have tried throwing an IllegalArgumentException but that wasn't shown in the console.

danhermann commented 5 years ago

@nielsbasjes, I typically throw an IllegalArgumentException or IllegalStateException as you did. In the beta release of Java plugin support in the upcoming Logstash 6.7 release, that exception is displayed in the console as you would expect.

nielsbasjes commented 5 years ago

Thanks. Just out of curiosity: What kind of timeline do you guys expect to have this feature released?

danhermann commented 5 years ago

The beta release of the Java plugin API will be in the 6.7 release (and is currently committed in the 6.7 and master branches of this repo). We cannot commit to a specific date for the GA release though it is currently targeted for about two months after that.

ylasri commented 5 years ago

I tried to compile a new codec plugin, but failed getting the execution right Could you help on this ?

[2019-04-11T14:23:35,906][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"asc_codec", :path=>"logstash/codecs/asc_codec", :error_message=>"\n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with lock_jars command\n\nno such file to load -- org/logstash/javaapi/asn-plugin/0.2.0/asn-plugin-0.2.0 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:356:in do_require'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:265:inblock in require_jar'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:307:in require_jar_with_block'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:264:inrequire_jar'", "C:/es7/logstash/lib/bootstrap/patches/jar_dependencies.rb:6:in require_jar'", "C:/es7/logstash/lib/logstash-codec-asc_codec_jars.rb:2:in

'", "org/jruby/RubyKernel.java:984:in require'", "C:/es7/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:inrequire'", "C:/es7/logstash/logstash-core/lib/logstash/codecs/asc_codec.rb:1:in <main>'", "org/jruby/RubyKernel.java:984:inrequire'", "C:/es7/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in require'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:181:inlegacy_lookup'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:156:in block in lookup'", "org/jruby/ext/thread/Mutex.java:165:insynchronize'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:152:in lookup'", "C:/es7/logstash/logstash-core/lib/logstash/plugins/registry.rb:206:inlookup_pipeline_plugin'", "C:/es7/logstash/logstash-core/lib/logstash/plugin.rb:137:in lookup'", "org/logstash/plugins/PluginFactoryExt.java:200:inplugin'", "org/logstash/plugins/PluginFactoryExt.java:308:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:117:inbuildInput'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "C:/es7/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:ininitialize'", "C:/es7/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "C:/es7/logstash/logstash-core/lib/logstash/agent.rb:325:inblock in converge_state'"]}

Example

encoding: utf-8

require "logstash/codecs/base" require "logstash/namespace" require "logstash-codec-asc_codec_jars" require "java"

class LogStash::Codecs::AsnCodec < LogStash::Codecs::Base config_name "asc_codec"

def self.javaClass() org.logstash.javaapi.AsnCodec.java_class; end end

require 'jar_dependencies' require_jar('org.logstash.javaapi', 'asn-plugin', '0.2.0')

jsvd commented 5 years ago

@synapticiel can you please move this comment to a new issue? there is no need to add debugging discussions to this one as it was created to track the progress of implementation. If necessary you can always link that new issue to this one.

nielsbasjes commented 5 years ago

The beta support for this was released with version 6.7 several weeks ago. So far the dependencies needed to build a plugin haven't been released to maven central yet.

Can you indicate when this will be done?

Right now I'm unable to automate my plugin build.

danhermann commented 5 years ago

@nielsbasjes, we would like to get to the point where we are publishing a minimal jar containing the Java plugin API to Maven, but that involves some non-trivial refactoring of dependencies in Logstash core and changes to our build and deployment processes. We will be releasing the Java plugin API as GA in the upcoming 7.2.0 release of Logstash without publishing those artifacts because we did not want to hold up the overall effort for that.

Until we are able to complete that work, plugin builds can be automated with the new gem gradle task that will perform all necessary steps to compile and package a Java plugin so long as you have copies of both the Logstash repository and your plugin's repository available locally.

nielsbasjes commented 5 years ago

Thanks for the update. Just wondering: Why don't you simply push the current "much too large" dependency to Maven Central as a part of the beta?

danhermann commented 5 years ago

Just wondering: Why don't you simply push the current "much too large" dependency to Maven Central as a part of the beta?

@nielsbasjes, that was an option but a big part of the reason we did not do that is that not all of the public classes in the logstash-core jar are part of the plugin API and are therefore not subject to the backward compatibility guarantees within major versions that we make for the plugin API itself. We did not want to give that impression by publishing a jar that contains nearly all of the Java source code for Logstash.

ghost commented 5 years ago

I want to use third-party jars in the java-plugins, I want to know if java-plugins supports this feature.Can you give me some help?

danhermann commented 5 years ago

@YSZYCF, yes, third-party jars are definitely supported. The example Java plugins use those in several places. For example, see the Apache Commons and Log4j2 dependencies in the Java filter:

https://github.com/logstash-plugins/logstash-filter-java_filter_example/blob/master/build.gradle

ghost commented 5 years ago

@danhermann OK

dkm199 commented 5 years ago

@danhermann hi.. i am working on a filter plugin in java. I chose to write some part of it in scala classes(for ease of coding) and used that code in java code. Unit tests works fine..configured gradle to compile scala first and then java code.. gem file generated has only one single jar. I was able to install the gem successfully into logstash 7.2 locally. But while testing the java filter plugin, I noticed, that java part of filter plugin code executes fine, but scala part of plugin code is not executed at all(I tracked this by setting debug fields into event in java code and scala code)... Do you know what might be going on here? any workarounds are appreciated. Thanks Java code accesses scala code(ExpressionEvaluator is a scala class) like below and it works fine, in the unit tests that I wrote. Boolean result = ExpressionEvaluator$.MODULE$.evaluateExpression(event, someString); also tried accessing scala class like below as well, it didn't help Boolean result = ExpressionEvaluator.evaluateExpression(event, someString);

dkm199 commented 5 years ago

nevermind, I figured out the issue, it needed scala library packaged into the jar as well, once that included, it worked fine.

damozhizhou commented 5 years ago

@danhermann ，hello，I used the example you supplyed in “https://www.elastic.co/guide/en/logstash/current/java-codec-plugin.html”，and success get the result by the java_stdin。However，when i set input from kafka，I got the error： here is my config：

Need your help, thank you very much~

danhermann commented 5 years ago

@damozhizhou, if you'll post your question in the Logstash forums and tag me (@danhermann) on it, I'll reply to you there.

danhermann commented 5 years ago

Closing this as Java plugins went GA in Logstash 7.2.0. Any additional work on Java plugins will be tracked in new issues.

For help with developing your own Java plugins for Logstash, please post in the Logstash forums. You may tag me (@danhermann) on your post there and I'll try to respond. If you've encountered a bug related to Java plugins, you can create a GitHub issue here.

nielsbasjes commented 5 years ago

@danhermann Congratulations this is now GA. What is the issue I can track about stripping and deploying the API library to maven?

The current process cannot be ran in a CI environment because of this missing jar.

danhermann commented 5 years ago

@nielsbasjes, could you open an issue here so we can track that effort separately?

karthikdeiva commented 3 years ago

Hi,

I built a custom JAVA Input plugin using the example JAVA plugin. I am able to install the GEM in the Logstash in my local. When I tried to install the custom plugin -the generated GEM file in my Dev server, getting the below exception,

Received fatal alert: handshake_failure

Could anyone brief on the process of installing the plugin developed in one machine in another Logstash instance.

Thanks, KarthikDeiva

nielsbasjes commented 3 years ago

Have a look at how I did it here https://www.github.com/nielsbasjes/yauaa/tree/master/udfs/logstash/logstash-filter/src/test/docker

I simply install everything in a docker image and check if the output meets my expectations.

Jesse-so-cool commented 3 years ago

Hi， I want to get the @metadata（like %{[@metadata][kafka][topic]} in my custom java output plugin，how can i do？

elastic / logstash

[META] Native support for Java plugins #9215

13c13 < def self.javaClass() nl.example.logstash.ReverseString.java_class; end

encoding: utf-8