search

logstash-plugins / logstash-input-kinesis

Logstash Plugin for AWS Kinesis Input
Apache License 2.0
45 stars 57 forks source link

Exception when running on Logstash 7.9.0 #87

Closed luizgpsantos closed 4 years ago

luizgpsantos commented 4 years ago

I have the following dummy pipeline:

input {
    kinesis {
        kinesis_stream_name => "test"
    }
}

output { stdout {} }

When I try to run it on Logstash 7.9.0 it will throw the following exception:

$ ./bin/logstash --version
logstash 7.9.0

$ ./bin/logstash -f ki.conf --log.level debug
[2020-08-28T15:49:37,260][DEBUG][logstash.javapipeline    ][main] Pipeline terminated by worker error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `logger' for #<Java::OrgApacheLoggingLog4jJcl::Log4jLog:0xfd1f23>>, :backtrace=>["/Apps/7.9.0.nosync/logstash-7.9.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-kinesis-2.1.1-java/lib/logstash/inputs/kinesis.rb:78:in `register'", "/Apps/7.9.0.nosync/logstash-7.9.0/logstash-core/lib/logstash/java_pipeline.rb:226:in `block in register_plugins'", "org/jruby/RubyArray.java:1809:in `each'", "/Apps/7.9.0.nosync/logstash-7.9.0/logstash-core/lib/logstash/java_pipeline.rb:225:in `register_plugins'", "/Apps/7.9.0.nosync/logstash-7.9.0/logstash-core/lib/logstash/java_pipeline.rb:359:in `start_inputs'", "/Apps/7.9.0.nosync/logstash-7.9.0/logstash-core/lib/logstash/java_pipeline.rb:309:in `start_workers'", "/Apps/7.9.0.nosync/logstash-7.9.0/logstash-core/lib/logstash/java_pipeline.rb:183:in `run'", "/Apps/7.9.0.nosync/logstash-7.9.0/logstash-core/lib/logstash/java_pipeline.rb:134:in `block in start'"], "pipeline.sources"=>["/Apps/7.9.0.nosync/logstash-7.9.0/ki.conf"], :thread=>"#<Thread:0x56bf0e3c run>"}
[2020-08-28T15:49:37,273][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
...
[2020-08-28T15:49:39,583][INFO ][logstash.runner          ] Logstash shut down.

The same pipeline runs just fine in Logstash 7.8.0:

$ ./bin/logstash --version
logstash 7.8.0

$ ./bin/logstash -f ki.conf --log.level debug
...
[2020-08-28T15:54:40,121][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
andsel commented 4 years ago

The problem is due to the fact that the plugins code rely on a runtime library https://github.com/logstash-plugins/logstash-input-kinesis/blob/c11b93c95de3789f62f985816ed6b6b5aeca1d1f/lib/logstash/inputs/kinesis.rb#L77 that's not anymore used since with 7.9.0 ships JCL (https://github.com/elastic/logstash/pull/11853/files) and the JCL implementation doesn't have that logger property

andsel commented 4 years ago

The problem is that the plugin uses commons logging instead of the facility @logger provided by LS enviroment. This is something hackish, in LS 7.8.0 the class of kinesis_logger = org.apache.commons.logging::LogFactory.getLog("com.amazonaws.services.kinesis").logger was Java::OrgApacheCommonsLoggingImpl::Jdk14Logger while in 7.9.0 is Java::OrgApacheLoggingLog4jJcl::Log4jLog In 7.8.0 the code uses the logger property from https://github.com/apache/commons-logging/blob/23b17c3889d2efcac8509c20f94ef66b73542136/src/main/java/org/apache/commons/logging/impl/Jdk14Logger.java#L162