search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.18k stars 3.5k forks source link

Logstash 7.17.7 or higher using Java 8 fails with NoSuchMethodError #14883

Closed davshapi closed 1 year ago

davshapi commented 1 year ago

Logstash information:

Plugins installed: logstash-codec-avro (3.3.1) logstash-codec-cef (6.2.6) logstash-codec-collectd (3.1.0) logstash-codec-dots (3.0.6) logstash-codec-edn (3.1.0) logstash-codec-edn_lines (3.1.0) logstash-codec-es_bulk (3.1.0) logstash-codec-fluent (3.4.1) logstash-codec-graphite (3.0.6) logstash-codec-json (3.1.1) logstash-codec-json_lines (3.1.0) logstash-codec-line (3.1.1) logstash-codec-msgpack (3.1.0) logstash-codec-multiline (3.1.1) logstash-codec-netflow (4.2.2) logstash-codec-netwitness (1.0.0) logstash-codec-plain (3.1.0) logstash-codec-rubydebug (3.1.0) logstash-filter-aggregate (2.10.0) logstash-filter-anonymize (3.0.6) logstash-filter-cidr (3.1.3) logstash-filter-clone (4.2.0) logstash-filter-csv (3.1.1) logstash-filter-date (3.1.15) logstash-filter-de_dot (1.0.4) logstash-filter-dissect (1.2.5) logstash-filter-dns (3.1.5) logstash-filter-drop (3.0.5) logstash-filter-elasticsearch (3.11.1) logstash-filter-fingerprint (3.3.2) logstash-filter-geoip (7.2.12) logstash-filter-grok (4.4.3) logstash-filter-http (1.2.1) logstash-filter-json (3.2.0) logstash-filter-kv (4.5.0) logstash-filter-memcached (1.1.0) logstash-filter-metrics (4.0.7) logstash-filter-multiline (3.0.4) logstash-filter-mutate (3.5.6) logstash-filter-prune (3.0.4) logstash-filter-ruby (3.1.8) logstash-filter-sleep (3.0.7) logstash-filter-split (3.1.8) logstash-filter-syslog_pri (3.1.1) logstash-filter-throttle (4.0.4) logstash-filter-translate (3.3.1) logstash-filter-truncate (1.0.5) logstash-filter-urldecode (3.0.6) logstash-filter-useragent (3.3.3) logstash-filter-uuid (3.0.5) logstash-filter-xml (4.1.3) logstash-input-azure_event_hubs (1.4.4) logstash-input-beats (6.2.6) └── logstash-input-elastic_agent (alias) logstash-input-couchdb_changes (3.1.6) logstash-input-dead_letter_queue (1.1.12) logstash-input-elasticsearch (4.12.3) logstash-input-exec (3.4.0) logstash-input-file (4.4.4) logstash-input-ganglia (3.1.4) logstash-input-gelf (3.3.2) logstash-input-generator (3.1.0) logstash-input-graphite (3.0.6) logstash-input-heartbeat (3.1.1) logstash-input-http (3.4.5) logstash-input-http_poller (5.1.0) logstash-input-imap (3.2.0) logstash-input-jms (3.2.2) logstash-input-pipe (3.1.0) logstash-input-redis (3.7.0) logstash-input-s3 (3.8.3) logstash-input-snmp (1.3.1) logstash-input-snmptrap (3.1.0) logstash-input-sqs (3.1.3) logstash-input-stdin (3.4.0) logstash-input-syslog (3.6.0) logstash-input-tcp (6.2.7) logstash-input-twitter (4.1.0) logstash-input-udp (3.5.0) logstash-input-unix (3.1.2) logstash-integration-elastic_enterprise_search (2.1.2) ├── logstash-output-elastic_app_search └── logstash-output-elastic_workplace_search logstash-integration-jdbc (5.2.6) ├── logstash-input-jdbc ├── logstash-filter-jdbc_streaming └── logstash-filter-jdbc_static logstash-integration-kafka (10.9.0) ├── logstash-input-kafka └── logstash-output-kafka logstash-integration-rabbitmq (7.3.1) ├── logstash-input-rabbitmq └── logstash-output-rabbitmq logstash-output-cloudwatch (3.0.10) logstash-output-csv (3.0.8) logstash-output-elasticsearch (11.4.1) logstash-output-email (4.1.1) logstash-output-file (4.3.0) logstash-output-graphite (3.1.6) logstash-output-http (5.2.5) logstash-output-lumberjack (3.1.9) logstash-output-nagios (3.0.6) logstash-output-null (3.0.5) logstash-output-pipe (3.0.6) logstash-output-redis (5.0.0) logstash-output-s3 (4.3.7) logstash-output-sns (4.0.8) logstash-output-sqs (6.0.0) logstash-output-stdout (3.1.4) logstash-output-tcp (6.0.3) logstash-output-udp (3.2.0) logstash-output-webhdfs (3.0.6) logstash-patterns-core (4.3.4)

JVM: openjdk version "1.8.0_362" OpenJDK Runtime Environment (Temurin)(build 1.8.0_362-b09) OpenJDK 64-Bit Server VM (Temurin)(build 25.362-b09, mixed mode)

Temurin installed through zypper using packages mirrored from https://packages.adoptium.net/artifactory/rpm/sles/12/x86_64

Using LS_JAVA_HOME defined java: /opt/openjdk-8 This is a symlink we setup on all our systems to point to the Temurin installation.

OS version: Linux hostname 4.12.14-16.120-azure #1 SMP Tue Dec 13 06:06:02 UTC 2022 (e336005) x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: Logstash fails at startup with an NoSuchMethodError (see below for full stack trace).

Steps to reproduce: This happens during startup on all our environments using this Logstash version, regardless of Logstash configuration.

I'm speculating that between 7.17.6 (which did not have this behavior) and 7.17.7, something changed in the build environment for Logstash 7.

Given this error, it looks to me like the issue is Logstash is expecting to find the method: MappedByteBuffer position(int newPosition)

as defined in Java 9 and higher, but Java 8 does not have this method. Instead MappedByteBuffer inherits the following from its parent class:

Buffer position(int newPosition)

Provide logs (if relevant): 2023-02-09T14:23:13,684][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::NoSuchMethodError", :message=>"java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;", :backtrace=>["org.logstash.ackedqueue.io.MmapPageIOV2.create(MmapPageIOV2.java:194)", "org.logstash.ackedqueue.Queue.newCheckpointedHeadpage(Queue.java:386)", "org.logstash.ackedqueue.Queue.openPages(Queue.java:209)", "org.logstash.ackedqueue.Queue.open(Queue.java:176)", "org.logstash.ackedqueue.ext.JRubyAckedQueueExt.open(JRubyAckedQueueExt.java:123)", "org.logstash.ackedqueue.ext.JRubyWrappedAckedQueueExt.initialize(JRubyWrappedAckedQueueExt.java:65)", "org.logstash.ackedqueue.QueueFactoryExt.create(QueueFactoryExt.java:62)", "org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:207)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:355)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:144)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:345)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:191)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:178)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:397)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:206)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:750)"]} warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handleJava::JavaLang::NoSuchMethodErrorforPipelineAction::Create

create at org/logstash/execution/ConvergeResultExt.java:135 add at org/logstash/execution/ConvergeResultExt.java:60 converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:405

andsel commented 1 year ago

Since JDK 1.4 a method Buffer position(int i) existed on Buffer and so inherited in ByteBuffer and MappedBytByffer With JDK 9 a new method named ByteBuffer position(int i) was added to ByteBuffer, and despite with Logstash 7.17.6 and 7.17.7 we haven't changed the the target and source settings to the compiler, it could be that this happened because we presumably we compiled 7.17.6 with a JDK8. The problem is that compiling with JDK 11, despite the source and target setting, the compiler selects the "wrong" method, which doesn't exists in JDK 8. This is explained very well in the blog post: https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/

We need to configure in Gradle the same concept of Maven's maven.compiler.release. Maybe the release option of the compiler task would help https://docs.gradle.org/current/dsl/org.gradle.api.tasks.compile.CompileOptions.html#org.gradle.api.tasks.compile.CompileOptions:release

andsel commented 1 year ago

It's reproducible downloading a JDK 8 (tested with 8.0.362) and setting the LS_JAVA_HOME

export LS_JAVA_HOME="/home/andrea/.sdkman/candidates/java/8.0.362-tem/"

Enabling the PQ, in config/logstash.yml set queue.type: persisted and run a simple pipeline:

logstash-7.17.6/bin/logstash -e "input{stdin{}} output{stdout{}}"
- vs - 
logstash-7.17.7/bin/logstash -e "input{stdin{}} output{stdout{}}"
roaksoax commented 1 year ago

I'm going to go ahead and close this issue, provided that we believe it has now been fixed and will be available in the next 7.x Logstash release (7.17.12).

Please note Lgostash 8.x doesn't support Java 8.