yaauie
commented
1 year ago Please note, this can either not be the same as #14605 or the backport #14752 did not address this correctly . This fails with a different stack trace and still occurs in 7.17.7
The referenced 7.x backport is tagged with 7.17.8, indicating that it is not expected to have been available to the 7.17.7 reported here. Additionally, the stack-trace posted on this issue aligns with the stack-trace on #14599 which is where the issue was first reported. I would advise upgrading to 7.17.8 to consume the fix.
An in-place remediation for <= 7.17.7 would be to backup the DLQ directory, delete any .log files in the DLQ's data directory that are exactly one byte in size, and then restart the associated pipeline.
ash-darin
andsel
Plugins installed
JVM
provided by container
"Using bundled JDK: /usr/share/logstash/jdk""jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.16+8 on 11.0.16+8 +indy +jit [linux-x86_64]"OS version
Linux testmachine 4.19.0-23-amd64 #1 SMP Debian 4.19.269-1 (2022-12-20) x86_64 GNU/LinuxDescription of the problem including expected versus actual behavior
Logstash is unable to start throwing
java.lang.IllegalArgumentException: newPosition < 0: (-1 < 0)logstash fails at startup with the following stack-trace. Please note, this can either not be the same as #14605 or the backport #14752 did not address this correctly . This fails with a different stack trace and still occurs in 7.17.7
Steps to reproduce
Take the files from #14605 (attached ) with this config:
dlq_sincedb_log.zip
copy the sincedb (with a preceding dot) file and the 1 byte files to /usr/share/logstash/data/dead_letter_queue/ (from the viewpoint of the container)
Provide logs (if relevant)
[2023-02-27T16:26:43,659][ERROR][logstash.javapipeline ][dead_letter_queue_valve] Pipeline error {:pipeline_id=>"dead_letter_queue_valve", :exception=>java.lang.IllegalArgumentException: newPosition < 0: (-1 < 0), :backtrace=>["java.nio.Buffer.createPositionException(java/nio/Buffer.java:318)", "java.nio.Buffer.position(java/nio/Buffer.java:293)", "java.nio.ByteBuffer.position(java/nio/ByteBuffer.java:1094)", "org.logstash.common.io.RecordIOReader.consumeBlock(org/logstash/common/io/RecordIOReader.java:184)", "org.logstash.common.io.RecordIOReader.consumeToStartOfEvent(org/logstash/common/io/RecordIOReader.java:238)", "org.logstash.common.io.RecordIOReader.readEvent(org/logstash/common/io/RecordIOReader.java:282)", "org.logstash.common.io.RecordIOReader.seekToNextEventPosition(org/logstash/common/io/RecordIOReader.java:156)", "org.logstash.common.io.DeadLetterQueueReader.seekToNextEvent(org/logstash/common/io/DeadLetterQueueReader.java:87)", "org.logstash.input.DeadLetterQueueInputPlugin.setInitialReaderState(org/logstash/input/DeadLetterQueueInputPlugin.java:95)", "org.logstash.input.DeadLetterQueueInputPlugin.lazyInitQueueReader(org/logstash/input/DeadLetterQueueInputPlugin.java:74)", "org.logstash.input.DeadLetterQueueInputPlugin.register(org/logstash/input/DeadLetterQueueInputPlugin.java:81)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_dead_letter_queue_minus_1_dot_1_dot_12.lib.logstash.inputs.dead_letter_queue.register(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-dead_letter_queue-1.1.12/lib/logstash/inputs/dead_letter_queue.rb:57)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1821)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_inputs(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:391)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:316)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:829)"], "pipeline.sources"=>["/usr/share/logstash/pipelines/dead_letter_queue_valve/active/100-input-dlq.config", "/usr/share/logstash/pipelines/dead_letter_queue_valve/active/101-filter-drop-events-older-than-35d.config", "/usr/share/logstash/pipelines/dead_letter_queue_valve/active/102-test-case.config", "/usr/share/logstash/pipelines/dead_letter_queue_valve/active/200-filter-extract-fields.config", "/usr/share/logstash/pipelines/dead_letter_queue_valve/active/201-filter-add-dlq-event-hash-for-jdbc-common-out.config", "/usr/share/logstash/pipelines/dead_letter_queue_valve/active/300-output-flatfile.config", "/usr/share/logstash/pipelines/dead_letter_queue_valve/active/301-output-elastic-siem.config"], :thread=>"#<Thread:0x249baad4 run>"}