ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 80 while PipelineAction::Create

[sasikiranvaddi]

We have two pipelines configured, "opensearch" and "logstash". During the startup of logstash process OpenSearch pipeline has been created and while converging Logstash pipeline the logstash is failing with ArrayIndexOutOfBoundsException.

The exception has occurred while processing the ThreadContext. at org.jruby.runtime.ThreadContext.popFrameReal(ThreadContext.java:570)

To recover from it, we produced a graceful restart of logstash process, after which we could see the pipelines running successfuly. [logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:opensearch, :logstash], :non_running_pipelines=>[]}

Versions:

Logstash: 8.13.0
Jruby: 9.4.5.0
Java: JDK11

[logstash.runner] Starting Logstash {'logstash.version'=>'8.13.0', 'jruby.version'=>'jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 11.0.23+9-suse-150000.3.113.1-x8664 on 11.0.23+9-suse-150000.3.113.1-x8664 +indy +jit [x86_64-linux]'}

    pipelines.yml: 
      - pipeline.id: logstash
        queue.type: persisted
        queue.max_bytes: 1024mb
        path.config: "/opt/logstash/resource/logstash.conf"
      - pipeline.id: opensearch
        queue.type: persisted
        queue.max_bytes: 1024mb
        path.config: "/opt/logstash/resource/searchengine.conf"

On a side note, there is an issue w.r.t metric, the logstash recommended to log an issue. Please let me know if it requires a separate ticket.

[org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: send_to. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
2024-07-03 20:40:38,246 pool-5-thread-1 ERROR An exception occurred processing Appender plain_console org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:945)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2241)
at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:945)
at org.jruby.RubyHash.to_s(org/jruby/RubyHash.java:1011)
at org.logstash.log.LoggerExt.error(org/logstash/log/LoggerExt.java:127)
at RUBY.update(/opt/logstash/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:53)
at RUBY.notify_to(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:102)
at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1601)
at RUBY.notify_to(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100)
at RUBY.notify_observers(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:64)
at RUBY.execute_task(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/timer_task.rb:317)
at RUBY.execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24)
at RUBY.execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19)
at RUBY.safe_execute(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/ivar.rb:169)
at RUBY.process_task(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/scheduled_task.rb:285)
at RUBY.process_tasks(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/timer_set.rb:165)
at RUBY.run(/opt/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79)

Please find the attached logs. ArrayOutOfBoundsandMetricException.txt

[robbavey]

@sasikiranvaddi Can you please share your pipeline configuration, and also debug logs - running with debug should provide us with the backtrace for the error

[sasikiranvaddi]

Hi @robbavey, The occurrence of ArrayIndexOutofBoundsException is spoardic. Please find attached pipeline configurations. logstash.conf.txt searchenginepipeline.conf.txt

[TheVastyDeep]

Similar to 16091?

[sasikiranvaddi]

Hi, Any thoughts on this issue?

[edmocosta]

Hi @sasikiranvaddi!

I couldn't reproduce this issue locally, could you please try running Logstash using the bundled JDK instead of the Suse's one? If the issue still persists with the Logstash's bundled JDK, my suggestion would be upgrading to 8.15.1 and give it a try. Regarding the metric log message, it's probably harmless but needs to be addressed. Thanks for reporting.