search

logstash-plugins / logstash-integration-jdbc

Logstash Integration Plugin for JDBC, including Logstash Input and Filter Plugins
Apache License 2.0
53 stars 54 forks source link

Loader inventory fetching records is blocking pipeline #89

Open ikus060 opened 3 years ago

ikus060 commented 3 years ago

Logstash information:

docker image bitnami/logstash:7.10.1-debian-10-r35

JVM (e.g. java -version):

N/A

OS version (uname -a if on a Unix-like system):

N/A

Description of the problem including expected versus actual behavior:

When using jdbc_static to fetch a reasonable list of records from MySQL database, the pipeline is blocked while recrods get fetched. This causes peak CPU usage as schedule intervale.

I would expect the pipeline to be blocked only during the derby database refresh. Not during the fetching.

Provide logs (if relevant):

When the following line of code get printed, the pipeline stop processing.

[2021-11-02T15:44:48,867][INFO ][logstash.filters.jdbc.readwritedatabase][main] loader inventory, fetched 54114 records in: 79.353 seconds
[2021-11-02T15:44:50,897][INFO ][logstash.filters.jdbc.readwritedatabase][main] loader inventory, saved fetched records to import file in: 2.028 seconds
[2021-11-02T15:44:53,203][INFO ][logstash.filters.jdbc.readwritedatabase][main] loader inventory, imported all fetched records in: 2.304 seconds

While attached to the logstash process using JMX. Here the relevent Thread dumps. worker0 is block by "Ruby-0-Thread-21"

"[main]>worker0" - Thread t@34
   java.lang.Thread.State: WAITING
        at java.base@11.0.9.1/jdk.internal.misc.Unsafe.park(Native Method)
        - waiting to lock <504ac9f5> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync) owned by "Ruby-0-Thread-21: :1" t@50
        at java.base@11.0.9.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
        at java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:1009)
        at java.base@11.0.9.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1324)
        at java.base@11.0.9.1/java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:738)
        at java.base@11.0.9.1/jdk.internal.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
        at java.base@11.0.9.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base@11.0.9.1/java.lang.reflect.Method.invoke(Method.java:566)
        at app//org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)
        at app//org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)
        at app//org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007a4440.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a5440.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.read_write_database.RUBY$method$fetch$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/read_write_database.rb:23)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007f6c40.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.lookup.RUBY$method$fetch$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/lookup.rb:144)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100807040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.lookup.RUBY$method$enhance$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/lookup.rb:87)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100807040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.lookup_processor.RUBY$block$enhance$1(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/lookup_processor.rb:47)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000101887440.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
        at app//org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
        at app//org.jruby.runtime.Block.yield(Block.java:184)
        at app//org.jruby.RubyArray.collect(RubyArray.java:2563)
        at app//org.jruby.RubyArray.map19(RubyArray.java:2577)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007a2840.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007c8040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007cf040.guardWithCatch(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007cfc40.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.lookup_processor.RUBY$method$enhance$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/lookup_processor.rb:47)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100813c40.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc_static.RUBY$method$filter$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc_static.rb:155)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000101aa4040.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:267)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/opt/bitnami/logstash/logstash-core/lib/logstash/filters/base.rb:159)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x000000010195e040.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:267)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/opt/bitnami/logstash/logstash-core/lib/logstash/filters/base.rb:178)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000101728440.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
        at app//org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
        at app//org.jruby.runtime.Block.yield(Block.java:184)
        at app//org.jruby.RubyArray.each(RubyArray.java:1809)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007a2840.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007c8040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007cf040.guardWithCatch(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007cfc40.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/opt/bitnami/logstash/logstash-core/lib/logstash/filters/base.rb:175)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000101d89040.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
        at app//org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:127)
        at app//org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:134)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset4.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset4.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset6.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset6.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset3.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset4.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset5.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset2.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset3.compute(Unknown Source)
        at app//org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:329)
        at app//org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(CompiledPipeline.java:323)
        at app//org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83)
        at java.base@11.0.9.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base@11.0.9.1/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base@11.0.9.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base@11.0.9.1/java.lang.reflect.Method.invoke(Method.java:566)
        at app//org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:426)
        at app//org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:293)
        at app//org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24)
        at app//org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86)
        at app//org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a4040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/opt/bitnami/logstash/logstash-core/lib/logstash/java_pipeline.rb:295)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007cc840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
        at app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
        at app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
        at app//org.jruby.runtime.Block.call(Block.java:139)
        at app//org.jruby.RubyProc.call(RubyProc.java:318)
        at app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
        at java.base@11.0.9.1/java.lang.Thread.run(Thread.java:834)

   Locked ownable synchronizers:
        - None

"Ruby-0-Thread-21: :1" - Thread t@50
   java.lang.Thread.State: RUNNABLE
        at java.base@11.0.9.1/java.net.SocketInputStream.socketRead0(Native Method)
        at java.base@11.0.9.1/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
        at java.base@11.0.9.1/java.net.SocketInputStream.read(SocketInputStream.java:168)
        at java.base@11.0.9.1/java.net.SocketInputStream.read(SocketInputStream.java:140)
        at com.mysql.cj.protocol.ReadAheadInputStream.fill(ReadAheadInputStream.java:107)
        at com.mysql.cj.protocol.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:150)
        at com.mysql.cj.protocol.ReadAheadInputStream.read(ReadAheadInputStream.java:180)
        - locked <6c1d30a0> (a com.mysql.cj.protocol.ReadAheadInputStream)
        at java.base@11.0.9.1/java.io.FilterInputStream.read(FilterInputStream.java:133)
        at com.mysql.cj.protocol.FullReadInputStream.readFully(FullReadInputStream.java:64)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:63)
        at com.mysql.cj.protocol.a.SimplePacketReader.readHeader(SimplePacketReader.java:45)
        at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:52)
        at com.mysql.cj.protocol.a.TimeTrackingPacketReader.readHeader(TimeTrackingPacketReader.java:41)
        at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:54)
        at com.mysql.cj.protocol.a.MultiPacketReader.readHeader(MultiPacketReader.java:44)
        at com.mysql.cj.protocol.a.NativeProtocol.readMessage(NativeProtocol.java:532)
        at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:702)
        at com.mysql.cj.protocol.a.NativeProtocol.sendCommand(NativeProtocol.java:641)
        at com.mysql.cj.protocol.a.NativeProtocol.sendQueryPacket(NativeProtocol.java:940)
        at com.mysql.cj.protocol.a.NativeProtocol.sendQueryString(NativeProtocol.java:886)
        at com.mysql.cj.NativeSession.execSQL(NativeSession.java:1073)
        at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1168)
        - locked <f06dd79> (a com.mysql.cj.jdbc.ConnectionImpl)
        at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
        at java.base@11.0.9.1/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base@11.0.9.1/java.lang.reflect.Method.invoke(Method.java:566)
        at app//org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:456)
        at app//org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:317)
        at app//org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:42)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007be040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010802a440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077dc40.reinvoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.adapters.jdbc.RUBY$block$execute$4(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/adapters/jdbc.rb:268)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010183a440.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
        at app//org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:66)
        at app//org.jruby.runtime.Block.yieldSpecific(Block.java:153)
        at app//org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:156)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007a2840.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a3840.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.database.logging.RUBY$method$log_connection_yield$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/database/logging.rb:38)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001073c7040.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:116)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:175)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:301)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007d5c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d6040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007edc40.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.adapters.jdbc.RUBY$block$execute$3(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/adapters/jdbc.rb:268)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010183b840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
        at app//org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
        at app//org.jruby.runtime.Block.yield(Block.java:184)
        at app//org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:499)
        at app//org.jruby.ir.targets.YieldSite.yield(YieldSite.java:114)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007d3440.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007f6840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.adapters.jdbc.RUBY$method$statement$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/adapters/jdbc.rb:704)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001073c4840.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.adapters.jdbc.RUBY$block$execute$2(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/adapters/jdbc.rb:263)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010183c040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
        at app//org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
        at app//org.jruby.runtime.Block.yield(Block.java:184)
        at app//org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:499)
        at app//org.jruby.ir.targets.YieldSite.yield(YieldSite.java:114)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007d3440.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007f6840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.connection_pool.threaded.RUBY$method$hold$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/connection_pool/threaded.rb:92)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000105a6b040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.database.connecting.RUBY$method$synchronize$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/database/connecting.rb:269)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000107447440.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.adapters.jdbc.RUBY$method$execute$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/adapters/jdbc.rb:262)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001073c6440.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:116)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:175)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:301)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007d5c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d6040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007edc40.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.dataset.actions.RUBY$method$execute$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/dataset/actions.rb:1093)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100f13440.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.adapters.jdbc.RUBY$method$fetch_rows$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/adapters/jdbc.rb:755)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x000000010aa07040.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.dataset.actions.RUBY$method$with_sql_each$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/dataset/actions.rb:946)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100949840.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.dataset.actions.RUBY$method$with_sql_first$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/dataset/actions.rb:954)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007b9c40.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a440.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.dataset.actions.RUBY$method$with_sql_single_value$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/dataset/actions.rb:962)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x000000010079c440.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:106)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:140)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:278)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077a440.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.dataset.actions.RUBY$method$single_value!$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/dataset/actions.rb:734)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.dataset.actions.RUBY$method$single_value!$0$__VARARGS__(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/dataset/actions.rb)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100e41840.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:248)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:255)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007a4440.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100a94040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.sequel_minus_5_dot_38_dot_0.lib.sequel.dataset.actions.RUBY$method$count$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/sequel-5.38.0/lib/sequel/dataset/actions.rb:110)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100903c40.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:248)
        at app//org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:255)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x00000001007a4440.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100a94040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.read_only_database.RUBY$method$count$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/read_only_database.rb:13)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010cc35840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.loader.RUBY$method$fetch$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/loader.rb:23)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007ec840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.read_write_database.RUBY$method$fill_local_table$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/read_write_database.rb:68)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010cc35840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.read_write_database.RUBY$method$repopulate_all$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/read_write_database.rb:9)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010cc35840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.repeating_load_runner.RUBY$method$repeated_load$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/repeating_load_runner.rb:9)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007ec840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.single_load_runner.RUBY$method$call$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/single_load_runner.rb:30)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_integration_minus_jdbc_minus_5_dot_0_dot_6.lib.logstash.filters.jdbc.single_load_runner.RUBY$method$call$0$__VARARGS__(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-integration-jdbc-5.0.6/lib/logstash/filters/jdbc/single_load_runner.rb)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100e28440.invokeStatic(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000100751840.invokeExact_MT(LambdaForm$MH)
        at app//org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)
        at app//org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)
        at app//org.jruby.RubyMethod.call(RubyMethod.java:131)
        at app//org.jruby.RubyMethod$INVOKER$i$call.call(RubyMethod$INVOKER$i$call.gen)
        at app//org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:174)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007be040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.RUBY$method$do_call$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:234)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007f6c40.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d7c40.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007d5840.linkToCallSite(LambdaForm$MH)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.RUBY$method$do_trigger$0(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:258)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010cc35840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010077e040.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.RUBY$block$start_work_thread$3(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:300)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010092e040.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
        at app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
        at app//org.jruby.runtime.Block.call(Block.java:143)
        at app//org.jruby.RubyProc.call(RubyProc.java:275)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x000000010081c440.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.DelegatingMethodHandle$Holder.delegate(DelegatingMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007a6440.guard(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.RUBY$block$start_work_thread$2(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:299)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x0000000109781840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
        at app//org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:66)
        at app//org.jruby.runtime.Block.yieldSpecific(Block.java:153)
        at app//org.jruby.RubyKernel.loop(RubyKernel.java:1442)
        at app//org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
        at app//org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:555)
        at app//org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:197)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$DMH/0x0000000100779c40.invokeVirtual(LambdaForm$DMH)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007cd840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.linkToCallSite(Invokers$Holder)
        at opt.bitnami.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.rufus_minus_scheduler_minus_3_dot_0_dot_9.lib.rufus.scheduler.jobs.RUBY$block$start_work_thread$1(/opt/bitnami/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/jobs.rb:289)
        at java.base@11.0.9.1/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
        at java.base@11.0.9.1/java.lang.invoke.LambdaForm$MH/0x00000001007cc840.invoke(LambdaForm$MH)
        at java.base@11.0.9.1/java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder)
        at app//org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
        at app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
        at app//org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
        at app//org.jruby.runtime.Block.call(Block.java:139)
        at app//org.jruby.RubyProc.call(RubyProc.java:318)
        at app//org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
        at java.base@11.0.9.1/java.lang.Thread.run(Thread.java:834)

   Locked ownable synchronizers:
        - locked <504ac9f5> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)

The problem seams to be related to Line 66: https://github.com/logstash-plugins/logstash-integration-jdbc/blob/master/lib/logstash/filters/jdbc/read_write_database.rb#L66

The write lock is aquired to early in the process. It should be aquired after fetching the records on Line 72

ikus060 commented 2 years ago

The following patch fixes the issue:

--- read_write_database.rb.orig 2021-12-08 20:39:59.499527593 +0000
+++ read_write_database.rb  2021-12-08 20:42:32.960977605 +0000
@@ -63,12 +63,12 @@

     def fill_local_table(loader)
       begin
-        @rwlock.writeLock().lock()
         start = Time.now.to_f
         records = loader.fetch
         records_size = records.size
         return if records_size.zero?
         logger.info("loader #{loader.id}, fetched #{records_size} records in: #{(Time.now.to_f - start).round(3)} seconds")
+        @rwlock.writeLock().lock()
         start = Time.now.to_f
         import_file = ::File.join(loader.staging_directory, loader.table.to_s)
         ::File.open(import_file, "w") do |fd|