search

cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

exception kills the whole pod/process #42

Closed zeph closed 4 years ago

zeph commented 4 years ago 
[2020-07-20T16:20:45,099][WARN ][logstash.inputs.s3snssqs ][main][1d73eb90caca850c22fe3cb27b23999564951e6ca4efcc2bdee6aa87ee7a837d] [Worker 1d73eb90caca850c22fe3cb27b23999564951e6ca4efcc2bdee6aa87ee7a837d/0] Incomplete message in read_file. We´ll throw skip_delete. {:filename=>"/tmp/tmplogstash20200720-1-yb8qzk/835487002321_elasticloadbalancing_eu-west-1_a7686967ea86745ff9fdbdd6be759eaf_20200712T1025Z_34.249.124.207_7u03rgv6.log"}
warning: thread "Worker 1d73eb90caca850c22fe3cb27b23999564951e6ca4efcc2bdee6aa87ee7a837d/0" terminated with exception (report_on_exception is true):
ThreadError: killed thread
             wakeup at org/jruby/RubyThread.java:1278
                run at org/jruby/RubyThread.java:1436
                run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:125
     yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413
               each at org/jruby/RubyArray.java:1809
     yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412
   process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405
              catch at org/jruby/RubyKernel.java:1189
   process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404
               poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336
               loop at org/jruby/RubyKernel.java:1442
               poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331
              catch at org/jruby/RubyKernel.java:1189
               poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330
                run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:83
   run_with_backoff at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:178
                run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:81
  run_worker_thread at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:334
warning: thread "Ruby-0-Thread-19: :1" terminated with exception (report_on_exception is true):
ThreadError: killed thread
             wakeup at org/jruby/RubyThread.java:1278
                run at org/jruby/RubyThread.java:1436
                run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:125
     yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413
               each at org/jruby/RubyArray.java:1809
     yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412
   process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405
              catch at org/jruby/RubyKernel.java:1189
   process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404
               poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336
               loop at org/jruby/RubyKernel.java:1442
               poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331
              catch at org/jruby/RubyKernel.java:1189
               poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330
                run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:83
   run_with_backoff at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:178
                run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:81
  run_worker_thread at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:334
[2020-07-20T16:20:45,464][ERROR][logstash.javapipeline    ][main][1d73eb90caca850c22fe3cb27b23999564951e6ca4efcc2bdee6aa87ee7a837d] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::S3SNSSQS codec=><LogStash::Codecs::Plain id=>"plain_a6577306-c5af-438d-b86d-6c9cde5037cb", enable_metric=>true, charset=>"UTF-8">, max_processing_time=>600, s3_default_options=>{"endpoint_discovery"=>"true"}, from_sns=>false, id=>"1d73eb90caca850c22fe3cb27b23999564951e6ca4efcc2bdee6aa87ee7a837d", region=>"eu-west-1", type=>"elb-logs", queue=>"elb-logs-queue", sqs_delete_on_failure=>true, enable_metric=>true, threads=>1, role_session_name=>"logstash", s3_role_session_name=>"logstash", delete_on_success=>false, include_object_properties=>[:last_modified, :content_type, :metadata], sqs_skip_delete=>false, visibility_timeout=>120, temporary_directory=>"/tmp/logstash", consumer_threads=>1>
  Error: killed thread
  Exception: ThreadError
  Stack: org/jruby/RubyThread.java:1278:in `wakeup'
org/jruby/RubyThread.java:1436:in `run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:125:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in `block in yield_messages'
org/jruby/RubyArray.java:1809:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in `yield_messages'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:in `block in process_messages'
org/jruby/RubyKernel.java:1189:in `catch'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:in `process_messages'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in `block in poll'
org/jruby/RubyKernel.java:1442:in `loop'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll'
org/jruby/RubyKernel.java:1189:in `catch'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:83:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:178:in `run_with_backoff'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:81:in `run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:334:in `block in run_worker_thread'
[2020-07-20T16:20:45,549][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ThreadError: killed thread>, :backtrace=>["org/jruby/RubyThread.java:1278:in `wakeup'", "org/jruby/RubyThread.java:1436:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:125:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in `block in yield_messages'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in `yield_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:in `block in process_messages'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:in `process_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in `block in poll'", "org/jruby/RubyKernel.java:1442:in `loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:83:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:178:in `run_with_backoff'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/sqs/poller.rb:81:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.1/lib/logstash/inputs/s3snssqs.rb:334:in `block in run_worker_thread'"]}
[2020-07-20T16:20:45,666][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
zeph commented 4 years ago

I believe it is related to the reach of max_processing_time when there are still data in the "message" / parsed-file

zeph commented 4 years ago

ok, attempting to test 2.1.0 that seems people are using with "no" issues... https://github.com/zeph/logstash-s3-sqs/commit/d7dbb37e6d68d4e250c17788fca40f53e9a17d75

zeph commented 4 years ago

not much better... 

logging-logstash-0 logstash Exception in thread "Worker 92b4c7a7186ec0db32579714425fcfa1fc3e61aea6a54a39178d633ab3e50554/0_1_2_3_4_5_6_7_8_9" java.lang.NullPointerException
logging-logstash-0 logstash     at org.jruby.internal.runtime.ThreadService.getMainThread(ThreadService.java:233)
logging-logstash-0 logstash     at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1822)
logging-logstash-0 logstash     at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
logging-logstash-0 logstash     at java.base/java.lang.Thread.run(Thread.java:834)
logging-logstash-0 logstash warning: thread "Worker 92b4c7a7186ec0db32579714425fcfa1fc3e61aea6a54a39178d633ab3e50554/0_1_2_3_4_5_6_7_8_9_10" terminated with exception (report_on_exception is true):
logging-logstash-0 logstash java.lang.NullPointerException
logging-logstash-0 logstash     at org.jruby.internal.runtime.ThreadService.getMainThread(ThreadService.java:233)
logging-logstash-0 logstash     at org.jruby.RubyThread.kill(RubyThread.java:1633)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_s3_minus_sns_minus_sqs_minus_2_dot_1_dot_0.lib.logstash.inputs.sqs.poller.RUBY$block$run$3(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:128)
logging-logstash-0 logstash     at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
logging-logstash-0 logstash     at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
logging-logstash-0 logstash     at org.jruby.runtime.Block.yieldValues(Block.java:200)
logging-logstash-0 logstash     at org.jruby.ir.targets.YieldSite.yieldValues(YieldSite.java:166)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.aws_minus_sdk_minus_resources_minus_2_dot_11_dot_501.lib.aws_minus_sdk_minus_resources.services.sqs.queue_poller.RUBY$block$yield_messages$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413)
logging-logstash-0 logstash     at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
logging-logstash-0 logstash     at org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
logging-logstash-0 logstash     at org.jruby.runtime.Block.yield(Block.java:184)
logging-logstash-0 logstash     at org.jruby.RubyArray.each(RubyArray.java:1809)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.aws_minus_sdk_minus_resources_minus_2_dot_11_dot_501.lib.aws_minus_sdk_minus_resources.services.sqs.queue_poller.RUBY$method$yield_messages$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.aws_minus_sdk_minus_resources_minus_2_dot_11_dot_501.lib.aws_minus_sdk_minus_resources.services.sqs.queue_poller.RUBY$block$process_messages$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405)
logging-logstash-0 logstash     at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
logging-logstash-0 logstash     at org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
logging-logstash-0 logstash     at org.jruby.runtime.Block.yield(Block.java:184)
logging-logstash-0 logstash     at org.jruby.exceptions.CatchThrow.enter(CatchThrow.java:32)
logging-logstash-0 logstash     at org.jruby.RubyKernel.rbCatch(RubyKernel.java:1189)
logging-logstash-0 logstash     at org.jruby.RubyKernel$INVOKER$s$rbCatch.call(RubyKernel$INVOKER$s$rbCatch.gen)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.aws_minus_sdk_minus_resources_minus_2_dot_11_dot_501.lib.aws_minus_sdk_minus_resources.services.sqs.queue_poller.RUBY$method$process_messages$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.aws_minus_sdk_minus_resources_minus_2_dot_11_dot_501.lib.aws_minus_sdk_minus_resources.services.sqs.queue_poller.RUBY$block$poll$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336)
logging-logstash-0 logstash     at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
logging-logstash-0 logstash     at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:66)
logging-logstash-0 logstash     at org.jruby.runtime.Block.yieldSpecific(Block.java:153)
logging-logstash-0 logstash     at org.jruby.RubyKernel.loop(RubyKernel.java:1442)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.aws_minus_sdk_minus_resources_minus_2_dot_11_dot_501.lib.aws_minus_sdk_minus_resources.services.sqs.queue_poller.RUBY$block$poll$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331)
logging-logstash-0 logstash     at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
logging-logstash-0 logstash     at org.jruby.runtime.BlockBody.yield(BlockBody.java:106)
logging-logstash-0 logstash     at org.jruby.runtime.Block.yield(Block.java:184)
logging-logstash-0 logstash     at org.jruby.exceptions.CatchThrow.enter(CatchThrow.java:32)
logging-logstash-0 logstash     at org.jruby.RubyKernel.rbCatch(RubyKernel.java:1189)
logging-logstash-0 logstash     at org.jruby.RubyKernel$INVOKER$s$rbCatch.call(RubyKernel$INVOKER$s$rbCatch.gen)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.aws_minus_sdk_minus_resources_minus_2_dot_11_dot_501.lib.aws_minus_sdk_minus_resources.services.sqs.queue_poller.RUBY$method$poll$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_s3_minus_sns_minus_sqs_minus_2_dot_1_dot_0.lib.logstash.inputs.sqs.poller.RUBY$block$run$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:83)
logging-logstash-0 logstash     at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:148)
logging-logstash-0 logstash     at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:66)
logging-logstash-0 logstash     at org.jruby.runtime.Block.yieldSpecific(Block.java:153)
logging-logstash-0 logstash     at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:156)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_s3_minus_sns_minus_sqs_minus_2_dot_1_dot_0.lib.logstash.inputs.sqs.poller.RUBY$method$run_with_backoff$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:178)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_s3_minus_sns_minus_sqs_minus_2_dot_1_dot_0.lib.logstash.inputs.sqs.poller.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:81)
logging-logstash-0 logstash     at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_s3_minus_sns_minus_sqs_minus_2_dot_1_dot_0.lib.logstash.inputs.s3snssqs.RUBY$block$run_worker_thread$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/s3snssqs.rb:329)
logging-logstash-0 logstash     at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
logging-logstash-0 logstash     at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
logging-logstash-0 logstash     at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
logging-logstash-0 logstash     at org.jruby.runtime.Block.call(Block.java:139)
logging-logstash-0 logstash     at org.jruby.RubyProc.call(RubyProc.java:318)
logging-logstash-0 logstash     at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
logging-logstash-0 logstash     at java.base/java.lang.Thread.run(Thread.java:834)
zeph commented 4 years ago

logging-logstash-0 logstash [2020-07-21T13:17:06,393][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#, :backtrace=>["org/jruby/RubyThread.java:1278:in wakeup'", "org/jruby/RubyThread.java:1436:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:125:in block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:inblock in yield_messages'", "org/jruby/RubyArray.java:1809:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:inyield_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:in block in process_messages'", "org/jruby/RubyKernel.java:1189:incatch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:in process_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:inblock in poll'", "org/jruby/RubyKernel.java:1442:in loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:inblock in poll'", "org/jruby/RubyKernel.java:1189:in catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:inpoll'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:83:in block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:178:inrun_with_backoff'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/sqs/poller.rb:81:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-sns-sqs-2.1.0/lib/logstash/inputs/s3snssqs.rb:329:inblock in run_worker_thread'"]} logging-logstash-0 logstash [2020-07-21T13:17:06,475][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

zeph commented 4 years ago 
NAME                 READY   STATUS    RESTARTS   AGE
logging-logstash-0   1/1     Running   57         22h
logging-logstash-1   1/1     Running   55         22h
logging-logstash-2   1/1     Running   57         22h
zeph commented 4 years ago

shall work... https://github.com/cherweg/logstash-input-s3-sns-sqs/pull/44

christianherweg0807 commented 4 years ago

Hey Zeph, the maximum processing time should usually never be reached...if so, something inside the log_processing module is blocked ... bad encoding, hanging multiline codecs ... etc. To escape from that, i decided initially to kill such an blocked processor. Thank you for the pull request, I´ll test your code. (I´m afk for 2 weeks now.)

regards Christian

christianherweg0807 commented 4 years ago

At the moment I don´t understand, why the hole pod is dying...this should prevent this: https://github.com/cherweg/logstash-input-s3-sns-sqs/blob/a10fe282a9c68e71b6a0ae7b0c4045a3c327931d/lib/logstash/inputs/s3snssqs.rb#L287-L303

There is an control-thread, running thee worker ... if the worker dies (or is killed) the control thread should rise a new worker until there is an Stop? signal...

I´ve to debug you stack trace...

hmmm...

Christian

zeph commented 4 years ago

@christianherweg0807 thanks for looking into this <3

christianherweg0807 commented 4 years ago

Might be releated to this: https://github.com/elastic/logstash/issues/10612

Normally we have a conrol_thead -> calling the worker -> watcher inside the worker takes care of the max_timeout -> kills worker if timeout reached -> Controll_thread rises a new worker if not stop?

In your setup the kill signal now causes logstash´s to exit ... thats not mentioned.

Could you increase the :max_processing_time to ensure that all messages could be processed? (if not you have a data problem)

christian

zeph commented 4 years ago

@christianherweg0807 I had to reduce the max_processing_time drastically... to 3600/1800 seconds... it cannot be that it takes 30min to parse and ingest a 20/100 Mb file. I have 3SQS with 3 consumer threads each (on configuration). 3x EKS nodes each with a logstash... on machines that have 4 vCPUs and 8Gb of RAM. Now, on the 3 queues I get intake rates from 100 up to 2000 new files per hour. The system is not keeping up... CPU utilization is below 50% and cannot get more than 4Mbit/sec of outbound network per node... and I have nodes that can get to 25 Gbit c5a.xlarge

This with my patched version of ur code, with the wake instead of the kill, it is stable... but there are for example no assigned messages out of a queue when I would expect 3x3 = 9 ... same for the other queues... it simply looks stuck... and I keep see the Extender kicking in... until it says it is going to delete the message due to reached max

zeph commented 4 years ago

right now the configuration is 3 nodes, 4 vCPU each

3 queues, 2 consumers each... pipeline.workers at default == amount of detected vCPUs

zeph commented 4 years ago

I think I have the culprit... I got grok which is triggered only on 1 of the 3 logs type https://github.com/elastic/logstash/issues/10801#issuecomment-494760315

it crashes when that sqs/file-type is enabled...

zeph commented 4 years ago

nope... it is not grok... I mean, it is defined, but none triggering it (I did revert to 2.1.1)

$ k8s get pod
NAME               READY   STATUS    RESTARTS   AGE
s3sqs-logstash-0   0/1     Running   31         16h
s3sqs-logstash-1   1/1     Running   34         16h
s3sqs-logstash-2   1/1     Running   35         16h
zeph commented 4 years ago

ok, back to my patched version... added an exporter of the Node stats API s3sqs-logstash-2 exporter time="2020-07-25T09:38:09Z" level=error msg="Get http://localhost:9600/_node/stats: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" source="main.go:63"

I did reduce the thread count per queue back to default one... and let's see how it goes

zeph commented 4 years ago

same :/ (on 2.1.2-patched)

$ k8s get pod
NAME               READY   STATUS    RESTARTS   AGE
s3sqs-logstash-0   2/2     Running   29         31h
s3sqs-logstash-1   2/2     Running   34         31h
s3sqs-logstash-2   2/2     Running   28         31h

s3sqs-logstash-2 exporter time="2020-07-26T09:36:04Z" level=error msg="Get http://localhost:9600/_node/stats: dial tcp 127.0.0.1:9600: connect: connection refused" source="main.go:63"

ok, must be not enough capacity on CPU at this point...

zeph commented 4 years ago

but it did consume all the queues... maybe I can run only a POD on a bigger machine

logstash

consumed_cf

zeph commented 4 years ago

@christianherweg0807 it is now stable, I'm running on a 16 vCPU machine...but with my patch

I can give a try again without the patch if u want, but I got a strong feeling it needs to be in place

what about u?

zeph commented 4 years ago

consumption

zeph commented 4 years ago

nope... reverting to my patched version... it restarted too many times in 20h

$ k8s get pod 
NAME               READY   STATUS    RESTARTS   AGE
s3sqs-logstash-0   2/2     Running   109        20h
zeph commented 4 years ago

grrr.... also my patched version...

s3sqs-logstash-0 logstash [2020-07-31T14:57:31,395][WARN ][logstash.inputs.s3snssqs ][main][f5b81930a4d2bb131e82ed4f120cd591ace099130b9b291a856649340ff9febb] issuing :stop_polling on "stop?" signal {:queue=>"cloudtrail-logs-queue"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,396][INFO ][logstash.inputs.s3snssqs ][main][f5b81930a4d2bb131e82ed4f120cd591ace099130b9b291a856649340ff9febb] [control_thread] restarting a thread 2_1...  {:thread=>"#<Thread:0x7de8ebdb@/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/s3snssqs.rb:294 dead>"}
s3sqs-logstash-0 logstash warning: thread "Worker 039b4f6d45be10cdd899b409d21c198a6f61a9d2390b5478edc66353ff953aca/1" terminated with exception (report_on_exception is true):
s3sqs-logstash-0 logstash ThreadError: killed thread
s3sqs-logstash-0 logstash              wakeup at org/jruby/RubyThread.java:1278
s3sqs-logstash-0 logstash                 run at org/jruby/RubyThread.java:1436
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:125
s3sqs-logstash-0 logstash      yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413
s3sqs-logstash-0 logstash                each at org/jruby/RubyArray.java:1809
s3sqs-logstash-0 logstash      yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412
s3sqs-logstash-0 logstash    process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405
s3sqs-logstash-0 logstash               catch at org/jruby/RubyKernel.java:1189
s3sqs-logstash-0 logstash    process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336
s3sqs-logstash-0 logstash                loop at org/jruby/RubyKernel.java:1442
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331
s3sqs-logstash-0 logstash               catch at org/jruby/RubyKernel.java:1189
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:83
s3sqs-logstash-0 logstash    run_with_backoff at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:178
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:81
s3sqs-logstash-0 logstash   run_worker_thread at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/s3snssqs.rb:334
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,406][WARN ][logstash.inputs.s3snssqs ][main][7f839a77c5266da032581d850ed39712538e1cf4be80af32ab5fa136b2d3107d] issuing :stop_polling on "stop?" signal {:queue=>"elb-logs-queue"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,406][WARN ][logstash.inputs.s3snssqs ][main][7f839a77c5266da032581d850ed39712538e1cf4be80af32ab5fa136b2d3107d] issuing :stop_polling on "stop?" signal {:queue=>"elb-logs-queue"}
s3sqs-logstash-0 logstash warning: thread "Worker 039b4f6d45be10cdd899b409d21c198a6f61a9d2390b5478edc66353ff953aca/0" terminated with exception (report_on_exception is true):warning: thread "Ruby-0-Thread-32: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:345" terminated with exception (report_on_exception is true):
s3sqs-logstash-0 logstash ThreadError: killed thread
s3sqs-logstash-0 logstash              wakeup at org/jruby/RubyThread.java:1278
s3sqs-logstash-0 logstash                 run at org/jruby/RubyThread.java:1436
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:125
s3sqs-logstash-0 logstash      yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413
s3sqs-logstash-0 logstash                each at org/jruby/RubyArray.java:1809
s3sqs-logstash-0 logstash      yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412
s3sqs-logstash-0 logstash    process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405
s3sqs-logstash-0 logstash               catch at org/jruby/RubyKernel.java:1189
s3sqs-logstash-0 logstash    process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336
s3sqs-logstash-0 logstash                loop at org/jruby/RubyKernel.java:1442
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331
s3sqs-logstash-0 logstash               catch at org/jruby/RubyKernel.java:1189
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:83
s3sqs-logstash-0 logstash    run_with_backoff at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:178
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:81
s3sqs-logstash-0 logstash   run_worker_thread at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/s3snssqs.rb:334
s3sqs-logstash-0 logstash warning: thread "Ruby-0-Thread-31: /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:345" terminated with exception (report_on_exception is true):
s3sqs-logstash-0 logstash ThreadError: killed thread
s3sqs-logstash-0 logstash              wakeup at org/jruby/RubyThread.java:1278
s3sqs-logstash-0 logstash                 run at org/jruby/RubyThread.java:1436
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:125
s3sqs-logstash-0 logstash      yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413
s3sqs-logstash-0 logstash                each at org/jruby/RubyArray.java:1809
s3sqs-logstash-0 logstash      yield_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412
s3sqs-logstash-0 logstash    process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405
s3sqs-logstash-0 logstash               catch at org/jruby/RubyKernel.java:1189
s3sqs-logstash-0 logstash    process_messages at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336
s3sqs-logstash-0 logstash                loop at org/jruby/RubyKernel.java:1442
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331
s3sqs-logstash-0 logstash               catch at org/jruby/RubyKernel.java:1189
s3sqs-logstash-0 logstash                poll at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:83
s3sqs-logstash-0 logstash    run_with_backoff at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:178
s3sqs-logstash-0 logstash                 run at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:81
s3sqs-logstash-0 logstash   run_worker_thread at /usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/s3snssqs.rb:334
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,406][WARN ][logstash.inputs.s3snssqs ][main][f5b81930a4d2bb131e82ed4f120cd591ace099130b9b291a856649340ff9febb] issuing :stop_polling on "stop?" signal {:queue=>"cloudtrail-logs-queue"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,407][INFO ][logstash.inputs.s3snssqs ][main][f5b81930a4d2bb131e82ed4f120cd591ace099130b9b291a856649340ff9febb] [control_thread] restarting a thread 0_1...  {:thread=>"#<Thread:0xe4b5154@/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/s3snssqs.rb:330 dead>"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,408][INFO ][logstash.inputs.s3snssqs ][main][7f839a77c5266da032581d850ed39712538e1cf4be80af32ab5fa136b2d3107d] [control_thread] restarting a thread 0_1...  {:thread=>"#<Thread:0x56526182 dead>"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,408][INFO ][logstash.inputs.s3snssqs ][main][7f839a77c5266da032581d850ed39712538e1cf4be80af32ab5fa136b2d3107d] [control_thread] restarting a thread 1_1...  {:thread=>"#<Thread:0x4006c228 dead>"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,418][WARN ][logstash.inputs.s3snssqs ][main][f5b81930a4d2bb131e82ed4f120cd591ace099130b9b291a856649340ff9febb] issuing :stop_polling on "stop?" signal {:queue=>"cloudtrail-logs-queue"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,418][INFO ][logstash.inputs.s3snssqs ][main][f5b81930a4d2bb131e82ed4f120cd591ace099130b9b291a856649340ff9febb] [control_thread] restarting a thread 1_1...  {:thread=>"#<Thread:0x4c8fcc@/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/s3snssqs.rb:294 dead>"}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,559][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<ThreadError: killed thread>, :backtrace=>["org/jruby/RubyThread.java:1278:in `wakeup'", "org/jruby/RubyThread.java:1436:in `run'", "/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:125:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in `block in yield_messages'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in `yield_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:in `block in process_messages'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:in `process_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in `block in poll'", "org/jruby/RubyKernel.java:1442:in `loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.501/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll'", "/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:83:in `block in run'", "/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:178:in `run_with_backoff'", "/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/sqs/poller.rb:81:in `run'", "/usr/share/logstash/vendor/local_gems/fdbe1a50/logstash-input-s3-sns-sqs-2.1.2/lib/logstash/inputs/s3snssqs.rb:334:in `block in run_worker_thread'"]}
s3sqs-logstash-0 logstash [2020-07-31T14:57:31,569][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
s3sqs-logstash-0 exporter time="2020-07-31T14:58:03Z" level=error msg="Get http://localhost:9600/_node/stats: dial tcp 127.0.0.1:9600: connect: connection refused" source="main.go:63"
zeph commented 4 years ago

nevermind... unpatched it managed to catch up with the load... the important thing was always to scale up vertically and not horizontally