search

cherweg / logstash-input-s3-sns-sqs

logstash input downloading files from s3 Bucket by OjectKey from SNS/SQS
Other
29 stars 35 forks source link

Logstash dies on non-existing S3 key #8

Closed teekaay closed 6 years ago

teekaay commented 6 years ago

When Logstash consumes a message with metadata from SQS but the respective file in S3 does not exists anymore under the specified key, the whole logstash system goes down (loglevel FATAL).

This offending line seems to be in lib/logstash/inputs/s3snssqs.rb (line 236). We need to wrap the error AWS::S3::KeyError and omit the processing of the file in this case because it causes a fatal error.

Here is a stacktrace

[2018-04-06T14:01:54,637][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<Errno::ENOENT: No such file or directory - /tmp/logstash/LOGFILE.log>, :backtrace=>["org/jruby/RubyFile.java:366:in `initialize'", "org/jruby/RubyIO.java:1154:in `open'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:338:in `read_plain_file'", 

"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:333:in `read_file'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:265:in `process_local_log'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:218:in `process_log'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:202:in `block in handle_message'", "org/jruby/RubyArray.java:1734:in `each'", 

"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:189:in `handle_message'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:470:in `block in thread_runner'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in `block in yield_messages'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in `yield_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:in `block in process_messages'", "org/jruby/RubyKernel.java:1114:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:in `process_messages'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in `block in poll'", "org/jruby/RubyKernel.java:1292:in `loop'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll'", "org/jruby/RubyKernel.java:1114:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:469:in `block in thread_runner'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:487:in `run_with_backoff'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.0/lib/logstash/inputs/s3snssqs.rb:468:in `block in thread_runner'"]}

[2018-04-06T14:01:54,643][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit