Thread crashes and leaves stale files

[danielkasen]

I'm running with multiple workers and about every 5 minutes I get this error in the log. It appears to crash all the threads and leave half processed logs in the /tmp/logstash dir. After about 3 days this fills up the disk. Any thoughts?

{"level":"FATAL","loggerName":"logstash.runner","timeMillis":1524016154842,"thread":"LogStash::Runner","logEvent":{"message":"An unexpected error occurred!","error":{"metaClass":{"metaClass":{"metaClass":{"error":"No such file or directory - /tmp/logstash/4T1300Z_10.0.8.219_4j2uczw1.log","backtrace":["org/jruby/RubyFile.java:366:in initialize'","org/jruby/RubyIO.java:1154:inopen'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:336:in read_plain_file'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:331:inread_file'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:263:in process_local_log'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:218:inprocess_log'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:202:in block in handle_message'","org/jruby/RubyArray.java:1734:ineach'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:189:in handle_message'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:490:inblock in thread_runner'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:413:in block in yield_messages'","org/jruby/RubyArray.java:1734:ineach'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:412:in yield_messages'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:405:inblock in process_messages'","org/jruby/RubyKernel.java:1114:in catch'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:404:inprocess_messages'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:336:in block in poll'","org/jruby/RubyKernel.java:1292:inloop'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in block in poll'","org/jruby/RubyKernel.java:1114:incatch'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in poll'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:489:inblock in thread_runner'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:507:in run_with_backoff'","/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-s3-sns-sqs-1.5.2/lib/logstash/inputs/s3snssqs.rb:488:inblock in thread_runner'"]}}}}}}

[danielkasen]

On more inspection this only seems to happen when I set the consumer_threads about 1. So looks like there's a thread condition that's failing.

[cherweg]

Hi Daniel,

i´ll take a look, but it´s difficult.

Christian

[christianherweg0807]

Clean after refactoring.