search

logstash-plugins / logstash-output-s3

Apache License 2.0
58 stars 151 forks source link

Fixes the no such file or directory issue. #251

Closed mashhurs closed 1 year ago

mashhurs commented 1 year ago

Issue description

When temporary file is staled and Logstash is closing the s3-output plugin, the plugin is accessing the stolen file and producing an error:

[2022-12-21T17:05:58,371][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", 
:exception=>#<Errno::ENOENT: No such file or directory - No such file or directory -
 /temp/s3-temp/4e46daa5-07af-4085-a819-bac9ced3d25a/test-2022.12.21/ls.s3.9b6828b5-cff3-4b97-b1e1-ff83af405817.2022-12-21T17.05.part0.txt>, :backtrace=>["org/jruby/RubyFileTest.java:248:in `size'", 
"/temp/logstash-output-s3/lib/logstash/outputs/s3/temporary_file.rb:50:in `size'",
 "/temp/logstash-output-s3/lib/logstash/outputs/s3.rb:373:in `upload_file'", "/temp/logstash-output-s3/lib/logstash/outputs/s3.rb:276:in `block in close'", "/temp/logstash-output-s3/lib/logstash/outputs/s3/file_repository.rb:79:in `block in each_files'", 
"/temp/logstash-output-s3/lib/logstash/outputs/s3/file_repository.rb:27:in `block in with_lock'", "org/jruby/ext/thread/Mutex.java:171:in `synchronize'", "/temp/logstash-output-s3/lib/logstash/outputs/s3/file_repository.rb:26:in `with_lock'", "/temp/logstash-output-s3/lib/logstash/outputs/s3/file_repository.rb:79:in `block in each_files'", "org/jruby/RubyArray.java:1865:in `each'", "/temp/logstash-output-s3/lib/logstash/outputs/s3/file_repository.rb:78:in `each_files'", "/temp/logstash-output-s3/lib/logstash/outputs/s3.rb:275:in `close'", "/logstash/logstash-core/lib/logstash/plugin.rb:98:in `do_close'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:75:in `do_close'", "org/jruby/RubyArray.java:1865:in `each'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:483:in `shutdown_workers'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:209:in `run'", "/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/logstash/config/s3output.conf"], :thread=>"#<Thread:0x16a00e28@/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

Solution

This PR introduces proper lock mechanism and atomic operations to eliminate the issue.

mashhurs commented 1 year ago

Covered by https://github.com/logstash-plugins/logstash-output-s3/pull/252