search

logstash-plugins / logstash-output-s3

Apache License 2.0
58 stars 151 forks source link

Gzip files not deleted on Linux #214

Open max-wittig opened 4 years ago

max-wittig commented 4 years ago

Similar issues:

The temporary folder is mounted inside a container. When I tried to manually delete tmpfiles inside the container, that seems to work. But the logstash doesn't delete files on it's own after upload

max-wittig commented 4 years ago

An error occured in the on_complete uploader {:exception=>ArgumentError, :message=>"parent directory is world writable, FileUtils#remove_entry_secure does not work; abort: \"/tmp/logstash/61d8c520-7e48-4223-9fa3-e0f23d42a42a\" (parent directory mode 40777)", :path=>"/tmp/logstash/61d8c520-7e48-4223-9fa3-e0f23d42a42a/ls.s3.64c857d6-1f27-4829-afe1-6723193bcc5e.2019-11-27T10.10.part26.txt.gz", :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:685:in remove_entry_secure'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:611:inblock in rm_r'", "org/jruby/RubyArray.java:1800:in each'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/fileutils.rb:609:inrm_r'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.10/lib/logstash/outputs/s3/temporary_file.rb:54:in delete!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.10/lib/logstash/outputs/s3.rb:379:inclean_temporary_file'", "org/jruby/RubyMethod.java:120:in call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.10/lib/logstash/outputs/s3/uploader.rb:52:inupload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.1.10/lib/logstash/outputs/s3/uploader.rb:29:in block in upload_async'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.5/lib/concurrent/executor/java_executor_service.rb:85:inrun'"]}

magical-travvo commented 4 years ago

Seeing the same thing in a Kubernetes hosted container, temp dir is mounted into an emptyDir volume, but old temp files aren't being cleaned up.

magical-travvo commented 4 years ago

This only appears to be an issue if the temporary directory is world writable i.e. a volume mount point. On our Kubernetes deployment, we've mounted /tmp/logstash as a volume, then changed the temporary file directory: temporary_directory => "/tmp/logstash/s3" The s3 directory gets created at container startup (assuming it doesn't already exist) and files are deleted correctly.