search

logstash-plugins / logstash-output-s3

Apache License 2.0
58 stars 151 forks source link

Too many open files in syste #238

Open kooksee opened 3 years ago

kooksee commented 3 years ago

Logstash information:

[2021-08-04T13:32:10,894][WARN ][logstash.outputs.s3 ][main] Uploading failed, retrying (#1 of Infinity) {:exception=>Errno::ENFILE, :message=>"Too many open files in system - Too many open files in system", :path=>"/var/folders/12/6l9lyttx0wnf3gx02zk58f1c0000gn/T/logstash/44e9a2be-dbee-4c60-bc53-3fdfc042e829/nginxlog/2021-08-02/22f73370abdda3c4/4e229836fcc80854/ls.s3.81cf3b10-3b68-4ca8-8a91-336d5c2eb34e.2021-08-04T13.31.part0.txt", :backtrace=>["org/jruby/ext/io/wait/IOWaitLibrary.java:107:inwait_readable'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:181:in rbuf_fill'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:157:inreaduntil'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:167:in readline'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:40:inread_status_line'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:29:in read_new'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:962:inconnect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:919:instart'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in method_missing'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/connection_pool.rb:285:instart_session'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/connection_pool.rb:92:in session_for'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/handler.rb:119:insession'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/handler.rb:71:in transmit'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/net_http/handler.rb:45:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/content_length.rb:12:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_request_signer.rb:88:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_host_id.rb:14:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/xml/error_handler.rb:8:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_redirects.rb:15:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/retry_errors.rb:108:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_md5s.rb:31:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/rest/handler.rb:7:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/user_agent.rb:12:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/endpoint.rb:41:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_validator.rb:21:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/raise_response_errors.rb:14:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/aws-sdk-core/plugins/param_converter.rb:20:incall'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/plugins/response_target.rb:21:in call'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/request.rb:70:insend_request'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.632/lib/seahorse/client/base.rb:207:in block in define_operation_methods'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:inblock in put_object'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in open_file'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:input_object'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in upload'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.632/lib/aws-sdk-resources/services/s3/object.rb:252:inupload_file'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.4/lib/logstash/outputs/s3/uploader.rb:40:in upload'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.3.4/lib/logstash/outputs/s3/uploader.rb:30:inblock in upload_async'", "/Users/barry/Downloads/logstash-7.14.0/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in run'"]}

Please include the following information:

  1. Logstash version (e.g. bin/logstash --version) logstash 7.14.0
  2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) mac
  3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes)

./bin/logstash -f ./config/heartbeat.conf

  1. How was the Logstash Plugin installed local

JVM (e.g. java -version):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

  1. JVM version (java -version)
  2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
  3. Value of the JAVA_HOME environment variable if set.

OS version (uname -a if on a Unix-like system):

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including (e.g.) pipeline definition(s), settings, locale, etc. The easier you make for us to reproduce it, the more likely that somebody will take the time to look at it.

1. 2. 3.

Provide logs (if relevant):

alexfrier commented 1 year ago

Has anyone looked into this further? I am experiencing this as well. Curious on if anyone's solved this.