search

dynatrace-oss / logstash-output-dynatrace

A Logstash output plugin for sending logs to Dynatrace
4 stars 1 forks source link

Intermittent Net::OpenTimeout - retry on plugin level or in Logstash? #20

Closed dominikskiba closed 2 years ago

dominikskiba commented 2 years ago

Is your feature request related to a problem? Please describe. We receive Net::OpenTimeout error occasionally while pushing logs to Dynatrace SaaS.

[2022-09-08T10:49:47,482][ERROR][logstash.javapipeline    ][logstash-dt] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"logstash-dt", :error=>"(OpenTimeout) Net::OpenTimeout", 
:exception=>Java::OrgJrubyExceptions::RuntimeError, 
:backtrace=>["E_3a_.logstash.logstash_minus_8_dot_3_dot_2.vendor.jruby.lib.ruby.stdlib.net.protocol.ssl_socket_connect(E:/logstash/logstash-8.3.2/vendor/jruby/lib/ruby/stdlib/net/protocol.rb:41)", 
"E_3a_.logstash.logstash_minus_8_dot_3_dot_2.vendor.jruby.lib.ruby.stdlib.net.http.connect(E:/logstash/logstash-8.3.2/vendor/jruby/lib/ruby/stdlib/net/http.rb:985)", 
"E_3a_.logstash.logstash_minus_8_dot_3_dot_2.vendor.jruby.lib.ruby.stdlib.net.http.do_start(E:/logstash/logstash-8.3.2/vendor/jruby/lib/ruby/stdlib/net/http.rb:924)", 
"E_3a_.logstash.logstash_minus_8_dot_3_dot_2.vendor.jruby.lib.ruby.stdlib.net.http.start(E:/logstash/logstash-8.3.2/vendor/jruby/lib/ruby/stdlib/net/http.rb:913)", 
"E_3a_.logstash.logstash_minus_8_dot_3_dot_2.vendor.jruby.lib.ruby.stdlib.net.http.request(E:/logstash/logstash-8.3.2/vendor/jruby/lib/ruby/stdlib/net/http.rb:1465)", 
"E_3a_.logstash.logstash_minus_8_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_dynatrace_minus_0_dot_2_dot_1.lib.logstash.outputs.dynatrace.send(E:/logstash/logstash-8.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-dynatrace-0.2.1/lib/logstash/outputs/dynatrace.rb:117)", 
"E_3a_.logstash.logstash_minus_8_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_dynatrace_minus_0_dot_2_dot_1.lib.logstash.outputs.dynatrace.multi_receive(E:/logstash/logstash-8.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-dynatrace-0.2.1/lib/logstash/outputs/dynatrace.rb:79)", 
"org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:143)", 
"org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:121)", 
"E_3a_.logstash.logstash_minus_8_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(E:/logstash/logstash-8.3.2/logstash-core/lib/logstash/java_pipeline.rb:300)"], 
:thread=>"#<Thread:0x1babbff4 sleep>"}

Describe the solution you'd like Possibility of retries in case of connection timeout. When this issue happens, we simply restart logstash service and it resumes normally.

Additional context I see Dynatrace plugin is able to handle and retry HTTP errors, but not this one. My question is, can connection timeout be handled and retried on plugin level, or it must be done in Logstash as described in this issue? The main reason I'm asking is because request to implement Logstash ability to restart failed pipelines has been open for 1,5 years now and doesn't have developer assigned yet. Thank you.

arminru commented 2 years ago

Hey @dominikskiba! Thank you for your report. We introduced retrying on Net::OpenTimeout in #25. You can find it in this release: https://github.com/dynatrace-oss/logstash-output-dynatrace/releases/tag/v0.3.2 and https://rubygems.org/gems/logstash-output-dynatrace/versions/0.3.2.