search

logstash-plugins / logstash-input-sqs

Apache License 2.0
16 stars 40 forks source link

Socket error due timeout kills the pipeline instead to go in backoff retry #55

Closed andsel closed 3 years ago

andsel commented 4 years ago

Sometimes is the SQS plugin is in run and encounter an error from the HTTP client library, the plugin doesn't retry to reconnect but kills the pipeline.

stack trace ``` [2020-09-26T00:17:14,174][ERROR][logstash.javapipeline ][main] A plugin had an unrecoverable error. Will restart this plugin. Pipeline_id:main Plugin: 4, id=>"1e95d5586024e71da3c6c432de1c06d5ed6f9b05af26dfe45bfe945c182278f0", region=>"eu-west-1", queue=>"MonSysCloudwatchMetricsQueue", enable_metric=>true, codec=>"json_6f7b71ca-94a4-4dcc-8b24-debafa66959d", enable_metric=>true, charset=>"UTF-8">, role_session_name=>"logstash", polling_frequency=>20 > Error: Net::OpenTimeout Exception: Seahorse::Client::NetworkingError Stack: uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:41:in `ssl_socket_connect' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:985:in `connect' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:919:in `start' uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/delegate.rb:83:in `method_missing' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/connection_pool.rb:285:in `start_session' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/connection_pool.rb:92:in `session_for' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/handler.rb:119:in `session' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/handler.rb:71:in `transmit' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/net_http/handler.rb:45:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/content_length.rb:12:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/xml/error_handler.rb:8:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/request_signer.rb:88:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/sqs_queue_urls.rb:13:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/query/handler.rb:27:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/endpoint_pattern.rb:27:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/endpoint_discovery.rb:67:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/endpoint.rb:41:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/plugins/response_target.rb:21:in `call' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/request.rb:70:in `send_request' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.429/lib/seahorse/client/base.rb:207:in `block in define_operation_methods' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:390:in `send_request' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:383:in `get_messages' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:332:in `block in poll' org/jruby/RubyKernel.java:1446:in `loop' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:331:in `block in poll' org/jruby/RubyKernel.java:1193:in `catch' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.429/lib/aws-sdk-resources/services/sqs/queue_poller.rb:330:in `poll' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-sqs-3.1.2/lib/logstash/inputs/sqs.rb:143:in `block in run' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-sqs-3.1.2/lib/logstash/inputs/sqs.rb:164:in `run_with_backoff' /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-sqs-3.1.2/lib/logstash/inputs/sqs.rb:142:in `run' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:328:in `inputworker' /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:320:in `block in start_input' ```

the problem is that the client throws Seahorse::Client::NetworkingError in case of Net::OpenTimeout but the rescue code https://github.com/logstash-plugins/logstash-input-sqs/blob/f21a643d7d0595ad1ef711fe5937a22725fccac6/lib/logstash/inputs/sqs.rb#L166 handle only subclasses of Aws::SQS::Errors::ServiceError