logstash-plugins / logstash-output-tcp

Apache License 2.0
9 stars 31 forks source link

Close the underlying socket if the SSL/TLS connection is closed #38

Closed imacube closed 5 years ago

imacube commented 5 years ago

I've been experiencing logstash-outputs-tcp throwing an EOFError and restarting logstash fixes it.

This logstash instance is sending to a logstash behind an AWS ELB that is running on kubernetes. My suspicion is the ELB is rotating IPs and that logstash isn't closing and reestablishing the TCP connection. I manually made this change a month ago and have not seen the problem recur.

[2019-03-02T22:39:11,889][WARN ][logstash.outputs.tcp     ] tcp output exception {:host=>"BF4114A6-096A-4732-8E82-C21EED89A910.eu-central-1.elb.amazonaws.com", :port=>9000, :exception=>#<EOFError: End of file reached>, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:857:in `sysread'", "/opt/git/logstash-output-tcp/lib/logstash/outputs/tcp.rb:158:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-json_lines-3.0.6/lib/logstash/codecs/json_lines.rb:48:in `encode'", "/opt/git/logstash-output-tcp/lib/logstash/outputs/tcp.rb:204:in `receive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `block in multi_receive'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:114:in `multi_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:97:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:390:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:389:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:304:in `block in start_workers'"]}
imacube commented 5 years ago

Yesterday the logstash instance with this patch hit the same error. This clearly didn't fix it so closing this while I try other ideas.