"Client write error, trying connect" - "IOError: Broken pipe" using openjdk-11

[buettner123]

Hi, we are using the logstash lumberjack output plugin in order to send logs collected by metricbeat to some other logstash instance. When trying to upgrade to openjdk 11 we are facing issues connecting:

• Version: 7.3.1 (also tried with 7.6.2)
• Operating System: CentOS 7
• Config File:

  
  # upstream:
  output {
  lumberjack {
    hosts => ["my-host"]
    port => 5000
    ssl_certificate => "/etc/logstash/ssl/lumberjack.cert"
    codec => json
  }
  }

downstream:

input { beats { id => "lumberjack" codec => json port => 5000 ssl_certificate => "..." ssl_key => "..." ssl => true }

}

- Steps to Reproduce:

  - install openjdk-11

java -version openjdk version "11.0.7" 2020-04-14 LTS OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS) waOpenJDK 64-Bit Server VM 18.9 (build 11.0.7+10-LTS, mixed mode, sharing)

  - start logstash and wait a few seconds

- Expected result:
  - the connection is established and you see messages appearing on the downstream

- Current result:
  - there is an error message:

[ERROR][logstash.outputs.lumberjack] Client write error, trying connect {:e=>#, :backtrace=>["org/jruby/ext/openssl/SSLSocket.java:950:in syswrite'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:116:insend_payload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:132:in write_sync'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:42:inwrite'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:65:in flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:219:inblock in buffer_flush'", "org/jruby/RubyHash.java:1419:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:216:inbuffer_flush'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/buffer.rb:159:in buffer_receive'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:52:inblock in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-json-3.0.5/lib/logstash/codecs/json.rb:42:in encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:31:inblock in encode'", "org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:45:in time'", "org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:44:intime'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:30:in encode'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-lumberjack-3.1.7/lib/logstash/outputs/lumberjack.rb:59:inreceive'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in block in multi_receive'", "org/jruby/RubyArray.java:1792:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:89:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:118:inmulti_receive'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:239:inblock in start_workers'"]}

The same configuration works fine with oracle jdk 8. According to https://www.elastic.co/support/matrix#matrix_jvm logstash 7.3 (and 7.6) both should support openjdk 11, seems the plugin does not, or am I missing something here?

If you need more details, please let me know.

[Purneau]

@buettner123 any progress?

[buettner123]

@Purneau still waiting for a response here and postponed the update for now. Might try again with logstash 7.10 and openjdk 14 next year or start searching for workarounds.

[Purneau]

@buettner123 For anyone running into this in the future:

We reproduced (logstash 7.10.1 with packaged Java 11) that this error is thrown when the connection is inactive for longer than the client_inactivity_timeout value of the beats input plugin (defaults to 60 seconds). We also observed that although the error is thrown, the data is still successfully sent over a fresh connection.

We increased client_inactivity_timeout to a value larger than our transfer interval.

It would be nice if the connection closure would be handled correctly by the lumberjack output plugin or that we could specify the same timeout value on the output plugin side, so that no connection attempt is being made over the old connection.

[buettner123]

@Purneau Thanks for the hint, will check this and report if it also fixes the issue for me!

[buettner123]

Increasing the client_inactivity_timeout on the downstream beats input solved the issue for us as well. I have different intervals on different types of data and it seems like the 60 secs exactly overlapped with one of the intervals. Faced the message again after rebooting the machine but it only appeared once and the data is successfully sent afterwards.

Closing the issue, thanks again for the hint @Purneau