High CPU usage when clients do not properly disconnect/connection reset

Logstash information: Logstash version: 8.4.3 JVM version: bundled

Description of the problem including expected versus actual behavior: The plugin is not properly detecting client disconnections/resets. It seems it keeps trying to read from the closed socket, making the CPU usage grow every time a client disconnects. It may be a JRuby issue (https://github.com/jruby/jruby/issues/7961).

The problem is happening on the socket.each method. When the client connection is closed/reset, the code expects it to raise an ECONNRESET, stopping the loop, closing the connection and removing it from the connection counter. Instead, it doesn't raise any error, hangs and burns CPU.

An alternative solution is to change it to read using a non-blocking approach. Apparently, the read_nonblock is not affected by this issue.

Hot threads at 2023-09-25T10:41:31+02:00, busiestThreads=10000: 
================================================================================
61.74 % of cpu usage, state: runnable, thread name: 'input|syslog|tcp|10.1.1.12:5000}', thread id: 1726 
    app//org.jruby.util.io.PosixShim.read(PosixShim.java:158)
    app//org.jruby.util.io.OpenFile$2.run(OpenFile.java:1330)
    app//org.jruby.util.io.OpenFile$2.run(OpenFile.java:1316)
...

Steps to reproduce:

Start Logstash with the following pipeline:

 input {
   syslog {
       port => 5555
   }
}

 output {   stdout {} }

Run the following client code, checking the CPU usage:

  require 'socket'

  HOST = 'localhost'
  PORT = 5555

  def connect_and_close
    socket = TCPSocket.new(HOST, PORT)
    linger = [1,0].pack('ii')
    socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_LINGER, linger)
    socket.close
  end

  def tcp_receiver(socket)
    socket.each { |line| puts line }
  rescue Errno::ECONNRESET
    puts "connection reset"
  end

  server_thread = Thread.new do
    server_socket = TCPServer.new(HOST, PORT)
    loop do
      socket = server_socket.accept

      Thread.new(socket) do |socket|
        tcp_receiver(socket)
      end
    end
  end

  sleep 1
  10.times { connect_and_close }

logstash-plugins / logstash-input-syslog

High CPU usage when clients do not properly disconnect/connection reset #74