Connect to jetty failed with java.nio.channels.ClosedSelectorException

[being23]

Hello, I wrote a custom processor for NiFi based on Jetty(Version:9.4.3). Today I found there were thousands of connections to one of the nodes, named after A . I used shell command ss to try to find where the connections came from and it showed that most of the connections came from one of the client hosts, similarly, named after B. In host B, I found most of the connections were in state: FIN-WAIT-1, while in host A most of the connections were in state: ESTABLISHED.

So I checked the log and found some warnings in the log, like the follwoing:

2017-11-24 16:29:52,642 WARN [ListenOConnectHttp (dec4403b-015d-1000-0000-000057a76a48) Web Server-320-acceptor-2@4a01a8cc-ServerConnector@4107e53{HTTP/1.1,[http/1.1]}{0.0.0.0:8083}] o.eclipse.jetty.server.AbstractConnector
java.nio.channels.ClosedSelectorException: null
        at sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)
        at org.eclipse.jetty.io.ManagedSelector.size(ManagedSelector.java:127)
        at org.eclipse.jetty.io.SelectorManager.chooseSelector(SelectorManager.java:157)
        at org.eclipse.jetty.io.SelectorManager.accept(SelectorManager.java:199)
        at org.eclipse.jetty.io.SelectorManager.accept(SelectorManager.java:184)
        at org.eclipse.jetty.server.ServerConnector.accepted(ServerConnector.java:362)
        at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:353)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
        at java.lang.Thread.run(Thread.java:745)

It showed that the selector chosen for the channel was closed. This might result in the client read/write timeout. My colleague told me if the client experience read/write timeout, connection will be closed and a new connection will be created. Here, I think the operation of closing the connection sent a FIN packet, but it didn't receive ACK packet from Jetty, so the connections were in state FIN-WAIT-1 on the host B. And the new created connection will suffer the same problem. Finally, there will be more and more connections by keeping creating new connections.

My question is why there is a FIN packet, but no ACK packet, is this related to the ClosedSelectorException? And how to solve the problem?

Thanks!

[sbordet]

A ClosedSelectorException can only happen if the selector has been closed, which in turn only happens when Jetty's ManagedSelector is stopped, which in turn happens when Jetty's SelectorManager is stopped, which means that either the client or the server is being stopped.

You don't say if the stack trace happened in host A or B.

Either case, you or something else is stopping Jetty on one node, and that is the cause of your problems.