Concurrency issues with AFSocketServerConnector

[kevink-sq]

Describe the bug We found our production Jetty servers would stop processing requests after traffic spikes due to junixsocket throwing ConcurrentModificationException. This bug may have been a regression introduced in optimizing the key removal: https://github.com/kohlschutter/junixsocket/commit/a7c310662393e4ce1a2c288a18a8bdbdb7140235

java.util.ConcurrentModificationException
        at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1597)
        at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1620)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.updateKeys(ManagedSelector.java:709)
        at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:539)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:455)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:248)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

A PR has been raised to propose a fix: https://github.com/kohlschutter/junixsocket/pull/141

To Reproduce Refer to https://github.com/kevink-sq/jetty-concurrency-issue-poc

Notes Other issues were also detected during isolated load tests in the poc and was fixed by introducing an atomic check to AFSelectorKey#cancel. This issue appeared when running vegeta load tests and forcibly killing the vegeta load:

2023-10-25 01:39:50.857:DEBUG:oejs.AbstractConnector:etp827671026-30-acceptor-0@5ee135f3-AFSocketServerConnector@6cec2a14[org.newsclub.net.unix.AFUNIXSocketAddress[path=junix-ingress.sock]]: Accept Closed by Interrupt
java.nio.channels.ClosedByInterruptException
        at org.newsclub.net.unix.jetty.AFSocketServerConnector.accept(AFSocketServerConnector.java:352)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by:
java.net.SocketException: Too many open files
        at org.newsclub.net.unix.NativeUnixSocket.accept(Native Method)
        at org.newsclub.net.unix.AFSocketImpl.accept0(AFSocketImpl.java:262)
        at org.newsclub.net.unix.AFServerSocket.accept1(AFServerSocket.java:307)
        at org.newsclub.net.unix.AFServerSocketChannel.accept(AFServerSocketChannel.java:108)
        at org.newsclub.net.unix.AFUNIXServerSocketChannel.accept(AFUNIXServerSocketChannel.java:44)
        at org.newsclub.net.unix.AFUNIXServerSocketChannel.accept(AFUNIXServerSocketChannel.java:27)
        at org.newsclub.net.unix.jetty.AFSocketServerConnector.accept(AFSocketServerConnector.java:340)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

[kohlschuetter]

Fixing with #141

[kohlschuetter]

Fixed in junixsocket 2.8.2. Thanks again for reporting and providing a patch, @kevink-sq !