On Further Investigation, we found that in Reservation class,
WaitReservationPermanent method, all threads are waiting at
semaphorePermanentTCP.acquire(permitsPermanentTCP);
The size of the queue private final BlockingQueue queue = new
LinkedBlockingQueue(); have reached to 4000 around.
This seems that release on the semaphore is never called. The permanent
TCP does not release the permits when the channel is closed.
However, the peer is able to receive the requests and even reply to them.
Relay uses the permanent connection, is it something to do with them. We
are using multiple manual relays and connect with them ?
You need to try to close the PeerConnection manually.
Any idea why request are not getting timed out and semaphore are not
releasing?
Most likely, closing the TCP connection does not release the semaphore.
I just created a testcase in TestConnection
Attaching the threaddump, if that can help. From my understanding thread
tagged with name "pool-2-thread-1" is the culprit and deadlocked.
When you create the permanent connection, you need to add something like
this:
I remember that I thought about releasing automatically the semaphore,
but decided against it. The idea of a PeerConnection is that it should
not matter if it is currently connected to a TCP channel or not. If it
is connect, then PeerConnection should use this connection, if not, then
PeerConnection creates a new connection. If you don't need the
connection anymore, then you call close, and the semaphore is released.
However, this is error prone, and I'll need to change this.
This seems that release on the semaphore is never called. The permanent TCP does not release the permits when the channel is closed.
You need to try to close the PeerConnection manually.
Most likely, closing the TCP connection does not release the semaphore. I just created a testcase in TestConnection
When you create the permanent connection, you need to add something like this:
peerConnection.closeFuture().addListener(new BaseFutureAdapter<FutureDone>() {
@Override
public void operationComplete(FutureDone future) throws Exception {
peerConnection.close();
}
});
I remember that I thought about releasing automatically the semaphore, but decided against it. The idea of a PeerConnection is that it should not matter if it is currently connected to a TCP channel or not. If it is connect, then PeerConnection should use this connection, if not, then PeerConnection creates a new connection. If you don't need the connection anymore, then you call close, and the semaphore is released.
However, this is error prone, and I'll need to change this.