Deadlock in Client.close()

[georgeto]

This scenario occured during execution of the JsonTest, but I'm pretty sure it is also possible in a real enviroment.

To understand this problem, it is important to know, that Selector.selectNow() synchronizes on Selector, Selector.keys() and Selector.selectedKeys().

public int selectNow() throws IOException {
    return lockAndDoSelect(0);
}

private int lockAndDoSelect(long timeout) throws IOException {
    synchronized (this) {
        if (!isOpen())
            throw new ClosedSelectorException();
        synchronized (publicKeys) {
            synchronized (publicSelectedKeys) {
                return doSelect(timeout);
            }
        }
    }
}

[Thread 1]

1. Client.update() locks selector
2. Client's received handler sends a tcp message with connection.sendTCP()

[Thread 2]

3. Client.stop() is called, and calls Client.close()
4. Client.close() closes the tcp connection
5. Client.close() grabs updateLock
6. Client.close() calls selector.selectNow(), and waits for [Thread 1] to unlock selector

[Thread 1]

6. Cause tcp connection is closed, Client.sendTCP() throws an IOException and calls Client.close()
7. Client.close() waits for [Thread 2] to unlock updateLock