aerospike / aerospike-client-java

Aerospike Java Client Library
Other
236 stars 212 forks source link

closing client hangup #87

Closed sirpiotrek closed 6 years ago

sirpiotrek commented 6 years ago

In the latest version of aerospike java client we met problems with closing connection to our server. Frequently when server is going down it is blocked on com.aerospike.client.async.AsyncClient.close() method. (we waited a few hours, before killing the service).

Thread 32575: (state = BLOCKED)

Maybe adding timeout in waitAsyncComplete method can resolve the issue.

sirpiotrek commented 6 years ago

After reviewing Cluster.close code, I think that the problem occures when notify from notifyAsyncComplete is called before waitAsyncComplete. In such scenario thread will be hung as in our case.

BrianNichols commented 6 years ago

I don't think that is the reason because both the notify and wait calls are synchronized. If notify() was called first, asyncComplete should be true and wait() will not be called.

It's possible the event loop never quits because pending count is invalid or a transaction is stuck in limbo forever. You can verify this by adding the following debug messages in Cluster.java. Let us know the results.

    private synchronized void waitAsyncComplete() {
        while (! asyncComplete) {
            try {
                System.out.println("call wait()");
                super.wait();
            }
            catch (InterruptedException ie) {
                System.out.println("thread interrupted");
            }
        }
    }

    private synchronized void notifyAsyncComplete() {
        System.out.println("notifyAsyncComplete() called");
        asyncComplete = true;
        super.notify();
    }