Improvement: Shutdown of client should not be blocked by pending connection attempts

[dpasek-senacor]

Currently the shutdown of a EventStoreDBClient is blocked until all connection attempts have been completed.

It would be better if the shutdown of the client will abort the running connection attempts.

The problem is located in the GrpcClient class in the dicover method:

    private boolean discover(UUID previousId, Optional<Endpoint> candidate) {
        long attempts = 1;
...   
        for (; ; ) {
            logger.debug("Start connection attempt ({}/{})", attempts, settings.getMaxDiscoverAttempts());
            if (doConnect()) {
                try {
                    serverInfo = ServerFeatures.getSupportedFeatures(settings, channel);
                } catch (Exception e) {
                    logger.error("An exception happened when fetching server supported features", e);
                    return false;
                }
                currentChannelId = UUID.randomUUID();

                logger.info("Connection created successfully");

                return true;
            } else {
                ++attempts;
                if (attempts > settings.getMaxDiscoverAttempts()) {
                    logger.error("Maximum discovery attempt count reached: {}", settings.getMaxDiscoverAttempts());
                    return false;
                }

                logger.warn("Unable to find a node. Retrying... ({}/{})", attempts, settings.getMaxDiscoverAttempts());
                sleep(settings.getDiscoveryInterval());
            }
        }
    }

The current implementation will loop all connecton attempts before allowing the client to process other messages like the Shutdown message. Depending of the number of attempts und the retry interval this might block a client shutdown for many seconds. With the current defaults an connection attempt to a server which does not respond/is not availble would block for more than 10s: 3 times (500ms Retry-Interval + 3000ms for timeout on reading the server features) = 10500ms until Shutdown message will be processed.