Increasing Initial connections per host causes NoAvailableHostsException (No hosts to borrow from)

[jsabin]

I am running a 6 node cluster in AWS. Setting the setInitConnsPerHost() with RING_DESCRIBE and TOKEN_AWARE to a value larger than 16 causes NoAvailableHostsException. I can reproduce the problem on a local single Cassandra node but I have to set the initial connections to a larger value.

This appears to be a timing issue with the initial connections because if I run the client in a debugger, the operation succeeds.

I cannot determine what the initial connections value should be because it is different for each cluster and I'm not sure if its related to the size of the cluster or if its network related. Changing setConnectTimeout() or setSocketTimeout() does not prevent the exception.

Here is the exception:

Exception in thread "main" com.netflix.astyanax.connectionpool.exceptions.NoAvailableHostsException: NoAvailableHostsException: [host=None(0.0.0.0):0, latency=0(0), attempts=0]No hosts to borrow from at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.(RoundRobinExecuteWithFailover.java:48) at com.netflix.astyanax.connectionpool.impl.TokenAwareConnectionPoolImpl.newExecuteWithFailover(TokenAwareConnectionPoolImpl.java:83) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:338) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:517) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.internalDescribeKeyspace(ThriftKeyspaceImpl.java:268) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.describeKeyspace(ThriftKeyspaceImpl.java:264) at Main.main(Main.java:55)

The code I used is this: Version 2.0.2 of Astyanax

public class Main { newColumnFamily(INDEX_COLUMN_FAMILY_NAME, StringSerializer.get(), StringSerializer.get()); private static final ColumnFamily<String, String> STRING_COLUMN_FAMILY = newColumnFamily("string_index", StringSerializer.get(), StringSerializer.get());

public static void main(String[] args)
        throws ConnectionException
{
    int maxConnections = Integer.parseInt(args[0]);
    int initialConnections = Integer.parseInt(args[1]);

    AstyanaxContext.Builder builder = new AstyanaxContext.Builder()
            .forCluster("us-east")
            .forKeyspace("myKeyspace")
            .withAstyanaxConfiguration(new AstyanaxConfigurationImpl()
                            .setDiscoveryType(NodeDiscoveryType.RING_DESCRIBE)
                            .setConnectionPoolType(ConnectionPoolType.TOKEN_AWARE)
            )
            .withConnectionPoolConfiguration(new ConnectionPoolConfigurationImpl("MyConnectionPool")
                            .setPort(9160)
                            .setMaxConnsPerHost(maxConnections)
                            .setInitConnsPerHost(initialConnections)
                            .setSeeds("localhost")
            )
            .withConnectionPoolMonitor(new Slf4jConnectionPoolMonitorImpl());

    AstyanaxContext<Keyspace> context = builder.buildKeyspace(ThriftFamilyFactory.getInstance());

    context.start();

    Keyspace keyspace = context.getClient();
    KeyspaceDefinition keyspaceDefinition = keyspace.describeKeyspace();
    System.out.println(keyspaceDefinition.getName());

    OperationResult<ColumnList<String>> result = keyspace.prepareQuery(STRING_COLUMN_FAMILY).getRow("myRow").execute();
    for (Column<String> stringColumn : result.getResult()) {
        System.out.println(stringColumn.getStringValue());
    }
}

}