Could not fullfill request on this host CassandraClient with Hector

[ricdong]

hi all,

I am using Cassandra with 3 data centers, each DC has more than 10 nodes.

the schema for a keyspace: {DC1:3, DC2:3, DC3:3}

now, I put some rows using hector with CL Local_Quorum in DC1 ，and then I get a row with the same CL Local_Quorum in DC1，some exceptions were occured:

2012-12-05 21:26:49,667 - WARN [pool-1-thread-3:JCLLoggerAdapter@379] - Could not fullfill request on this host CassandraClient172.16.74.31:9160-1 2012-12-05 21:26:49,668 - WARN [pool-1-thread-3:JCLLoggerAdapter@437] - Exception: me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException() at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:163) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:145) at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258) at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131) at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:167) at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:66) at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:62) at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85) at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:61) org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: TimedOutException() at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12830) at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:762) at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734) at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:151) ... 19 more

any suggestion with this issue? thanks.

[ricdong]

sorry, more details abort the version we use.

Cassandra with dsc-1.0.5, and Hector with hector-core-1.0-5.jar

[eiti-kimura-movile]

I suggest you upgrade the hector version and retry the operation

[ricdong]

thanks eitkimura.

I just upgrade the version to hector-core-1.1-2.jar, but it still throw the same exception with the CL LocalQuorum for read.

[fatemabw]

Hi, I am using Hector client to communicate with cassandra on which client-node encryption is enabled, but when I tried connecting Hector client it gives errors. I set the truststore and password as well in my code but didn't figure out what exactly need to be done to use Hector client to connect to encrypted node. Trying to figure it out from a while, but unsuccessful. Any help will be appreciated.

Thanks, Fatema.

The error which I am getting -

16 [main] INFO me.prettyprint.cassandra.connection.CassandraHostRetryService - Downed Host Retry service started with queue size -1 and retry delay 10s 44 [main] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - SSL enabled for client<->server communications. 45 [main] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - Properties: 45 [main] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.truststore = /home/fatemabw/.truststore 45 [main] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.protocol = TLS 45 [main] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.store.type = JKS 45 [main] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.cipher.suites = TLS_RSA_WITH_AES_128_CBC_SHA 149 [main] ERROR me.prettyprint.cassandra.connection.HConnectionManager - Could not start connection pool for host 192.168.73.142(192.168.73.142):9160 149 [main] INFO me.prettyprint.cassandra.connection.CassandraHostRetryService - Host detected as down was added to retry queue: 192.168.73.142(192.168.73.142):9160 151 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - SSL enabled for client<->server communications. 151 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - Properties: 151 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.truststore = /home/fatemabw/.truststore 151 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.protocol = TLS 151 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.store.type = JKS 151 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] INFO me.prettyprint.cassandra.connection.factory.HThriftClientFactoryImpl - ssl.cipher.suites = TLS_RSA_WITH_AES_128_CBC_SHA 152 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] WARN me.prettyprint.cassandra.connection.CassandraHostRetryService - Downed 192.168.73.142(192.168.73.142):9160 host still appears to be down: Could not get client socket: 239 [main] INFO me.prettyprint.cassandra.service.JmxMonitor - Registering JMX me.prettyprint.cassandra.service_Test Cluster:ServiceType=hector,MonitorType=hector Exception in thread "main" me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client. at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:394) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249) at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113) at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243) at me.prettyprint.cassandra.model.MutatorImpl.insert(MutatorImpl.java:69) at cassandra.Encypted_client_insert.insert(Encypted_client_insert.java:45) at cassandra.Encypted_client_insert.main(Encypted_client_insert.java:85)