Multi-DC support & Cassandra: how to set DC name for DCAwareRoundRobinPolicy?

[jpotter]

We're running cassandra in multiple data centers, where US-East is 10.1.x.x and US-West-2 is 10.3.x.x. Cassandra itself communicates cross-DC using public IP addresses, but apps in each DC should only be connecting to the local DCs cassandra instances.

When starting up cyanite, I'm getting the following error where cyanite's cassandra client is somehow determining it's in "us-west-2", even though these particular logs are coming from us-east.

How can I set the DC name for DCAwareRoundRobinPolicy?

Thanks, Jeff

DEBUG [2014-11-17 19:56:02,632] main - io.cyanite.config - building :store with io.cyanite.store/cassandra-metric-store INFO [2014-11-17 19:56:02,635] main - io.cyanite.store - creating cassandra metric store INFO [2014-11-17 19:56:06,649] main - com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'us-west-2' for DCAwareRoundRobinPolicy (if this is incorrect, pl ease provide the correct datacenter name with DCAwareRoundRobinPolicy constructor) INFO [2014-11-17 19:56:06,670] Cassandra Java Driver worker-0 - com.datastax.driver.core.Cluster - New Cassandra host /10.3.2.237:9042 added INFO [2014-11-17 19:56:06,693] Cassandra Java Driver worker-1 - com.datastax.driver.core.Cluster - New Cassandra host /10.1.0.94:9042 added INFO [2014-11-17 19:56:06,704] Cassandra Java Driver worker-1 - com.datastax.driver.core.Cluster - New Cassandra host /10.1.2.37:9042 added INFO [2014-11-17 19:56:06,707] Cassandra Java Driver worker-1 - com.datastax.driver.core.Cluster - New Cassandra host /10.3.1.229:9042 added ERROR [2014-11-17 19:56:11,794] Cassandra Java Driver blocking tasks worker-0 - com.datastax.driver.core.Session - Error creating pool to /10.3.1.229:9042 com.datastax.driver.core.TransportException: [/10.3.1.229:9042] Cannot connect at com.datastax.driver.core.Connection.(Connection.java:106)

[meltser]

Cluster.Builder builder = Cluster.builder(); builder.withLoadBalancingPolicy(new TokenAwarePolicy(new DCAwareRoundRobinPolicy("us-east"))); ....

[jpotter]

Thanks, Igor -- yes, that looks like the right java code for this. I'll see if I can work it into the cyanite codebase, but there presumably should be a setting option in cyanite to pass that through to the cassandra client?

[pyr]

@jpotter definitely. the options are forwarded almost as is to alia, so it should be possible

[mpenet]

sure, it is possible, see https://github.com/mpenet/alia/blob/master/src/qbits/alia/cluster_options.clj#L29-L31 https://github.com/mpenet/alia/blob/master/src/qbits/alia/policy/load_balancing.clj

that would look like this in the cluster call:

(alia/cluster {:load-balancing-policy (load-balancing/dc-aware-round-robin-policy "us-east"), ...})

if that's not sugarified enough you can extend the multimethod with your own option

[chjohnst]

Also seeing an issue here where I have two datacenter's running in a single cluster and cyanite just seems to hang around right after it connects to the all the nodes in the DC and then finally the nodes in the other DC. I am going to bump my version to the latest master branch but I suspect cyanite is not support in this type of topology?

[chjohnst]

BTW this is on Datastax Enterprise 4.6.

TRACE [2015-02-05 07:48:10,584] main - com.datastax.driver.core.RequestHandler - Querying node /10.250.0.6:9042 TRACE [2015-02-05 07:48:10,584] main - com.datastax.driver.core.Connection - [/10.250.0.6:9042-1] writing request PREPARE SELECT path,data,time FROM metric WHERE path IN ? AND tenant = '' AND rollup = ? AND period = ? AND time >= ? AND time <= ? ORDER BY time ASC; TRACE [2015-02-05 07:48:10,584] New I/O worker #6 - com.datastax.driver.core.Connection - [/10.250.0.6:9042-1] request sent successfully TRACE [2015-02-05 07:48:10,586] New I/O worker #6 - com.datastax.driver.core.Connection - [/10.250.0.6:9042-1] received: ERROR INVALID: ORDER BY could not be used on columns missing in select clause.

[jpotter]

ORDER BY is broken in DSE 4.6. I have an open ticket with them; they say it might be fixed in 4.6.1 due out in the next few weeks, but it sounds like the fix is in a newer dot release of cassandra than what will be in 4.6.1 so don't hold your breath.

The issue in this bug isn't relegated to the ORDER BY bug.

[chjohnst]

Found the regression that is present in 4.6 and just recently fixed in 4.5.5 and newer upstream Cassandra 2.x. We were thinking the ordering of the select statement might help but not 100% sure.

https://issues.apache.org/jira/browse/CASSANDRA-8286

[jpotter]

Yup, that's the one!

[jpotter]

I'm told that DSE 4.6.1 will have C* 2.0.12 which contains the fix for CASSANDRA-8286 and 4.6.1 should be out in about 3 weeks from now.

On Feb 5, 2015, at 7:49 AM, Christopher Johnston notifications@github.com wrote:

BTW this is on Datastax Enterprise 4.6.

TRACE [2015-02-05 07:48:10,584] main - com.datastax.driver.core.RequestHandler - Querying node /10.250.0.6:9042 TRACE [2015-02-05 07:48:10,584] main - com.datastax.driver.core.Connection - [/10.250.0.6:9042-1] writing request PREPARE SELECT path,data,time FROM metric WHERE path IN ? AND tenant = '' AND rollup = ? AND period = ? AND time >= ? AND time <= ? ORDER BY time ASC; TRACE [2015-02-05 07:48:10,584] New I/O worker #6 - com.datastax.driver.core.Connection - [/10.250.0.6:9042-1] request sent successfully TRACE [2015-02-05 07:48:10,586] New I/O worker #6 - com.datastax.driver.core.Connection - [/10.250.0.6:9042-1] received: ERROR INVALID: ORDER BY could not be used on columns missing in select clause.

— Reply to this email directly or view it on GitHub.

[pyr]

Thanks for the details @jpotter !