search

Netflix / Priam

Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
Apache License 2.0
1.04k stars 294 forks source link

v1/cassconfig/double_ring api call creating new token larger than 2**127 #362

Closed gaahrdner closed 9 years ago

gaahrdner commented 10 years ago

Hi y'all, kind of at a loss here. Using Priam v2.0.8.39 with DataStax Enterprise 4.5.1 and have been trying to leverage Priam's great capabilities as a management/admin tool. I've written a script that calls the double_ring endpoint on a cluster, and then will double the size of the auto scaling group for a cassandra cluster. Without fail, one newly created node will be unable to start up, as the token will be too large. Here's a log snippet from the failing node:

INFO [main] 2014-09-29 13:04:43,629 YamlConfigurationLoader.java (line 76) Loading settings from file:/etc/dse/cassandra/cassandra.yaml 
INFO [main] 2014-09-29 13:04:43,741 DatabaseDescriptor.java (line 142) Data files directories: [/vol00/cassandra/data] 
INFO [main] 2014-09-29 13:04:43,741 DatabaseDescriptor.java (line 143) Commit log directory: /vol00/cassandra/commitlog 
INFO [main] 2014-09-29 13:04:43,741 DatabaseDescriptor.java (line 183) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap 
INFO [main] 2014-09-29 13:04:43,742 DatabaseDescriptor.java (line 197) disk_failure_policy is stop 
INFO [main] 2014-09-29 13:04:43,742 DatabaseDescriptor.java (line 198) commit_failure_policy is stop 
INFO [main] 2014-09-29 13:04:43,749 DatabaseDescriptor.java (line 268) Global memtable threshold is enabled at 1024MB 
INFO [main] 2014-09-29 13:04:43,780 Ec2Snitch.java (line 67) EC2Snitch using region: us-west-2, zone: 2c. 
INFO [main] 2014-09-29 13:04:43,781 Ec2MultiRegionSnitch.java (line 51) EC2Snitch using publicIP as identifier: /54.69.161.164 
INFO [main] 2014-09-29 13:04:44,022 DatabaseDescriptor.java (line 401) Not using multi-threaded compaction 
ERROR [main] 2014-09-29 13:04:44,025 DatabaseDescriptor.java (line 116) Fatal configuration error org.apache.cassandra.exceptions.ConfigurationException: Token must be <= 2**127 
    at org.apache.cassandra.dht.RandomPartitioner$1.validate(RandomPartitioner.java:95) 
    at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:440) 
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:111) 
    at com.datastax.bdp.config.DseConfig.<clinit>(DseConfig.java:204) 
    at com.datastax.bdp.cassandra.metrics.UserObjectLatencyPlugin.isEnabled(UserObjectLatencyPlugin.java:114) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.commons.lang3.reflect.MethodUtils.invokeExactStaticMethod(MethodUtils.java:241) 
    at org.apache.commons.lang3.reflect.MethodUtils.invokeExactStaticMethod(MethodUtils.java:352) 
    at com.datastax.bdp.plugin.PluginManager.isPluginEnabled(PluginManager.java:94) 
    at com.datastax.bdp.plugin.PluginManager.registerPluginWithDeps(PluginManager.java:61) 
    at com.datastax.bdp.plugin.PluginManager.registerAllPlugins(PluginManager.java:52) 
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:392) 
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) 
    at com.datastax.bdp.server.DseDaemon.main(DseDaemon.java:574) 
INFO [Thread-1] 2014-09-29 13:04:44,028 DseDaemon.java (line 477) DSE shutting down...

This happens without fail, regardless of the size of the starting cluster or it's data store size, so the new cluster is always coming up unbalanced, i.e. from 3 -> 5 nodes instead of 6.

Sadly I've no real idea how to fix this; I can confirm the offending token is generated by Priam and placed into SimpleDB, so in my mind the problem lies in either how Priam initially generates the new tokens or DataStax's Cassandra version is mucking with the aforementioned calculation.

Thoughts? I'd love to get this resolved and am happy to help as I can.

timiblossom commented 10 years ago

How did you double your cluster? Why did you use your own script if Priam has the REST API for doubling? There is a code in Priam that has the logic to generate the tokens and you can read that to double-check on the token list. The code is located at:

priam/src/main/java/com/netflix/priam/utils/TokenManager.java

gaahrdner commented 10 years ago

Hi @timiblossom, as I mentioned, the script I wrote just calls the Priam REST API on one cluster member, and then makes an API call to AWS to enlarge the auto-scaling group that the cluster resides in. Is this not the correct way to double a cluster? I'm pretty sure I gleaned it off of some mailing list that I can not currently locate, sorry.

timiblossom commented 10 years ago

@gaahrdner here is the line where we compute the tokens:

https://github.com/Netflix/Priam/blob/master/priam/src/main/java/com/netflix/priam/utils/TokenManager.java#L39

You can read the code and take a look at the Priam's generated tokens. According to the code, a token can't never be greater than 2^127.

gaahrdner commented 9 years ago

I don't work for the company that was using this anymore, so I'm gonna close this issue.