search

mesosphere-backup / dcos-cassandra-service

DEPRECATED—Open source Apache Cassandra running on DC/OS is now replaced by mesosphere/dcos-commons/frameworks/cassandra. This repository will be deleted at the end of 2017.
Apache License 2.0
116 stars 54 forks source link

Node replacement failed #121

Closed brndnmtthws closed 7 years ago

brndnmtthws commented 8 years ago

I attempted to replace a node and got the following:

cat: /etc/ld.so.conf.d/*.conf: No such file or directory
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file apache-cassandra-2.2.5/bin/../logs/gc.log due to No such file or directory

java.lang.RuntimeException: A node required to move the data consistently is down (/10.0.1.122). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false
    at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:264)
    at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:147)
    at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:82)
    at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1230)
    at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:924)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:709)
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)

and:

CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I
CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
INFO  18:59:37 Loading settings from file:/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/apache-cassandra-2.2.5/conf/cassandra.yaml
INFO  18:59:37 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=false; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=cassandra; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=16; concurrent_reads=16; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_user_defined_functions=false; endpoint_snitch=GossipingPropertyFileSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.0.2.66; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_validity_in_ms=2000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=10.0.2.66; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/saved_caches; seed_provider=[{class_name=com.mesosphere.dcos.cassandra.DcosSeedProvider, parameters=[{seeds_url=http://cassandra.marathon.mesos:9000/v1/seeds}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  18:59:37 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  18:59:37 Global memtable on-heap threshold is enabled at 509MB
INFO  18:59:37 Global memtable off-heap threshold is enabled at 509MB
INFO  18:59:37 Unable to load cassandra-topology.properties; compatibility mode disabled
WARN  18:59:37 Only 28482 MB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  18:59:37 Retrieved response {"isSeed":false,"seeds":["10.0.1.121","10.0.1.123"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds
INFO  18:59:37 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]
INFO  18:59:37 Hostname: ip-10-0-2-66.us-west-2.compute.internal
INFO  18:59:37 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.8.0_74
INFO  18:59:37 Heap size: 2136997888/2136997888
INFO  18:59:37 Code Cache Non-heap memory: init = 2555904(2496K) used = 3151552(3077K) committed = 3211264(3136K) max = 251658240(245760K)
INFO  18:59:37 Metaspace Non-heap memory: init = 0(0K) used = 13007080(12702K) committed = 13238272(12928K) max = -1(-1K)
INFO  18:59:37 Compressed Class Space Non-heap memory: init = 0(0K) used = 1637264(1598K) committed = 1703936(1664K) max = 1073741824(1048576K)
INFO  18:59:37 Par Eden Space Heap memory: init = 83886080(81920K) used = 62144072(60687K) committed = 83886080(81920K) max = 83886080(81920K)
INFO  18:59:37 Par Survivor Space Heap memory: init = 10485760(10240K) used = 0(0K) committed = 10485760(10240K) max = 10485760(10240K)
INFO  18:59:37 CMS Old Gen Heap memory: init = 2042626048(1994752K) used = 0(0K) committed = 2042626048(1994752K) max = 2042626048(1994752K)
INFO  18:59:37 Classpath: apache-cassandra-2.2.5/bin/../conf:apache-cassandra-2.2.5/bin/../build/classes/main:apache-cassandra-2.2.5/bin/../build/classes/thrift:apache-cassandra-2.2.5/bin/../lib/ST4-4.0.8.jar:apache-cassandra-2.2.5/bin/../lib/airline-0.6.jar:apache-cassandra-2.2.5/bin/../lib/antlr-runtime-3.5.2.jar:apache-cassandra-2.2.5/bin/../lib/apache-cassandra-2.2.5.jar:apache-cassandra-2.2.5/bin/../lib/apache-cassandra-clientutil-2.2.5.jar:apache-cassandra-2.2.5/bin/../lib/apache-cassandra-thrift-2.2.5.jar:apache-cassandra-2.2.5/bin/../lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:apache-cassandra-2.2.5/bin/../lib/commons-cli-1.1.jar:apache-cassandra-2.2.5/bin/../lib/commons-codec-1.2.jar:apache-cassandra-2.2.5/bin/../lib/commons-lang3-3.1.jar:apache-cassandra-2.2.5/bin/../lib/commons-math3-3.2.jar:apache-cassandra-2.2.5/bin/../lib/compress-lzf-0.8.4.jar:apache-cassandra-2.2.5/bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:apache-cassandra-2.2.5/bin/../lib/crc32ex-0.1.1.jar:apache-cassandra-2.2.5/bin/../lib/disruptor-3.0.1.jar:apache-cassandra-2.2.5/bin/../lib/ecj-4.4.2.jar:apache-cassandra-2.2.5/bin/../lib/guava-16.0.jar:apache-cassandra-2.2.5/bin/../lib/high-scale-lib-1.0.6.jar:apache-cassandra-2.2.5/bin/../lib/jackson-core-asl-1.9.2.jar:apache-cassandra-2.2.5/bin/../lib/jackson-mapper-asl-1.9.2.jar:apache-cassandra-2.2.5/bin/../lib/jamm-0.3.0.jar:apache-cassandra-2.2.5/bin/../lib/javax.inject.jar:apache-cassandra-2.2.5/bin/../lib/jbcrypt-0.3m.jar:apache-cassandra-2.2.5/bin/../lib/jcl-over-slf4j-1.7.7.jar:apache-cassandra-2.2.5/bin/../lib/jna-4.0.0.jar:apache-cassandra-2.2.5/bin/../lib/joda-time-2.4.jar:apache-cassandra-2.2.5/bin/../lib/json-simple-1.1.jar:apache-cassandra-2.2.5/bin/../lib/libthrift-0.9.2.jar:apache-cassandra-2.2.5/bin/../lib/log4j-over-slf4j-1.7.7.jar:apache-cassandra-2.2.5/bin/../lib/logback-classic-1.1.3.jar:apache-cassandra-2.2.5/bin/../lib/logback-core-1.1.3.jar:apache-cassandra-2.2.5/bin/../lib/lz4-1.3.0.jar:apache-cassandra-2.2.5/bin/../lib/metrics-core-3.1.0.jar:apache-cassandra-2.2.5/bin/../lib/metrics-logback-3.1.0.jar:apache-cassandra-2.2.5/bin/../lib/metrics-statsd-common-4.1.2.jar:apache-cassandra-2.2.5/bin/../lib/metrics2-statsd-4.1.2.jar:apache-cassandra-2.2.5/bin/../lib/netty-all-4.0.23.Final.jar:apache-cassandra-2.2.5/bin/../lib/ohc-core-0.3.4.jar:apache-cassandra-2.2.5/bin/../lib/ohc-core-j8-0.3.4.jar:apache-cassandra-2.2.5/bin/../lib/reporter-config-base-3.0.0.jar:apache-cassandra-2.2.5/bin/../lib/reporter-config3-3.0.0.jar:apache-cassandra-2.2.5/bin/../lib/seedprovider-0.1.0.jar:apache-cassandra-2.2.5/bin/../lib/sigar-1.6.4.jar:apache-cassandra-2.2.5/bin/../lib/slf4j-api-1.7.7.jar:apache-cassandra-2.2.5/bin/../lib/snakeyaml-1.11.jar:apache-cassandra-2.2.5/bin/../lib/snappy-java-1.1.1.7.jar:apache-cassandra-2.2.5/bin/../lib/stream-2.5.2.jar:apache-cassandra-2.2.5/bin/../lib/super-csv-2.1.0.jar:apache-cassandra-2.2.5/bin/../lib/thrift-server-0.3.7.jar:apache-cassandra-2.2.5/bin/../lib/jsr223/*/*.jar:apache-cassandra-2.2.5/bin/../lib/jamm-0.3.0.jar
INFO  18:59:37 JVM Arguments: [-ea, -javaagent:apache-cassandra-2.2.5/bin/../lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms2048M, -Xmx2048M, -Xmn100M, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=apache-cassandra-2.2.5/bin/../conf/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -XX:+UseCondCardMark, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -Xloggc:apache-cassandra-2.2.5/bin/../logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=apache-cassandra-2.2.5/bin/../lib/sigar-bin, -Dcassandra.metricsReporterConfigFile=metrics-reporter-config.yaml, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=apache-cassandra-2.2.5/bin/../logs, -Dcassandra.storagedir=apache-cassandra-2.2.5/bin/../data, -Dcassandra-foreground=yes]
INFO  18:59:38 JNA mlockall successful
WARN  18:59:38 jemalloc shared library could not be preloaded to speed up memory allocations
WARN  18:59:38 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
INFO  18:59:38 Initializing SIGAR library
INFO  18:59:38 Checked OS settings and found them configured for optimal performance.
INFO  18:59:39 Initializing system.sstable_activity
INFO  18:59:41 Initializing key cache with capacity of 100 MBs.
INFO  18:59:41 Initializing row cache with capacity of 0 MBs
INFO  18:59:41 Initializing counter cache with capacity of 50 MBs
INFO  18:59:41 Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  18:59:41 Initializing system.hints
INFO  18:59:41 Initializing system.compaction_history
INFO  18:59:41 Initializing system.peers
INFO  18:59:41 Initializing system.schema_columnfamilies
INFO  18:59:41 Initializing system.schema_functions
INFO  18:59:41 Initializing system.IndexInfo
INFO  18:59:41 Initializing system.schema_columns
INFO  18:59:41 Initializing system.schema_triggers
INFO  18:59:41 Initializing system.local
INFO  18:59:41 Initializing system.schema_usertypes
INFO  18:59:41 Initializing system.batchlog
INFO  18:59:41 Initializing system.available_ranges
INFO  18:59:42 Initializing system.schema_aggregates
INFO  18:59:42 Initializing system.paxos
INFO  18:59:42 Initializing system.peer_events
INFO  18:59:42 Initializing system.size_estimates
INFO  18:59:42 Initializing system.compactions_in_progress
INFO  18:59:42 Initializing system.schema_keyspaces
INFO  18:59:42 Initializing system.range_xfers
INFO  18:59:43 Initializing system_distributed.parent_repair_history
INFO  18:59:43 Initializing system_distributed.repair_history
INFO  18:59:43 Initializing system_auth.role_permissions
INFO  18:59:43 Initializing system_auth.resource_role_permissons_index
INFO  18:59:43 Initializing system_auth.roles
INFO  18:59:43 Initializing system_auth.role_members
INFO  18:59:43 Initializing system_traces.sessions
INFO  18:59:43 Initializing system_traces.events
INFO  18:59:43 Completed loading (104 ms; 7 keys) KeyCache cache
INFO  18:59:43 Replaying /var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/commitlog/CommitLog-5-1462561114318.log, /var/lib/mesos/slave/slaves/20aac1e4-622d-461c-abb5-d99d9ba55ec7-S6/frameworks/20aac1e4-622d-461c-abb5-d99d9ba55ec7-0001/executors/node-2_9e70ce50-098d-4b87-85ed-c2f99ee0386a_executor/runs/64ea900c-1bfa-4393-b18c-447a7b2666b1/volume/commitlog/CommitLog-5-1462561114319.log
INFO  18:59:46 Log replay complete, 15 replayed mutations
INFO  18:59:46 Cassandra version: 2.2.5
INFO  18:59:46 Thrift API version: 20.1.0
INFO  18:59:46 CQL supported versions: 3.3.1 (default: 3.3.1)
INFO  18:59:47 Initializing index summary manager with a memory pool size of 101 MB and a resize interval of 60 minutes
INFO  18:59:47 Loading persisted ring state
INFO  18:59:47 Retrieved response {"isSeed":false,"seeds":["10.0.1.121","10.0.1.123"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds
INFO  18:59:47 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]
INFO  18:59:47 Starting Messaging Service on /10.0.2.66:7000 (eth0)
INFO  18:59:47 Retrieved response {"isSeed":false,"seeds":["10.0.1.121","10.0.1.123"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds
INFO  18:59:47 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]
INFO  18:59:47 Handshaking version with /10.0.1.123
INFO  18:59:47 Handshaking version with /10.0.1.121
INFO  18:59:52 Node /10.0.1.121 has restarted, now UP
INFO  18:59:52 Node /10.0.1.122 has restarted, now UP
INFO  18:59:52 InetAddress /10.0.1.122 is now DOWN
INFO  18:59:52 Node /10.0.1.123 has restarted, now UP
INFO  18:59:53 Starting up server gossip
INFO  18:59:54 Retrieved response {"isSeed":false,"seeds":["10.0.1.121","10.0.1.123"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds
INFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]
INFO  18:59:54 Retrieved response {"isSeed":false,"seeds":["10.0.1.121","10.0.1.123"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds
INFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]
INFO  18:59:54 Retrieved response {"isSeed":false,"seeds":["10.0.1.121","10.0.1.123"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds
INFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]
INFO  18:59:54 Retrieved response {"isSeed":false,"seeds":["10.0.1.121","10.0.1.123"]} from URL http://cassandra.marathon.mesos:9000/v1/seeds
INFO  18:59:54 Retrieved remote seeds [/10.0.1.121, /10.0.1.123]
WARN  18:59:54 Detected previous bootstrap failure; retrying
INFO  18:59:54 JOINING: waiting for ring information
INFO  18:59:54 JOINING: schema complete, ready to bootstrap
INFO  18:59:54 JOINING: waiting for pending range calculation
INFO  18:59:54 JOINING: calculation complete, ready to bootstrap
INFO  18:59:54 JOINING: getting bootstrap token
INFO  18:59:55 Handshaking version with /10.0.1.121
INFO  18:59:55 Node /10.0.1.121 is now part of the cluster
INFO  18:59:55 Node /10.0.1.121 state jump to NORMAL
INFO  18:59:56 JOINING: sleeping 30000 ms for pending range setup
WARN  18:59:56 Not marking nodes down due to local pause of 16496601209 > 5000000000
INFO  18:59:56 Updating topology for /10.0.1.121
INFO  18:59:56 Updating topology for /10.0.1.121
INFO  18:59:56 Node /10.0.1.122 is now part of the cluster
INFO  18:59:56 Node /10.0.1.122 state jump to shutdown
INFO  18:59:56 Updating topology for /10.0.1.122
INFO  18:59:56 Updating topology for /10.0.1.122
INFO  18:59:56 InetAddress /10.0.1.122 is now DOWN
INFO  18:59:56 Node /10.0.1.123 is now part of the cluster
INFO  18:59:56 Node /10.0.1.123 state jump to NORMAL
INFO  18:59:56 Updating topology for /10.0.1.123
INFO  18:59:56 Updating topology for /10.0.1.123
INFO  18:59:56 InetAddress /10.0.1.121 is now UP
INFO  18:59:56 InetAddress /10.0.1.121 is now UP
INFO  18:59:56 Handshaking version with /10.0.1.123
INFO  18:59:56 InetAddress /10.0.1.123 is now UP
INFO  19:00:26 JOINING: Starting to bootstrap...
Exception (java.lang.RuntimeException) encountered during startup: A node required to move the data consistently is down (/10.0.1.122). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false
ERROR 19:00:26 Exception encountered during startup
java.lang.RuntimeException: A node required to move the data consistently is down (/10.0.1.122). If you wish to move the data from a potentially inconsistent replica, restart the node with -Dcassandra.consistent.rangemovement=false
    at org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:264) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:147) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:82) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1230) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:924) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:709) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300) [apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [apache-cassandra-2.2.5.jar:2.2.5]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) [apache-cassandra-2.2.5.jar:2.2.5]
WARN  19:00:26 No local state or state is in silent shutdown, not announcing shutdown
erikriverson commented 8 years ago

@brndnmtthws Were you ever able to make progress with this? We are seeing same error message.

brndnmtthws commented 8 years ago

I ended up just recreating the cluster. So no, I didn't make progress.

verma7 commented 7 years ago

@brndnmtthws @erikriverson The fix we found is to do nodetool removenode <host_uuid>. You can get the <host_uuid> from nodetool status.

sdyrotronc commented 6 years ago

@verma7 how do you use nodetool? I can't find anything in the DCOS documentation on this.

nickbp commented 6 years ago

The code for this service has moved to a different repo.

Here's an example of invoking nodetool from within the nodes themselves using dcos task exec. In practice you'd probably want to do this from a separate client machine but this works in a pinch: https://github.com/mesosphere/dcos-commons/blob/master/frameworks/cassandra/tests/nodetool.py#L4

TLDR: You'd want to run something like dcos task exec TASK_ID bash -c 'JAVA_HOME=$(ls -d jdk*/jre/) apache-cassandra-*/bin/nodetool COMMAND'