Kong / docker-cassandra

Docker Cassandra 2.1.8
5 stars 6 forks source link

my cassandra docker stops without warning #2

Closed sloev closed 8 years ago

sloev commented 8 years ago

my cassandra docker container stops without warning. i have the container running alone without connection to kong and it just stops after initializing.

What can the problem be? im totally confused!

here is my log

INFO 07:22:35 Initializing system.schema_triggers INFO 07:22:37 Opening /var/lib/cassandra/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/system-schema_triggers-ka-1 (32 bytes) INFO 07:22:37 Initializing system.compaction_history INFO 07:22:37 Opening /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-1 (4529 bytes) INFO 07:22:37 Initializing system.batchlog INFO 07:22:37 Initializing system.sstable_activity INFO 07:22:37 Opening /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-1 (2564 bytes) INFO 07:22:37 Initializing system.peer_events INFO 07:22:37 Initializing system.compactions_in_progress INFO 07:22:37 Initializing system.hints INFO 07:22:37 Initializing system.schema_keyspaces INFO 07:22:37 Opening /var/lib/cassandra/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-53 (251 bytes) INFO 07:22:37 Opening /var/lib/cassandra/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-54 (156 bytes) INFO 07:22:37 reading saved cache /var/lib/cassandra/saved_caches/system-schema_keyspaces-b0f2235744583cdb9631c43e59ce3676-KeyCache-b.db INFO 07:22:37 Initializing system.range_xfers INFO 07:22:37 Initializing system.schema_columnfamilies INFO 07:22:37 Opening /var/lib/cassandra/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-53 (12216 bytes) INFO 07:22:37 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697-KeyCache-b.db INFO 07:22:37 Initializing system.paxos INFO 07:22:37 Initializing system.schema_columns INFO 07:22:37 Opening /var/lib/cassandra/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-53 (18955 bytes) INFO 07:22:37 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columns-296e9c049bec3085827dc17d3df2122a-KeyCache-b.db INFO 07:22:37 Initializing system.schema_usertypes INFO 07:22:37 Opening /var/lib/cassandra/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1 (32 bytes) INFO 07:22:37 Initializing system.IndexInfo INFO 07:22:37 Opening /var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/system-IndexInfo-ka-33 (623 bytes) INFO 07:22:37 Opening /var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/system-IndexInfo-ka-34 (77 bytes) INFO 07:22:37 reading saved cache /var/lib/cassandra/saved_caches/system-IndexInfo-9f5c6374d48532299a0a5094af9ad1e3-KeyCache-b.db INFO 07:22:37 Initializing system.peers INFO 07:22:37 Opening /var/lib/cassandra/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-1 (30 bytes) INFO 07:22:37 Initializing system.size_estimates INFO 07:22:37 Opening /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-5 (182079 bytes) INFO 07:22:37 Initializing system.local INFO 07:22:37 Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-7 (139 bytes) INFO 07:22:37 Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-5 (5726 bytes) INFO 07:22:37 Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-6 (80 bytes) INFO 07:22:37 reading saved cache /var/lib/cassandra/saved_caches/system-local-7ad54392bcdd35a684174e047860b377-KeyCache-b.db INFO 07:22:38 Enqueuing flush of local: 865 (0%) on-heap, 0 (0%) off-heap INFO 07:22:38 Writing Memtable-local@636237220(0.162KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit) INFO 07:22:38 Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-tmp-ka-8-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1444029756991, position=279) INFO 07:22:38 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-7-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-5-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-8-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-6-Data.db')] INFO 07:22:38 Initializing kong.oauth2_credentials INFO 07:22:38 Initializing kong.oauth2_credentials.oauth2_credentials_client_secret_idx INFO 07:22:38 Initializing kong.oauth2_credentials.oauth2_credentials_consumer_id_idx INFO 07:22:38 Initializing kong.oauth2_credentials.oauth2_credentials_client_id_idx INFO 07:22:38 Initializing kong.response_ratelimiting_metrics INFO 07:22:38 Initializing kong.apis INFO 07:22:38 Initializing kong.apis.apis_request_host_idx INFO 07:22:38 Initializing kong.apis.apis_request_path_idx INFO 07:22:38 Initializing kong.apis.apis_name_idx INFO 07:22:39 Initializing kong.oauth2_authorization_codes INFO 07:22:39 Initializing kong.oauth2_authorization_codes.oauth2_authorization_codes_authenticated_userid_idx INFO 07:22:39 Initializing kong.oauth2_authorization_codes.oauth2_authorization_codes_code_idx INFO 07:22:39 Initializing kong.plugins INFO 07:22:39 Initializing kong.plugins.plugins_consumer_id_idx INFO 07:22:39 Initializing kong.plugins.plugins_api_id_idx INFO 07:22:39 Initializing kong.plugins.plugins_name_idx INFO 07:22:39 Initializing kong.schema_migrations INFO 07:22:39 Initializing kong.consumers INFO 07:22:39 Initializing kong.consumers.consumers_username_idx INFO 07:22:39 Initializing kong.consumers.consumers_custom_id_idx INFO 07:22:39 Initializing kong.acls INFO 07:22:39 Initializing kong.acls.acls_group_idx INFO 07:22:39 Initializing kong.acls.acls_consumer_id INFO 07:22:39 Initializing kong.keyauth_credentials INFO 07:22:39 Initializing kong.keyauth_credentials.keyauth_credentials_key_idx INFO 07:22:39 Compacted 4 sstables to [/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-9,]. 6,084 bytes to 5,708 (~93% of original) in 365ms = 0.014914MB/s. 4 total partitions merged to 1. Partition merge counts were {4:1, } INFO 07:22:39 Initializing kong.keyauth_credentials.keyauth_consumer_id INFO 07:22:39 Initializing kong.basicauth_credentials INFO 07:22:39 Initializing kong.basicauth_credentials.basicauth_consumer_id INFO 07:22:39 Initializing kong.basicauth_credentials.basicauth_credentials_username_idx INFO 07:22:39 Initializing kong.oauth2_tokens INFO 07:22:39 Initializing kong.oauth2_tokens.oauth2_tokens_refresh_token_idx INFO 07:22:39 Initializing kong.oauth2_tokens.oauth2_credential_id_idx INFO 07:22:39 Initializing kong.oauth2_tokens.oauth2_tokens_authenticated_userid_idx INFO 07:22:39 Initializing kong.oauth2_tokens.oauth2_tokens_access_token_idx INFO 07:22:39 Initializing kong.ratelimiting_metrics INFO 07:22:39 Initializing kong.jwt_secrets INFO 07:22:39 Initializing kong.jwt_secrets.jwt_secrets_secret_idx INFO 07:22:39 Initializing kong.jwt_secrets.jwt_secrets_key_idx INFO 07:22:39 Initializing kong.jwt_secrets.jwt_secrets_consumer_id_idx INFO 07:22:39 Initializing system_traces.sessions INFO 07:22:39 Initializing system_traces.events INFO 07:22:39 completed pre-loading (10 keys) key cache. INFO 07:22:39 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log, /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log, /var/lib/cassandra/commitlog/CommitLog-4-1444029293210.log INFO 07:22:39 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log INFO 07:22:39 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log (CL version 4, messaging version 8) INFO 07:22:41 Copy GC in 241ms. CMS Old Gen: 28562480 -> 98866384; Eden Space: 83026448 -> 0; INFO 07:22:42 Copy GC in 256ms. CMS Old Gen: 98866384 -> 166250400; Eden Space: 83886080 -> 0; Survivor Space: 10485760 -> 10485752 INFO 07:22:42 Finished reading /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log INFO 07:22:42 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log INFO 07:22:42 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log (CL version 4, messaging version 8) INFO 07:22:43 Copy GC in 1149ms. CMS Old Gen: 166250400 -> 232092144; Eden Space: 83886080 -> 0; Survivor Space: 10485752 -> 10485760 INFO 07:22:44 Pool Name Active Pending Completed Blocked All Time Blocked INFO 07:22:44 CounterMutationStage 0 0 0 0 0 INFO 07:22:44 ReadStage 0 0 0 0 0 INFO 07:22:44 RequestResponseStage 0 0 0 0 0 INFO 07:22:44 ReadRepairStage 0 0 0 0 0 INFO 07:22:44 MutationStage 0 0 950 0 0 INFO 07:22:44 GossipStage 0 0 0 0 0 INFO 07:22:44 AntiEntropyStage 0 0 0 0 0 INFO 07:22:44 CacheCleanupExecutor 0 0 0 0 0 INFO 07:22:44 MigrationStage 0 0 0 0 0 INFO 07:22:44 Sampler 0 0 0 0 0 INFO 07:22:44 ValidationExecutor 0 0 0 0 0 INFO 07:22:44 MiscStage 0 0 0 0 0 INFO 07:22:44 CommitLogArchiver 0 0 0 0 0 INFO 07:22:44 MemtableFlushWriter 0 0 1 0 0 INFO 07:22:44 MemtableReclaimMemory 0 0 1 0 0 INFO 07:22:44 MemtablePostFlush 0 0 3 0 0 INFO 07:22:44 CompactionExecutor 0 0 2 0 0 INFO 07:22:44 InternalResponseStage 0 0 0 0 0 INFO 07:22:44 CompactionManager 0 0 INFO 07:22:44 MessagingService n/a 0/0 INFO 07:22:44 Cache Type Size Capacity KeysToSave INFO 07:22:44 KeyCache 744 25165824 all INFO 07:22:44 RowCache 0 0 all INFO 07:22:44 ColumnFamily Memtable ops,data INFO 07:22:44 system.schema_triggers 0,0 INFO 07:22:44 system.local 0,0 INFO 07:22:44 system.peers 0,0 INFO 07:22:44 system.batchlog 0,0 INFO 07:22:44 system.size_estimates 0,0 INFO 07:22:44 system.schema_usertypes 0,0 INFO 07:22:44 system.compactions_in_progress 0,0 INFO 07:22:44 system.schema_keyspaces 0,0 INFO 07:22:44 system.compaction_history 9,1187 INFO 07:22:44 system.paxos 0,0 INFO 07:22:44 system.schema_columns 0,0 INFO 07:22:44 system.schema_columnfamilies 0,0 INFO 07:22:44 system.IndexInfo 0,0 INFO 07:22:44 system.range_xfers 0,0 INFO 07:22:44 system.peer_events 0,0 INFO 07:22:44 system.hints 0,0 INFO 07:22:44 system.sstable_activity 4,688 INFO 07:22:44 kong.acls 0,0 INFO 07:22:44 kong.keyauth_credentials 0,0 INFO 07:22:44 kong.jwt_secrets 5,916 INFO 07:22:44 kong.apis 18,2745 INFO 07:22:44 kong.oauth2_authorization_codes 0,0 INFO 07:22:44 kong.consumers 6,1154 INFO 07:22:44 kong.response_ratelimiting_metrics 0,0 INFO 07:22:44 kong.oauth2_tokens 0,0 INFO 07:22:44 kong.ratelimiting_metrics 0,0 INFO 07:22:44 kong.oauth2_credentials 0,0 INFO 07:22:44 kong.schema_migrations 10,3989 INFO 07:22:44 kong.basicauth_credentials 0,0 INFO 07:22:44 kong.plugins 6,1378 INFO 07:22:44 system_traces.sessions 0,0 INFO 07:22:44 system_traces.events 0,0 CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I INFO 07:24:26 Hostname: 6088cbad8c54 INFO 07:24:26 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml INFO 07:24:26 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_rpc_address=127.0.0.1; cas_contention_timeout_in_ms=1000; client_encryption_options=; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000] INFO 07:24:26 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 07:24:26 Global memtable on-heap threshold is enabled at 121MB INFO 07:24:26 Global memtable off-heap threshold is enabled at 121MB INFO 07:24:27 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml INFO 07:24:27 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_rpc_address=127.0.0.1; cas_contention_timeout_in_ms=1000; client_encryption_options=; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000] INFO 07:24:27 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_75 INFO 07:24:27 Heap size: 509607936/509607936 INFO 07:24:27 Code Cache Non-heap memory: init = 2555904(2496K) used = 714304(697K) committed = 2555904(2496K) max = 50331648(49152K) INFO 07:24:27 Eden Space Heap memory: init = 83886080(81920K) used = 61469704(60029K) committed = 83886080(81920K) max = 83886080(81920K) INFO 07:24:27 Survivor Space Heap memory: init = 10485760(10240K) used = 0(0K) committed = 10485760(10240K) max = 10485760(10240K) INFO 07:24:27 CMS Old Gen Heap memory: init = 415236096(405504K) used = 0(0K) committed = 415236096(405504K) max = 415236096(405504K) INFO 07:24:27 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 15576832(15211K) committed = 21757952(21248K) max = 85983232(83968K) INFO 07:24:27 Classpath: /etc/cassandra/conf:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.8.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.8.jar:/usr/share/cassandra/cassandra-driver-core-2.0.9.2.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar WARN 07:24:27 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root. WARN 07:24:27 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. INFO 07:24:27 Initializing key cache with capacity of 24 MBs. INFO 07:24:27 Initializing row cache with capacity of 0 MBs INFO 07:24:27 Initializing counter cache with capacity of 12 MBs INFO 07:24:27 Scheduling counter cache save to every 7200 seconds (going to save all keys). INFO 07:24:28 Initializing system.schema_triggers INFO 07:24:29 Opening /var/lib/cassandra/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/system-schema_triggers-ka-1 (32 bytes) INFO 07:24:29 Initializing system.compaction_history INFO 07:24:29 Opening /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-1 (4529 bytes) INFO 07:24:29 Initializing system.batchlog INFO 07:24:29 Initializing system.sstable_activity INFO 07:24:29 Opening /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-1 (2564 bytes) INFO 07:24:29 Initializing system.peer_events INFO 07:24:29 Initializing system.compactions_in_progress INFO 07:24:29 Initializing system.hints INFO 07:24:29 Initializing system.schema_keyspaces INFO 07:24:29 Opening /var/lib/cassandra/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-53 (251 bytes) INFO 07:24:29 Opening /var/lib/cassandra/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-54 (156 bytes) INFO 07:24:29 reading saved cache /var/lib/cassandra/saved_caches/system-schema_keyspaces-b0f2235744583cdb9631c43e59ce3676-KeyCache-b.db INFO 07:24:29 Initializing system.range_xfers INFO 07:24:29 Initializing system.schema_columnfamilies INFO 07:24:29 Opening /var/lib/cassandra/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-53 (12216 bytes) INFO 07:24:29 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697-KeyCache-b.db INFO 07:24:29 Initializing system.paxos INFO 07:24:29 Initializing system.schema_columns INFO 07:24:29 Opening /var/lib/cassandra/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-53 (18955 bytes) INFO 07:24:29 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columns-296e9c049bec3085827dc17d3df2122a-KeyCache-b.db INFO 07:24:29 Initializing system.schema_usertypes INFO 07:24:29 Opening /var/lib/cassandra/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1 (32 bytes) INFO 07:24:29 Initializing system.IndexInfo INFO 07:24:29 Opening /var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/system-IndexInfo-ka-33 (623 bytes) INFO 07:24:29 Opening /var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/system-IndexInfo-ka-34 (77 bytes) INFO 07:24:29 reading saved cache /var/lib/cassandra/saved_caches/system-IndexInfo-9f5c6374d48532299a0a5094af9ad1e3-KeyCache-b.db INFO 07:24:29 Initializing system.peers INFO 07:24:29 Opening /var/lib/cassandra/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-1 (30 bytes) INFO 07:24:29 Initializing system.size_estimates INFO 07:24:29 Opening /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-5 (182079 bytes) INFO 07:24:29 Initializing system.local INFO 07:24:29 Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-9 (5708 bytes) INFO 07:24:29 reading saved cache /var/lib/cassandra/saved_caches/system-local-7ad54392bcdd35a684174e047860b377-KeyCache-b.db INFO 07:24:31 Enqueuing flush of local: 865 (0%) on-heap, 0 (0%) off-heap INFO 07:24:31 Writing Memtable-local@1322057440(0.162KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit) INFO 07:24:31 Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-tmp-ka-10-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1444029869500, position=279) INFO 07:24:31 Initializing kong.oauth2_credentials INFO 07:24:31 Initializing kong.oauth2_credentials.oauth2_credentials_client_secret_idx INFO 07:24:31 Initializing kong.oauth2_credentials.oauth2_credentials_consumer_id_idx INFO 07:24:31 Initializing kong.oauth2_credentials.oauth2_credentials_client_id_idx INFO 07:24:31 Initializing kong.response_ratelimiting_metrics INFO 07:24:31 Initializing kong.apis INFO 07:24:31 Initializing kong.apis.apis_request_host_idx INFO 07:24:31 Initializing kong.apis.apis_request_path_idx INFO 07:24:31 Initializing kong.apis.apis_name_idx INFO 07:24:31 Initializing kong.oauth2_authorization_codes INFO 07:24:31 Initializing kong.oauth2_authorization_codes.oauth2_authorization_codes_authenticated_userid_idx INFO 07:24:31 Initializing kong.oauth2_authorization_codes.oauth2_authorization_codes_code_idx INFO 07:24:31 Initializing kong.plugins INFO 07:24:31 Initializing kong.plugins.plugins_consumer_id_idx INFO 07:24:31 Initializing kong.plugins.plugins_api_id_idx INFO 07:24:31 Initializing kong.plugins.plugins_name_idx INFO 07:24:31 Initializing kong.schema_migrations INFO 07:24:31 Initializing kong.consumers INFO 07:24:31 Initializing kong.consumers.consumers_username_idx INFO 07:24:31 Initializing kong.consumers.consumers_custom_id_idx INFO 07:24:31 Initializing kong.acls INFO 07:24:31 Initializing kong.acls.acls_group_idx INFO 07:24:31 Initializing kong.acls.acls_consumer_id INFO 07:24:31 Initializing kong.keyauth_credentials INFO 07:24:31 Initializing kong.keyauth_credentials.keyauth_credentials_key_idx INFO 07:24:31 Initializing kong.keyauth_credentials.keyauth_consumer_id INFO 07:24:31 Initializing kong.basicauth_credentials INFO 07:24:31 Initializing kong.basicauth_credentials.basicauth_consumer_id INFO 07:24:31 Initializing kong.basicauth_credentials.basicauth_credentials_username_idx INFO 07:24:31 Initializing kong.oauth2_tokens INFO 07:24:31 Initializing kong.oauth2_tokens.oauth2_tokens_refresh_token_idx INFO 07:24:31 Initializing kong.oauth2_tokens.oauth2_credential_id_idx INFO 07:24:31 Initializing kong.oauth2_tokens.oauth2_tokens_authenticated_userid_idx INFO 07:24:31 Initializing kong.oauth2_tokens.oauth2_tokens_access_token_idx INFO 07:24:31 Initializing kong.ratelimiting_metrics INFO 07:24:31 Initializing kong.jwt_secrets INFO 07:24:31 Initializing kong.jwt_secrets.jwt_secrets_secret_idx INFO 07:24:31 Initializing kong.jwt_secrets.jwt_secrets_key_idx INFO 07:24:31 Initializing kong.jwt_secrets.jwt_secrets_consumer_id_idx INFO 07:24:31 Initializing system_traces.sessions INFO 07:24:31 Initializing system_traces.events INFO 07:24:31 completed pre-loading (10 keys) key cache. INFO 07:24:31 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log, /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log, /var/lib/cassandra/commitlog/CommitLog-4-1444029293210.log, /var/lib/cassandra/commitlog/CommitLog-4-1444029756991.log INFO 07:24:31 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log INFO 07:24:31 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log (CL version 4, messaging version 8) INFO 07:24:34 Copy GC in 1817ms. CMS Old Gen: 40366848 -> 109544256; Eden Space: 83886080 -> 0; INFO 07:24:34 Pool Name Active Pending Completed Blocked All Time Blocked INFO 07:24:34 CounterMutationStage 0 0 0 0 0 INFO 07:24:34 ReadStage 0 0 0 0 0 INFO 07:24:34 RequestResponseStage 0 0 0 0 0 INFO 07:24:34 ReadRepairStage 0 0 0 0 0 INFO 07:24:34 MutationStage 0 0 604 0 0 INFO 07:24:34 GossipStage 0 0 0 0 0 INFO 07:24:34 AntiEntropyStage 0 0 0 0 0 INFO 07:24:34 CacheCleanupExecutor 0 0 0 0 0 INFO 07:24:34 MigrationStage 0 0 0 0 0 INFO 07:24:34 Sampler 0 0 0 0 0 INFO 07:24:34 ValidationExecutor 0 0 0 0 0 INFO 07:24:34 MiscStage 0 0 0 0 0 INFO 07:24:34 CommitLogArchiver 0 0 0 0 0 INFO 07:24:34 MemtableFlushWriter 0 0 1 0 0 INFO 07:24:34 MemtableReclaimMemory 0 0 1 0 0 INFO 07:24:34 MemtablePostFlush 0 0 3 0 0 INFO 07:24:34 CompactionExecutor 0 0 2 0 0 INFO 07:24:34 InternalResponseStage 0 0 0 0 0 INFO 07:24:34 CompactionManager 0 0 INFO 07:24:34 MessagingService n/a 0/0 INFO 07:24:34 Cache Type Size Capacity KeysToSave INFO 07:24:34 KeyCache 816 25165824 all INFO 07:24:34 RowCache 0 0 all INFO 07:24:34 ColumnFamily Memtable ops,data INFO 07:24:34 system.schema_triggers 0,0 INFO 07:24:34 system.local 0,0 INFO 07:24:34 system.peers 0,0 INFO 07:24:34 system.batchlog 0,0 INFO 07:24:34 system.size_estimates 0,0 INFO 07:24:34 system.schema_usertypes 0,0 INFO 07:24:34 system.compactions_in_progress 0,0 INFO 07:24:34 system.schema_keyspaces 0,0 INFO 07:24:34 system.compaction_history 0,0 INFO 07:24:34 system.paxos 0,0 INFO 07:24:34 system.schema_columns 0,0 INFO 07:24:34 system.schema_columnfamilies 0,0 INFO 07:24:34 system.IndexInfo 0,0 INFO 07:24:34 system.range_xfers 0,0 INFO 07:24:34 system.peer_events 0,0 INFO 07:24:34 system.hints 0,0 INFO 07:24:34 system.sstable_activity 0,0 INFO 07:24:34 kong.acls 0,0 INFO 07:24:34 kong.keyauth_credentials 0,0 INFO 07:24:34 kong.jwt_secrets 5,916 INFO 07:24:34 kong.apis 18,2805 INFO 07:24:34 kong.oauth2_authorization_codes 0,0 INFO 07:24:34 kong.consumers 6,1154 INFO 07:24:34 kong.response_ratelimiting_metrics 0,0 INFO 07:24:34 kong.oauth2_tokens 0,0 INFO 07:24:34 kong.ratelimiting_metrics 0,0 INFO 07:24:34 kong.oauth2_credentials 0,0 INFO 07:24:34 kong.schema_migrations 10,3989 INFO 07:24:34 kong.basicauth_credentials 0,0 INFO 07:24:34 kong.plugins 6,1378 INFO 07:24:34 system_traces.sessions 0,0 INFO 07:24:34 system_traces.events 0,0 INFO 07:24:35 Copy GC in 217ms. CMS Old Gen: 109544256 -> 175288592; Eden Space: 83886080 -> 0; INFO 07:24:35 Finished reading /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log INFO 07:24:35 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log INFO 07:24:35 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log (CL version 4, messaging version 8) INFO 07:24:36 Copy GC in 578ms. CMS Old Gen: 175288592 -> 242561856; Eden Space: 83886080 -> 0; CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I INFO 07:27:35 Hostname: 6088cbad8c54 INFO 07:27:35 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml INFO 07:27:36 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_rpc_address=127.0.0.1; cas_contention_timeout_in_ms=1000; client_encryption_options=; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000] INFO 07:27:36 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 07:27:36 Global memtable on-heap threshold is enabled at 121MB INFO 07:27:36 Global memtable off-heap threshold is enabled at 121MB INFO 07:27:36 Loading settings from file:/etc/cassandra/default.conf/cassandra.yaml INFO 07:27:36 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_rpc_address=127.0.0.1; cas_contention_timeout_in_ms=1000; client_encryption_options=; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000] INFO 07:27:36 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_75 INFO 07:27:36 Heap size: 509607936/509607936 INFO 07:27:36 Code Cache Non-heap memory: init = 2555904(2496K) used = 714240(697K) committed = 2555904(2496K) max = 50331648(49152K) INFO 07:27:36 Eden Space Heap memory: init = 83886080(81920K) used = 61467032(60026K) committed = 83886080(81920K) max = 83886080(81920K) INFO 07:27:36 Survivor Space Heap memory: init = 10485760(10240K) used = 0(0K) committed = 10485760(10240K) max = 10485760(10240K) INFO 07:27:36 CMS Old Gen Heap memory: init = 415236096(405504K) used = 0(0K) committed = 415236096(405504K) max = 415236096(405504K) INFO 07:27:37 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 15577360(15212K) committed = 21757952(21248K) max = 85983232(83968K) INFO 07:27:37 Classpath: /etc/cassandra/conf:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.8.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.8.jar:/usr/share/cassandra/cassandra-driver-core-2.0.9.2.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar WARN 07:27:37 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root. WARN 07:27:37 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. INFO 07:27:37 Initializing key cache with capacity of 24 MBs. INFO 07:27:37 Initializing row cache with capacity of 0 MBs INFO 07:27:37 Initializing counter cache with capacity of 12 MBs INFO 07:27:37 Scheduling counter cache save to every 7200 seconds (going to save all keys). INFO 07:27:37 Initializing system.schema_triggers INFO 07:27:38 Opening /var/lib/cassandra/data/system/schema_triggers-0359bc7171233ee19a4ab9dfb11fc125/system-schema_triggers-ka-1 (32 bytes) INFO 07:27:38 Initializing system.compaction_history INFO 07:27:38 Opening /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/system-compaction_history-ka-1 (4529 bytes) INFO 07:27:38 Initializing system.batchlog INFO 07:27:38 Initializing system.sstable_activity INFO 07:27:38 Opening /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-1 (2564 bytes) INFO 07:27:38 Initializing system.peer_events INFO 07:27:38 Initializing system.compactions_in_progress INFO 07:27:38 Initializing system.hints INFO 07:27:38 Initializing system.schema_keyspaces INFO 07:27:38 Opening /var/lib/cassandra/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-53 (251 bytes) INFO 07:27:38 Opening /var/lib/cassandra/data/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-54 (156 bytes) INFO 07:27:38 reading saved cache /var/lib/cassandra/saved_caches/system-schema_keyspaces-b0f2235744583cdb9631c43e59ce3676-KeyCache-b.db INFO 07:27:39 Initializing system.range_xfers INFO 07:27:39 Initializing system.schema_columnfamilies INFO 07:27:39 Opening /var/lib/cassandra/data/system/schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697/system-schema_columnfamilies-ka-53 (12216 bytes) INFO 07:27:39 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columnfamilies-45f5b36024bc3f83a3631034ea4fa697-KeyCache-b.db INFO 07:27:39 Initializing system.paxos INFO 07:27:39 Initializing system.schema_columns INFO 07:27:39 Opening /var/lib/cassandra/data/system/schema_columns-296e9c049bec3085827dc17d3df2122a/system-schema_columns-ka-53 (18955 bytes) INFO 07:27:39 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columns-296e9c049bec3085827dc17d3df2122a-KeyCache-b.db INFO 07:27:39 Initializing system.schema_usertypes INFO 07:27:39 Opening /var/lib/cassandra/data/system/schema_usertypes-3aa752254f82350b8d5c430fa221fa0a/system-schema_usertypes-ka-1 (32 bytes) INFO 07:27:39 Initializing system.IndexInfo INFO 07:27:39 Opening /var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/system-IndexInfo-ka-33 (623 bytes) INFO 07:27:39 Opening /var/lib/cassandra/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/system-IndexInfo-ka-34 (77 bytes) INFO 07:27:39 reading saved cache /var/lib/cassandra/saved_caches/system-IndexInfo-9f5c6374d48532299a0a5094af9ad1e3-KeyCache-b.db INFO 07:27:39 Initializing system.peers INFO 07:27:39 Opening /var/lib/cassandra/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-1 (30 bytes) INFO 07:27:39 Initializing system.size_estimates INFO 07:27:39 Opening /var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/system-size_estimates-ka-5 (182079 bytes) INFO 07:27:39 Initializing system.local INFO 07:27:39 Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-10 (139 bytes) INFO 07:27:39 Opening /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-9 (5708 bytes) INFO 07:27:39 reading saved cache /var/lib/cassandra/saved_caches/system-local-7ad54392bcdd35a684174e047860b377-KeyCache-b.db INFO 07:27:40 Enqueuing flush of local: 865 (0%) on-heap, 0 (0%) off-heap INFO 07:27:40 Writing Memtable-local@1166861382(0.162KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit) INFO 07:27:40 Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-tmp-ka-11-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1444030058808, position=279) INFO 07:27:40 Initializing kong.oauth2_credentials INFO 07:27:41 Initializing kong.oauth2_credentials.oauth2_credentials_client_secret_idx INFO 07:27:41 Initializing kong.oauth2_credentials.oauth2_credentials_consumer_id_idx INFO 07:27:41 Initializing kong.oauth2_credentials.oauth2_credentials_client_id_idx INFO 07:27:41 Initializing kong.response_ratelimiting_metrics INFO 07:27:41 Initializing kong.apis INFO 07:27:41 Initializing kong.apis.apis_request_host_idx INFO 07:27:41 Initializing kong.apis.apis_request_path_idx INFO 07:27:41 Initializing kong.apis.apis_name_idx INFO 07:27:41 Initializing kong.oauth2_authorization_codes INFO 07:27:41 Initializing kong.oauth2_authorization_codes.oauth2_authorization_codes_authenticated_userid_idx INFO 07:27:41 Initializing kong.oauth2_authorization_codes.oauth2_authorization_codes_code_idx INFO 07:27:41 Initializing kong.plugins INFO 07:27:41 Initializing kong.plugins.plugins_consumer_id_idx INFO 07:27:41 Initializing kong.plugins.plugins_api_id_idx INFO 07:27:41 Initializing kong.plugins.plugins_name_idx INFO 07:27:41 Initializing kong.schema_migrations INFO 07:27:41 Initializing kong.consumers INFO 07:27:41 Initializing kong.consumers.consumers_username_idx INFO 07:27:41 Initializing kong.consumers.consumers_custom_id_idx INFO 07:27:41 Initializing kong.acls INFO 07:27:41 Initializing kong.acls.acls_group_idx INFO 07:27:41 Initializing kong.acls.acls_consumer_id INFO 07:27:41 Initializing kong.keyauth_credentials INFO 07:27:41 Initializing kong.keyauth_credentials.keyauth_credentials_key_idx INFO 07:27:41 Initializing kong.keyauth_credentials.keyauth_consumer_id INFO 07:27:41 Initializing kong.basicauth_credentials INFO 07:27:41 Initializing kong.basicauth_credentials.basicauth_consumer_id INFO 07:27:41 Initializing kong.basicauth_credentials.basicauth_credentials_username_idx INFO 07:27:41 Initializing kong.oauth2_tokens INFO 07:27:41 Initializing kong.oauth2_tokens.oauth2_tokens_refresh_token_idx INFO 07:27:41 Initializing kong.oauth2_tokens.oauth2_credential_id_idx INFO 07:27:41 Initializing kong.oauth2_tokens.oauth2_tokens_authenticated_userid_idx INFO 07:27:41 Initializing kong.oauth2_tokens.oauth2_tokens_access_token_idx INFO 07:27:41 Initializing kong.ratelimiting_metrics INFO 07:27:41 Initializing kong.jwt_secrets INFO 07:27:41 Initializing kong.jwt_secrets.jwt_secrets_secret_idx INFO 07:27:41 Initializing kong.jwt_secrets.jwt_secrets_key_idx INFO 07:27:41 Initializing kong.jwt_secrets.jwt_secrets_consumer_id_idx INFO 07:27:41 Initializing system_traces.sessions INFO 07:27:41 Initializing system_traces.events INFO 07:27:41 completed pre-loading (11 keys) key cache. INFO 07:27:41 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log, /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log, /var/lib/cassandra/commitlog/CommitLog-4-1444029293210.log, /var/lib/cassandra/commitlog/CommitLog-4-1444029756991.log, /var/lib/cassandra/commitlog/CommitLog-4-1444029869500.log INFO 07:27:41 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log INFO 07:27:41 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log (CL version 4, messaging version 8) INFO 07:27:43 Copy GC in 235ms. CMS Old Gen: 49192208 -> 119953704; Eden Space: 83886080 -> 0; Survivor Space: 10485760 -> 10485752 INFO 07:27:43 Copy GC in 229ms. CMS Old Gen: 119953704 -> 186705496; Eden Space: 83886080 -> 0; Survivor Space: 10485752 -> 10485760 INFO 07:27:43 Finished reading /var/lib/cassandra/commitlog/CommitLog-4-1443979306341.log INFO 07:27:43 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log INFO 07:27:43 Replaying /var/lib/cassandra/commitlog/CommitLog-4-1443979306342.log (CL version 4, messaging version 8)

sloev commented 8 years ago

i think i found the answer in that cassandra was runnning out of memory. the fix for me at least was to reduce cassandras memory by the following modifications: docker exec -i -t cassandra bash vi /etc/cassandra/default.conf/cassandra-env.sh

and uncommenting/modifying the MAX_HEAP_SIZE and NEW_HEAP_SIZE

to a soothing 512M, 200M

now it works!!!