openshift / origin-metrics

79 stars 113 forks source link

cluster metrics empty charts error #378

Closed Coyote5 closed 6 years ago

Coyote5 commented 7 years ago

I deployed the metrics components according to the https://docs.openshift.org/1.2/install_config/cluster_metrics.html. All component work well. But the charts remain empty. I don't know what's wrong with it.

openshift v1.2.1 kubernetes v1.2.0-36-g4a3f9c5 etcd 2.2.5

metrics container version: v1.2.1

hawkular-cassandra-1 logs

About to generate seeds
--
  | Trying to access the Seed list [try #1]
  | Trying to access the Seed list [try #2]
  | Trying to access the Seed list [try #3]
  | Setting seeds to be hawkular-cassandra-1-2np6q
  | cat: /etc/ld.so.conf.d/*.conf: No such file or directory
  | CompilerOracle: inline org/apache/cassandra/db/AbstractNativeCell.compareTo (Lorg/apache/cassandra/db/composites/Composite;)I
  | CompilerOracle: inline org/apache/cassandra/db/composites/AbstractSimpleCellNameType.compareUnsigned (Lorg/apache/cassandra/db/composites/Composite;Lorg/apache/cassandra/db/composites/Composite;)I
  | CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
  | CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
  | CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
  | CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
  | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
  | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
  | CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
  | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
  | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
  | CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
  | INFO  03:14:55 Loading settings from file:/opt/apache-cassandra-2.2.4/conf/cassandra.yaml
  | INFO  03:14:55 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-2np6q; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-2np6q; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=hawkular-cassandra-1-2np6q}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001
  | ; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
  | INFO  03:14:56 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
  | INFO  03:14:56 Global memtable on-heap threshold is enabled at 125MB
  | INFO  03:14:56 Global memtable off-heap threshold is enabled at 125MB
  | WARN  03:14:56 Only 33449 MB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
  | INFO  03:14:56 Loading settings from file:/opt/apache-cassandra-2.2.4/conf/cassandra.yaml
  | INFO  03:14:56 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-2np6q; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-2np6q; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=hawkular-cassandra-1-2np6q}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001
  | ; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
  | INFO  03:14:56 Hostname: hawkular-cassandra-1-2np6q
  | INFO  03:14:56 JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_77
  | INFO  03:14:56 Heap size: 526385152/526385152
  | INFO  03:14:56 Code Cache Non-heap memory: init = 2555904(2496K) used = 3349824(3271K) committed = 3407872(3328K) max = 251658240(245760K)
  | INFO  03:14:56 Metaspace Non-heap memory: init = 0(0K) used = 13175792(12866K) committed = 13369344(13056K) max = -1(-1K)
  | INFO  03:14:56 Compressed Class Space Non-heap memory: init = 0(0K) used = 1539536(1503K) committed = 1572864(1536K) max = 1073741824(1048576K)
  | INFO  03:14:56 Par Eden Space Heap memory: init = 83886080(81920K) used = 52196248(50972K) committed = 83886080(81920K) max = 83886080(81920K)
  | INFO  03:14:56 Par Survivor Space Heap memory: init = 10485760(10240K) used = 0(0K) committed = 10485760(10240K) max = 10485760(10240K)
  | INFO  03:14:56 CMS Old Gen Heap memory: init = 432013312(421888K) used = 0(0K) committed = 432013312(421888K) max = 432013312(421888K)
  | INFO  03:14:56 Classpath: /opt/apache-cassandra/conf:/opt/apache-cassandra/build/classes/main:/opt/apache-cassandra/build/classes/thrift:/opt/apache-cassandra/lib/ST4-4.0.8.jar:/opt/apache-cassandra/lib/airline-0.6.jar:/opt/apache-cassandra/lib/antlr-runtime-3.5.2.jar:/opt/apache-cassandra/lib/apache-cassandra-2.2.4.jar:/opt/apache-cassandra/lib/apache-cassandra-clientutil-2.2.4.jar:/opt/apache-cassandra/lib/apache-cassandra-thrift-2.2.4.jar:/opt/apache-cassandra/lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:/opt/apache-cassandra/lib/commons-cli-1.1.jar:/opt/apache-cassandra/lib/commons-codec-1.2.jar:/opt/apache-cassandra/lib/commons-lang3-3.1.jar:/opt/apache-cassandra/lib/commons-math3-3.2.jar:/opt/apache-cassandra/lib/compress-lzf-0.8.4.jar:/opt/apache-cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/apache-cassandra/lib/crc32ex-0.1.1.jar:/opt/apache-cassandra/lib/disruptor-3.0.1.jar:/opt/apache-cassandra/lib/ecj-4.4.2.jar:/opt/apache-cassandra/lib/guava-16.0.jar:/opt/apache-cassandra/lib/high-scale-lib-1.0.6.jar:/opt/apache-cassandra/lib/jackson-core-asl-1.9.2.jar:/opt/apache-cassandra/lib/jackson-mapper-asl-1.9.2.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar:/opt/apache-cassandra/lib/javax.inject.jar:/opt/apache-cassandra/lib/jbcrypt-0.3m.jar:/opt/apache-cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/jna-4.0.0.jar:/opt/apache-cassandra/lib/joda-time-2.4.jar:/opt/apache-cassandra/lib/json-simple-1.1.jar:/opt/apache-cassandra/lib/libthrift-0.9.2.jar:/opt/apache-cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/apache-cassandra/lib/logback-classic-1.1.3.jar:/opt/apache-cassandra/lib/logback-core-1.1.3.jar:/opt/apache-cassandra/lib/lz4-1.3.0.jar:/opt/apache-cassandra/lib/metrics-core-3.1.0.jar:/opt/apache-cassandra/lib/metrics-logback-3.1.0.jar:/opt/apache-cassandra/lib/netty-all-4.0.23.Final.jar:/opt/apache-cassandra/lib/ohc-core-0.3.4.jar:/opt/apache-cassandra/lib/ohc-core-j8-0.3.4.jar:/opt/apache-cassandra/lib/reporter-config-base-3.0.0.jar:/opt/apache-cassandra/lib/report
  | er-config3-3.0.0.jar:/opt/apache-cassandra/lib/sigar-1.6.4.jar:/opt/apache-cassandra/lib/slf4j-api-1.7.7.jar:/opt/apache-cassandra/lib/snakeyaml-1.11.jar:/opt/apache-cassandra/lib/snappy-java-1.1.1.7.jar:/opt/apache-cassandra/lib/stream-2.5.2.jar:/opt/apache-cassandra/lib/super-csv-2.1.0.jar:/opt/apache-cassandra/lib/thrift-server-0.3.7.jar:/opt/apache-cassandra/lib/jsr223/*/*.jar:/opt/apache-cassandra/lib/jamm-0.3.0.jar
  | WARN  03:14:56 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
  | WARN  03:14:56 jemalloc shared library could not be preloaded to speed up memory allocations
  | WARN  03:14:56 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
  | WARN  03:14:56 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
  | INFO  03:14:56 Initializing SIGAR library
  | WARN  03:14:56 Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true
  | WARN  03:14:57 Directory /cassandra_data/data doesn't exist
  | WARN  03:14:57 Directory /cassandra_data/commitlog doesn't exist
  | WARN  03:14:57 Directory /opt/apache-cassandra/data/saved_caches doesn't exist
  | INFO  03:14:58 Initializing system.sstable_activity
  | INFO  03:14:58 Initializing system.hints
  | INFO  03:14:58 Initializing system.compaction_history
  | INFO  03:14:58 Initializing system.peers
  | INFO  03:14:58 Initializing system.schema_columnfamilies
  | INFO  03:14:58 Initializing system.schema_functions
  | INFO  03:14:59 Initializing system.IndexInfo
  | INFO  03:14:59 Initializing system.schema_columns
  | INFO  03:14:59 Initializing system.schema_triggers
  | INFO  03:14:59 Initializing system.local
  | INFO  03:14:59 Initializing system.schema_usertypes
  | INFO  03:14:59 Initializing system.batchlog
  | INFO  03:14:59 Initializing system.available_ranges
  | INFO  03:14:59 Initializing system.schema_aggregates
  | INFO  03:14:59 Initializing system.paxos
  | INFO  03:14:59 Initializing system.peer_events
  | INFO  03:14:59 Initializing system.size_estimates
  | INFO  03:14:59 Initializing system.compactions_in_progress
  | INFO  03:14:59 Initializing system.schema_keyspaces
  | INFO  03:14:59 Initializing system.range_xfers
  | INFO  03:15:00 Initializing key cache with capacity of 25 MBs.
  | INFO  03:15:00 Initializing row cache with capacity of 0 MBs
  | INFO  03:15:00 Initializing counter cache with capacity of 12 MBs
  | INFO  03:15:00 Scheduling counter cache save to every 7200 seconds (going to save all keys).
  | INFO  03:15:00 Completed loading (2 ms; 1 keys) KeyCache cache
  | INFO  03:15:00 No commitlog files found; skipping replay
  | INFO  03:15:00 Cassandra version: 2.2.4
  | INFO  03:15:00 Thrift API version: 20.1.0
  | INFO  03:15:00 CQL supported versions: 3.3.1 (default: 3.3.1)
  | INFO  03:15:00 Initializing index summary manager with a memory pool size of 25 MB and a resize interval of 60 minutes
  | INFO  03:15:00 Loading persisted ring state
  | INFO  03:15:00 Loading settings from file:/opt/apache-cassandra-2.2.4/conf/cassandra.yaml
  | INFO  03:15:00 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-2np6q; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-2np6q; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=hawkular-cassandra-1-2np6q}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001
  | ; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
  | WARN  03:15:00 No host ID found, created 9470a281-f805-41f0-8a4d-66c3834a9101 (Note: This should happen exactly once per node).
  | INFO  03:15:00 Starting up server gossip
  | INFO  03:15:00 Loading settings from file:/opt/apache-cassandra-2.2.4/conf/cassandra.yaml
  | INFO  03:15:00 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-2np6q; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-2np6q; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=hawkular-cassandra-1-2np6q}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001
  | ; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
  | INFO  03:15:01 Starting Encrypted Messaging Service on SSL port 7001
  | INFO  03:15:01 Loading settings from file:/opt/apache-cassandra-2.2.4/conf/cassandra.yaml
  | INFO  03:15:01 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-2np6q; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-2np6q; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=hawkular-cassandra-1-2np6q}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001
  | ; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
  | INFO  03:15:01 Loading settings from file:/opt/apache-cassandra-2.2.4/conf/cassandra.yaml
  | INFO  03:15:01 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-2np6q; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-2np6q; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=hawkular-cassandra-1-2np6q}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001
  | ; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
  | INFO  03:15:01 This node will not auto bootstrap because it is configured to be a seed node.
  | INFO  03:15:01 Loading settings from file:/opt/apache-cassandra-2.2.4/conf/cassandra.yaml
  | INFO  03:15:01 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=hawkular-metrics; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=hawkular-cassandra-1-2np6q; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=hawkular-cassandra-1-2np6q; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=hawkular-cassandra-1-2np6q}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001
  | ; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
  | INFO  03:15:01 Generated random tokens. tokens are [695600860656407051, 7622425143117478407, -8171264075460542907, -8043948284838418579, 7137256265444406643, -4011259020723959081, -5027031888343864918, -696509492927545021, 3326260439021004378, -6944841654166827813, 3814621498758139121, -2403864332465733716, 5335224265132171274, -5239431022300298534, 8126657851341163712, 2438931300550685236, 8354847936819753091, 9193973200106745013, 4890399326658780876, 5358407006395890878, -6918387922396500477, -1868232014356742995, 5408322907540669628, -4570560556108417686, -7634987903569802357, 729208968040998555, 3675613992891887007, -63687764619025883, 3173340665508977224, 4938902332970895404, -2836145277780970493, -6135319952779911473, -8476238571757256327, 6365422029594901637, 3179168846482010858, -8730126011178729098, 7956779441882758430, 7145006042883690581, -6227891325644754124, 6934285040892445772, -3752390056449919444, -835784484965003074, 2015356733400589809, 7466766323189046069, -5793283798066204702, -4698229830198696033, -2531332375981523652, -2736016604763439434, 8149031702581753250, -8102648144669298519, -3998520988837143576, 2723012419084714572, -5635203096266899741, 8998820449402374942, -732764991968674186, -3163902124038382808, -5714910024485129615, 2138607509276132760, 2417404127612238135, -5669028943443154005, -1339134596323720560, 5815508075043947645, 9145495319966450834, -4069857832036237037, -1128724742488453595, -1419944362820841158, -3459828644704675064, 8949827566746189410, -7519142783876691141, -5133836213742857912, 19814433860611714, -1072776745544774957, -3363069295198541366, 9204388187690259354, 3408947212068663088, -7442049626511639175, -4949874341028957036, 1310779945826436756, 4853485469217147326, 3795842788891174878, -5778224618354971503, 8371125690233622614, 4958870610063799593, 1550211806201677607, -3346676264755626583, 9096729970318354303, -5899153742752194667, 3689550857217678949, -380109697917018132, 5408973331857956320, 661652705468049932, 3337979190864679311, 6688545713394891889, 42810482
  | 71245078675, -1164161638910619658, -8750189134690995163, -2654624127115685897, 2321637035858651572, 2277617982866875549, -7637828830740208656, -9065492610247616930, -5263753622348555513, -4888059958623946579, -318420347391438020, -4210426155492353981, -455449752225114020, -3860959924969973869, 6990892786637260822, 972316045187849505, 947283115942344810, -6252713313826801831, -513710729281454317, -5619988890627819682, -385521242918248345, 8289636501060756835, -6972860767179694062, -3029523265675126977, -8587084061525391158, 3575997387536146448, -8442929250962629104, -3393846330792193463, 3096259369501738228, 630195432209367787, -866792212437950031, 3336503046638008216, -921424670687001957, 6958279007619699878, -1995558145755594424, 51515882724253496, 8797420254771847795, -4750398773022060286, -3185468955304387610, 8198303654512729471, -5761981954960469341, 1155654271391122189, 1674006611503659590, 5343155896807933963, -8170772404069674338, -8139334237902164208, 4466227104407651196, 601346770577422822, 976073496892752328, 4770463111867673400, 1436833687617638286, -1967369957042893200, -2847044553785334632, -5476758356278784821, 1141040555372102112, 3743009056852130947, -8475798837534891590, 3708943264292659280, -7250973891311402728, -7538974401662481710, 4252025406004123241, 6946491808867264153, -5242405323920758983, 8729176617766520939, -3257351306601872746, 9038516354664661694, 4320718506927930801, -6749262186725563759, 2174557876973345613, 1601572226241446199, -6386456400486366043, 7501416584035313187, -3180774961654610304, -8478274892765030821, 4409650132350858134, -3398228510352148371, 729170237029709770, 5867551848608601839, 1086565809974305024, 8905830824619271706, 5441527004072545364, -4773562377235226261, 3371797618238641189, 674556355868321232, 4234244049117994712, -4214263383528692752, -8274969043126175697, -7348673866288557886, 9061907362494437099, -4074345973282474108, 5208003571870618845, -3934018191239921539, -3188462529263446771, 2118998632389310213, -5623677718334085372, -441441852590814962, -18265
  | 59161758123028, 2139636378201362447, 4118510517232284301, -1036707936137122512, -8388679675912095506, -152233146497604257, 2972265027973782339, -7426303327948542893, 5043710121395603461, 1954809339461830696, 3842573179746204442, 5464103401354072805, -2762050376616544798, 1919767699196287589, -3201225368577580663, 6057897841328685176, -6108029114547664149, -7539295645335185054, 1693074421061494132, 3937816485683624913, 187231695251375898, -3880493145524084223, -7525568362126864414, 8853579202768378346, -431877068709544106, -3541598587671035009, -2081547730438168609, 3796076102037527634, -7215932790113387748, -629080614641828026, 8352353651535612171, 8666715387962997947, 4625787035357807703, -4043136564254112815, -7797289785618867371, 8378865277915231889, -9089958327088025153, 9128052969586925546, -8912215696757144121, 8802457654637268070, 85996560319071268, 970477037397447608, 2404189846477715611, -6284677745760281152, -5254061834110339671, 5744088905614720354, 1707250344121452870, -6425247063123952501, -4731348353277052807, -7346236764230689444, 2606207149203221821, 3515883748495270931, 4402461431395043848, 7615256838801518032, -8270435648019047333, -2032510669644670501, -2333629271022951990, -6880469905990333994, 2552395293702148096, -7066871366892655342, -8283298468018271190, 961750218658389458, -7431222131473678450, 1416615670953635927, -3754376893848422087, -4423757393836275054, -198898023947095659]
  | INFO  03:15:01 Create new Keyspace: KSMetaData{name=system_traces, strategyClass=SimpleStrategy, strategyOptions={replication_factor=2}, cfMetaData={sessions=org.apache.cassandra.config.CFMetaData@7347971b[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(706172616d6574657273:org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type))),comment=tracing sessions,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UUIDType,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=client, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=command, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=coordinator, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=request, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=duration, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=parameters, type=org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassan
  | dra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false], events=org.apache.cassandra.config.CFMetaData@2eb33ef3[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.TimeUUIDType,org.apache.cassandra.db.marshal.UTF8Type),comment=tracing events,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UUIDType,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=activity, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=thread, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=event_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=source, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=source_elapsed, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}],compactionStrategyClass=class org.apa
  | che.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@40fbd792}
  | INFO  03:15:02 Loading org.apache.cassandra.config.CFMetaData@1d40a4a8[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(706172616d6574657273:org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type))),comment=tracing sessions,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UUIDType,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=client, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=command, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=coordinator, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=request, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=duration, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=parameters, type=org.apache.cassandra.db.marshal.MapType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cas
  | sandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:02 Initializing system_traces.sessions
  | INFO  03:15:02 Loading org.apache.cassandra.config.CFMetaData@a3d7592[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.TimeUUIDType,org.apache.cassandra.db.marshal.UTF8Type),comment=tracing events,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UUIDType,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=activity, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=session_id, type=org.apache.cassandra.db.marshal.UUIDType, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=thread, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=event_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=source, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=source_elapsed, type=org.apache.cassandra.db.marshal.Int32Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:02 Initializing system_traces.events
  | INFO  03:15:02 Create new Keyspace: KSMetaData{name=system_distributed, strategyClass=SimpleStrategy, strategyOptions={replication_factor=3}, cfMetaData={parent_repair_history=org.apache.cassandra.config.CFMetaData@25f66a6d[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(7265717565737465645f72616e676573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type),7375636365737366756c5f72616e676573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type),636f6c756d6e66616d696c795f6e616d6573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))),comment=Repair history,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=requested_ranges, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=exception_message, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=successful_ranges, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=finished_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=
  | REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=exception_stacktrace, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=columnfamily_names, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=parent_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false], repair_history=org.apache.cassandra.config.CFMetaData@5d82e7c0[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.TimeUUIDType,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(7061727469636970616e7473:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.InetAddressType))),comment=Repair history,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=status, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=id, type=org.ap
  | ache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=coordinator, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=finished_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=participants, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.InetAddressType), kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=exception_stacktrace, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=parent_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=range_end, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=range_begin, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=exception_message, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=columnfamily_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=1, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bl
  | oomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@2758b0f1}
  | INFO  03:15:02 Loading org.apache.cassandra.config.CFMetaData@67330218[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(7265717565737465645f72616e676573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type),7375636365737366756c5f72616e676573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type),636f6c756d6e66616d696c795f6e616d6573:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))),comment=Repair history,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=requested_ranges, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=exception_message, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=successful_ranges, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=finished_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=exception_stacktrace, type=org.apache.cassandra.db.marshal.UTF8Type, ki
  | nd=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=columnfamily_names, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=parent_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:02 Initializing system_distributed.parent_repair_history
  | INFO  03:15:02 Loading org.apache.cassandra.config.CFMetaData@1987b6f1[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.TimeUUIDType,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(7061727469636970616e7473:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.InetAddressType))),comment=Repair history,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=0,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=status, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=coordinator, type=org.apache.cassandra.db.marshal.InetAddressType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=finished_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=participants, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.InetAddressType), kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=exception_stacktrace, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=parent_id, type=org.apache.cassandra.db.marshal.TimeUUIDType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=range_end, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, compo
  | nentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=range_begin, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=exception_message, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=keyspace_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=started_at, type=org.apache.cassandra.db.marshal.TimestampType, kind=REGULAR, componentIndex=1, indexName=null, indexType=null}, ColumnDefinition{name=columnfamily_name, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=1, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:02 Initializing system_distributed.repair_history
  | INFO  03:15:03 Node hawkular-cassandra-1-2np6q/172.88.4.2 state jump to NORMAL
  | INFO  03:15:03 Create new Keyspace: KSMetaData{name=system_auth, strategyClass=SimpleStrategy, strategyOptions={replication_factor=1}, cfMetaData={role_permissions=org.apache.cassandra.config.CFMetaData@39326281[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(7065726d697373696f6e73:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))),comment=permissions granted to db roles,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=permissions, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=1, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false], roles=org.apache.cassandra.config.CFMetaData@6bf98e5c[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,cfType=Standard,comparator=org.apache.cassandra.db.marshal.Comp
  | ositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(6d656d6265725f6f66:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))),comment=role definitions,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=member_of, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=salted_hash, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=can_login, type=org.apache.cassandra.db.marshal.BooleanType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=is_superuser, type=org.apache.cassandra.db.marshal.BooleanType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false], resource_role_permissons_index=org.apache.cassandra.config.CFMetaData@295679cb[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marsh
  | al.UTF8Type),comment=index of db roles with permissions granted on a resource,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false], role_members=org.apache.cassandra.config.CFMetaData@3707a72e[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),comment=role memberships lookup table,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=member, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.S
  | izeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]}, durableWrites=true, userTypes=org.apache.cassandra.config.UTMetaData@62028ea4}
  | INFO  03:15:03 Loading org.apache.cassandra.config.CFMetaData@7b4ea4df[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(7065726d697373696f6e73:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))),comment=permissions granted to db roles,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=permissions, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=1, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:03 Initializing system_auth.role_permissions
  | INFO  03:15:03 Loading org.apache.cassandra.config.CFMetaData@1f03b8a9[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),comment=index of db roles with permissions granted on a resource,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:03 Initializing system_auth.resource_role_permissons_index
  | INFO  03:15:03 Loading org.apache.cassandra.config.CFMetaData@7d4b865b[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.ColumnToCollectionType(6d656d6265725f6f66:org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type))),comment=role definitions,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=member_of, type=org.apache.cassandra.db.marshal.SetType(org.apache.cassandra.db.marshal.UTF8Type), kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=salted_hash, type=org.apache.cassandra.db.marshal.UTF8Type, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=can_login, type=org.apache.cassandra.db.marshal.BooleanType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=is_superuser, type=org.apache.cassandra.db.marshal.BooleanType, kind=REGULAR, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:03 Initializing system_auth.roles
  | INFO  03:15:03 Loading org.apache.cassandra.config.CFMetaData@7dac3aa2[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),comment=role memberships lookup table,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, ColumnDefinition{name=member, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
  | INFO  03:15:03 Initializing system_auth.role_members
  | INFO  03:15:03 Waiting for gossip to settle before accepting client requests...
  | INFO  03:15:11 No gossip backlog; proceeding
  | INFO  03:15:11 Netty using native Epoll event loop
  | INFO  03:15:11 Enabling encrypted CQL connections between client and server
  | INFO  03:15:11 Using Netty Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c, netty-codec=netty-codec-4.0.23.Final.208198c, netty-codec-http=netty-codec-http-4.0.23.Final.208198c, netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c, netty-common=netty-common-4.0.23.Final.208198c, netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
  | INFO  03:15:11 Starting listening for CQL clients on hawkular-cassandra-1-2np6q/172.88.4.2:9042...
  | INFO  03:15:12 Binding thrift service to hawkular-cassandra-1-2np6q/172.88.4.2:9160
  | INFO  03:15:12 enabling encrypted thrift connections between client and server
  | INFO  03:15:12 Listening for thrift clients...
  | INFO  03:15:13 Created default superuser role 'cassandra'

Hawkular Metrics logs:

/opt/hawkular/auth /opt/jboss
Certificate was added to keystore
[Storing hawkular-metrics.truststore]
/opt/jboss
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/wildfly

  JAVA: /usr/lib/jvm/java/bin/java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

=========================================================================

03:21:24,908 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.1.Final
03:21:28,084 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
03:21:28,291 INFO  [org.jboss.as] (MSC service thread 1-7) WFLYSRV0049: WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) starting
03:21:32,358 INFO  [org.jboss.as.repository] (ServerService Thread Pool -- 7) WFLYDR0001: Content added at location /opt/jboss/wildfly/standalone/data/content/b3/00e06fc2560e8955d3f0f100a45693a69cca79/content
03:21:32,397 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
03:21:32,440 INFO  [org.xnio] (MSC service thread 1-4) XNIO version 3.3.4.Final
03:21:32,455 INFO  [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.3.4.Final
03:21:32,568 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 38) WFLYCLINF0001: Activating Infinispan subsystem.
03:21:32,593 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 46) WFLYNAM0001: Activating Naming Subsystem
03:21:32,588 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 54) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
03:21:32,652 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 44) WFLYJSF0007: Activated the following JSF Implementations: [main]
03:21:32,679 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 53) WFLYSEC0002: Activating Security Subsystem
03:21:32,706 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 37) WFLYIO001: Worker 'default' has auto-configured to 8 core threads with 64 task threads based on your 4 available processors
03:21:32,708 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 56) WFLYWS0002: Activating WebServices Extension
03:21:32,740 INFO  [org.jboss.as.connector] (MSC service thread 1-6) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.2.Final)
03:21:32,758 INFO  [org.jboss.as.security] (MSC service thread 1-3) WFLYSEC0001: Current PicketBox version=4.9.4.Final
03:21:32,790 INFO  [org.jboss.as.naming] (MSC service thread 1-2) WFLYNAM0003: Starting Naming Service
03:21:32,809 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-6) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
03:21:32,871 INFO  [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 4.0.18.Final
03:21:32,900 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 55) WFLYUT0003: Undertow 1.3.15.Final starting
03:21:32,903 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0003: Undertow 1.3.15.Final starting
03:21:32,924 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 33) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
03:21:32,960 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-6) WFLYJCA0018: Started Driver service with driver-name = h2
03:21:33,195 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 55) WFLYUT0014: Creating file handler for path '/opt/jboss/wildfly/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
03:21:33,314 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0012: Started server default-server.
03:21:33,321 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0018: Host default-host starting
03:21:33,653 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener default listening on 0.0.0.0:8080
03:21:33,678 INFO  [org.jboss.as.ejb3] (MSC service thread 1-1) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 16 (per class), which is derived from the number of CPUs on this host.
03:21:33,679 INFO  [org.jboss.as.ejb3] (MSC service thread 1-3) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 64 (per class), which is derived from thread worker pool sizing.
03:21:33,819 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
03:21:33,990 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-3) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss/wildfly/standalone/deployments
03:21:34,060 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0027: Starting deployment of "hawkular-metrics-api-jaxrs.war" (runtime-name: "hawkular-metrics-api-jaxrs.war")
03:21:34,506 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-2) ISPN000128: Infinispan version: Infinispan 'Mahou' 8.1.0.Final
03:21:34,778 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0006: Undertow HTTPS listener httpsServer listening on 0.0.0.0:8443
03:21:35,208 INFO  [org.jboss.ws.common.management] (MSC service thread 1-8) JBWS022052: Starting JBossWS 5.1.3.Final (Apache CXF 3.1.4) 
03:21:37,385 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment hawkular-metrics-api-jaxrs.war
03:21:37,651 INFO  [org.hibernate.validator.internal.util.Version] (MSC service thread 1-1) HV000001: Hibernate Validator 5.2.3.Final
03:21:38,342 INFO  [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0006: Starting Services for CDI deployment: hawkular-metrics-api-jaxrs.war
03:21:38,417 INFO  [org.jboss.weld.Version] (MSC service thread 1-3) WELD-000900: 2.3.2 (Final)
03:21:38,457 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0009: Starting weld service for deployment hawkular-metrics-api-jaxrs.war
03:21:39,769 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200002: Initializing metrics service
03:21:40,099 INFO  [com.datastax.driver.core.NettyUtil] (metricsservice-lifecycle-thread) Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
03:21:41,112 INFO  [org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp] (ServerService Thread Pool -- 65) HAWKMETRICS200001: Hawkular Metrics starting
03:21:41,125 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 65) RESTEASY002225: Deploying javax.ws.rs.core.Application: class org.hawkular.metrics.api.jaxrs.HawkularMetricsRestApp
03:21:41,411 INFO  [com.datastax.driver.core.policies.DCAwareRoundRobinPolicy] (metricsservice-lifecycle-thread) Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
03:21:41,412 INFO  [com.datastax.driver.core.Cluster] (metricsservice-lifecycle-thread) New Cassandra host hawkular-cassandra/172.30.118.70:9042 added
03:21:41,648 INFO  [org.hawkular.metrics.schema.SchemaManager] (metricsservice-lifecycle-thread) HAWKMETRICS300002: Creating schema for keyspace hawkular_metrics
03:21:41,724 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 65) WFLYUT0021: Registered web context: /hawkular/metrics
03:21:41,760 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 34) WFLYSRV0010: Deployed "hawkular-metrics-api-jaxrs.war" (runtime-name : "hawkular-metrics-api-jaxrs.war")
03:21:41,887 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
03:21:41,888 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
03:21:41,888 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 10.0.0.Final (WildFly Core 2.0.10.Final) started in 20209ms - Started 374 of 664 services (385 services are lazy, passive or on-demand)
03:21:59,881 INFO  [org.hawkular.metrics.core.service.MetricsServiceImpl] (metricsservice-lifecycle-thread) HAWKMETRICS100001: Using a key space of 'hawkular_metrics'
03:21:59,968 INFO  [org.hawkular.metrics.api.jaxrs.MetricsServiceLifecycle] (metricsservice-lifecycle-thread) HAWKMETRICS200005: Metrics service started

Heapster logs:

Endpoint Check in effect. Checking https://hawkular-metrics:443/hawkular/metrics/status
The endpoint check has successfully completed.
Starting Heapster with the following arguments: --source=kubernetes:https://kubernetes.default.svc:443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=Q3zVG2NRp7mMSn7&filter=label(container_name:^system.slice.*|^user.slice) --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users=system:master-proxy --metric_resolution=10s
I0913 03:24:38.282680       1 heapster.go:60] heapster --source=kubernetes:https://kubernetes.default.svc:443?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250 --sink=hawkular:https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=Q3zVG2NRp7mMSn7&filter=label(container_name:^system.slice.*|^user.slice) --tls_cert=/secrets/heapster.cert --tls_key=/secrets/heapster.key --tls_client_ca=/secrets/heapster.client-ca --allowed_users=system:master-proxy --metric_resolution=10s
I0913 03:24:38.282894       1 heapster.go:61] Heapster version 1.1.0-beta1
I0913 03:24:38.283515       1 configs.go:60] Using Kubernetes client with master "https://kubernetes.default.svc:443" and version "v1"
I0913 03:24:38.283552       1 configs.go:61] Using kubelet port 10250
I0913 03:24:38.342473       1 driver.go:322] Initialised Hawkular Sink with parameters {_system https://hawkular-metrics:443?tenant=_system&labelToTenant=pod_namespace&labelNodeId=nodename&caCert=/hawkular-cert/hawkular-metrics-ca.certificate&user=hawkular&pass=Q3zVG2NRp7mMSn7&filter=label(container_name:^system.slice.*|^user.slice) 0xc8201a8780  5}
I0913 03:24:38.507789       1 heapster.go:87] Starting with Hawkular-Metrics Sink
I0913 03:24:38.507835       1 heapster.go:87] Starting with Metric Sink
I0913 03:24:38.527978       1 heapster.go:166] Starting heapster on port 8082
I0913 03:24:45.000332       1 manager.go:79] Scraping metrics start: 2017-09-13 03:24:30 +0000 UTC, end: 2017-09-13 03:24:40 +0000 UTC
I0913 03:24:45.170125       1 manager.go:152] ScrapeMetrics: time: 169.572113ms size: 46
I0913 03:24:55.000339       1 manager.go:79] Scraping metrics start: 2017-09-13 03:24:40 +0000 UTC, end: 2017-09-13 03:24:50 +0000 UTC
I0913 03:24:55.132540       1 manager.go:152] ScrapeMetrics: time: 132.0242ms size: 45
I0913 03:25:05.000287       1 manager.go:79] Scraping metrics start: 2017-09-13 03:24:50 +0000 UTC, end: 2017-09-13 03:25:00 +0000 UTC
I0913 03:25:05.123318       1 manager.go:152] ScrapeMetrics: time: 122.828388ms size: 46
jsanda commented 6 years ago

I am going to close this issue since it has not had any recent activity. Please feel free to reopen if your issue is unresolved.