pods created but can't connect (from Janus), no nodetool status?

[Vince-Cercury]

Cassandra pod seem to be up and running

cassandra-1                             2/2       Running            0          1h
cassandra-2                             2/2       Running            0          1h

I'm also able to get Prometheus to scrape the metrics. But I'm not able to use cassandra. Janus cannot connect to it. It works fine with default Kubernetes statefulset example but not when I replace it with this helm chart.

com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: TokenRangeOfflineException: [host=10.2.14.17(10.2.14.17):9160, latency=2(2), attempts=1]UnavailableException()
    at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165)
    at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:65)

it looks like Cassandra uses a different port as per the service (9042). I did edit the service to map 9160 to 9042 to try to make Janus happy, but no luck

And I'm not able to run nodetool status for troubleshooting:

bash: nodetool: command not found

[deimosfr]

nodetool is just not in your path. I've fixed it in the latest docker image and updated the values.yaml file. Please update it.

On my side, I'm running multiple cassandra with this helm and get the correct ports open:

tcp        0      0 10.233.68.18:7000       0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:45533           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:7199            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:9160            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:41641           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:8778            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:9042            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:5555            0.0.0.0:*               LISTEN      - 

Can you please tell me what you have on your side?

Can you confirm you're targeting Cassandra this way: cassandra-0.cassandra..svc. ?

[Vince-Cercury]

Sorry for late reply.

Thanks for fixing path to nodetool.

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.2.25.8  103.71 KiB  256          100.0%            c8a6c1b4-c445-428e-8a90-b5e5dbea49d0  Rack1

On each of the cassandra node, the ring consist of only 1 node. For some reason the node 1 and 2 didn't see the seed or were not able to communicate with it.

How I installed the chart:

helm install --namespace=vince-cassandra-chart2 kubernetes/

Would you know what's happening? Kubernetes 1.9.3

UPDATE:

I noticed inside the node:

#!/bin/bash

export CLUSTER_DOMAIN=$(hostname -d | awk -F"." '{print $(NF-1),".",$NF}' | sed 's/ //g')
export CASSANDRA_SEEDS=cassandra-0.cassandra.vince-cassandra-chart2.svc.$CLUSTER_DOMAIN,cassandra-1.cassandra.vince-cassandra-chart2.svc.$CLUSTER_DOMAIN

Seems ok

The logs of the cassandra-1:

10.2.25.13
Starting Cassandra on 10.2.25.13
CASSANDRA_CONF_DIR /etc/cassandra
CASSANDRA_AUTO_BOOTSTRAP true
CASSANDRA_BROADCAST_ADDRESS 10.2.25.13
CASSANDRA_BROADCAST_RPC_ADDRESS 10.2.25.13
CASSANDRA_CFG /etc/cassandra/cassandra.yaml
CASSANDRA_CLUSTER_NAME vince-cassandra-chart3
CASSANDRA_COMPACTION_THROUGHPUT_MB_PER_SEC 150
CASSANDRA_CONCURRENT_COMPACTORS 2
CASSANDRA_CONCURRENT_READS 128
CASSANDRA_CONCURRENT_WRITES 128
CASSANDRA_COUNTER_CACHE_SIZE_IN_MB
CASSANDRA_DC DC1
CASSANDRA_DISK_OPTIMIZATION_STRATEGY ssd
CASSANDRA_ENDPOINT_SNITCH SimpleSnitch
CASSANDRA_GC_WARN_THRESHOLD_IN_MS
CASSANDRA_INTERNODE_COMPRESSION dc
CASSANDRA_KEY_CACHE_SIZE_IN_MB
CASSANDRA_LISTEN_ADDRESS 10.2.25.13
CASSANDRA_LISTEN_INTERFACE
CASSANDRA_LOG_JSON false
CASSANDRA_LOG_GC false
CASSANDRA_LOG_GC_VERBOSE false
CASSANDRA_LOG_PATH /var/log/cassandra
CASSANDRA_LOG_TO_FILES false
CASSANDRA_MEMTABLE_ALLOCATION_TYPE offheap_objects
CASSANDRA_MEMTABLE_CLEANUP_THRESHOLD 0.2
CASSANDRA_MEMTABLE_FLUSH_WRITERS 4
CASSANDRA_MIGRATION_WAIT 1
CASSANDRA_NUM_TOKENS 256
CASSANDRA_OPEN_JMX true
CASSANDRA_RACK Rack1
CASSANDRA_RING_DELAY 30000
CASSANDRA_RPC_ADDRESS 0.0.0.0
CASSANDRA_RPC_INTERFACE
CASSANDRA_SEEDS 10.2.25.13
CASSANDRA_SEED_PROVIDER org.apache.cassandra.locator.SimpleSeedProvider
FOUND CASSANDRA_YAML_batch_size_fail_threshold_in_kb batch_size_fail_threshold_in_kb 640
FOUND CASSANDRA_YAML_max_hints_delivery_threads max_hints_delivery_threads 4
FOUND CASSANDRA_YAML_row_cache_size_in_mb row_cache_size_in_mb 128
FOUND CASSANDRA_YAML_batch_size_warn_threshold_in_kb batch_size_warn_threshold_in_kb 64
FOUND CASSANDRA_YAML_hinted_handoff_throttle_in_kb hinted_handoff_throttle_in_kb 4096
FOUND CASSANDRA_YAML_row_cache_save_period row_cache_save_period 14400
changed ownership of '/var/lib/cassandra' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/triggers' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/logback-stdout.xml' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/logback-files.xml' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/cassandra.yaml' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/logback-json-stdout.xml' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/cassandra-env.sh' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/jvm.options' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/logback-json-files.xml' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/prometheus.yaml' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra/cassandra-rackdc.properties' from root:root to cassandra:cassandra
changed ownership of '/etc/cassandra' from root:root to cassandra:cassandra
changed ownership of '/var/log/cassandra' from root:root to cassandra:cassandra
/etc/resolv.conf
nameserver 10.10.84.132
search vince-cassandra-chart3.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/etc/cassandra/cassandra.yaml
# Cassandra storage config YAML

# NOTE:
#   See http://wiki.apache.org/cassandra/StorageConfiguration for
#   full explanations of configuration directives
# /NOTE

# The name of the cluster. This is mainly used to prevent machines in
# one logical cluster from joining another.
cluster_name: vince-cassandra-chart3

# This defines the number of tokens randomly assigned to this node on the ring
# The more tokens, relative to other nodes, the larger the proportion of data
# that this node will store. You probably want all nodes to have the same number
# of tokens assuming they have equal hardware capability.
#
# If you leave this unspecified, Cassandra will use the default of 1 token for legacy compatibility,
# and will use the initial_token as described below.
#
# Specifying initial_token will override this setting on the node's initial start,
# on subsequent starts, this setting will apply even if initial token is set.
#
# If you already have a cluster with 1 token per node, and wish to migrate to
# multiple tokens per node, see http://wiki.apache.org/cassandra/Operations
num_tokens: 256

# initial_token allows you to specify tokens manually.  While you can use # it with
# vnodes (num_tokens > 1, above) -- in which case you should provide a
# comma-separated list -- it's primarily used when adding nodes # to legacy clusters
# that do not have vnodes enabled.
# initial_token:

# See http://wiki.apache.org/cassandra/HintedHandoff
# May either be "true" or "false" to enable globally, or contain a list
# of data centers to enable per-datacenter.
# hinted_handoff_enabled: DC1,DC2
hinted_handoff_enabled: true
# this defines the maximum amount of time a dead host will have hints
# generated.  After it has been dead this long, new hints for it will not be
# created until it has been seen alive and gone down again.
max_hint_window_in_ms: 10800000 # 3 hours
# Maximum throttle in KBs per second, per delivery thread.  This will be
# reduced proportionally to the number of nodes in the cluster.  (If there
# are two nodes in the cluster, each delivery thread will use the maximum
# rate; if there are three, each will throttle to half of the maximum,
# since we expect two nodes to be delivering hints simultaneously.)
hinted_handoff_throttle_in_kb: 4096
# Number of threads with which to deliver hints;
# Consider increasing this number when you have multi-dc deployments, since
# cross-dc handoff tends to be slower
max_hints_delivery_threads: 4

# Maximum throttle in KBs per second, total. This will be
# reduced proportionally to the number of nodes in the cluster.
batchlog_replay_throttle_in_kb: 1024

# Authentication backend, implementing IAuthenticator; used to identify users
# Out of the box, Cassandra provides org.apache.cassandra.auth.{AllowAllAuthenticator,
# PasswordAuthenticator}.
#
# - AllowAllAuthenticator performs no checks - set it to disable authentication.
# - PasswordAuthenticator relies on username/password pairs to authenticate
#   users. It keeps usernames and hashed passwords in system_auth.credentials table.
#   Please increase system_auth keyspace replication factor if you use this authenticator.
authenticator: AllowAllAuthenticator

# Authorization backend, implementing IAuthorizer; used to limit access/provide permissions
# Out of the box, Cassandra provides org.apache.cassandra.auth.{AllowAllAuthorizer,
# CassandraAuthorizer}.
#
# - AllowAllAuthorizer allows any action to any user - set it to disable authorization.
# - CassandraAuthorizer stores permissions in system_auth.permissions table. Please
#   increase system_auth keyspace replication factor if you use this authorizer.
authorizer: AllowAllAuthorizer

# Validity period for permissions cache (fetching permissions can be an
# expensive operation depending on the authorizer, CassandraAuthorizer is
# one example). Defaults to 2000, set to 0 to disable.
# Will be disabled automatically for AllowAllAuthorizer.
permissions_validity_in_ms: 2000

# Refresh interval for permissions cache (if enabled).
# After this interval, cache entries become eligible for refresh. Upon next
# access, an async reload is scheduled and the old value returned until it
# completes. If permissions_validity_in_ms is non-zero, then this must be
# also.
# Defaults to the same value as permissions_validity_in_ms.
# permissions_update_interval_in_ms: 1000

# The partitioner is responsible for distributing groups of rows (by
# partition key) across nodes in the cluster.  You should leave this
# alone for new clusters.  The partitioner can NOT be changed without
# reloading all data, so when upgrading you should set this to the
# same partitioner you were already using.
#
# Besides Murmur3Partitioner, partitioners included for backwards
# compatibility include RandomPartitioner, ByteOrderedPartitioner, and
# OrderPreservingPartitioner.
#
partitioner: org.apache.cassandra.dht.Murmur3Partitioner

# Directories where Cassandra should store data on disk.  Cassandra
# will spread data evenly across them, subject to the granularity of
# the configured compaction strategy.
# If not set, the default directory is $CASSANDRA_HOME/data/data.
data_file_directories:
  - /var/lib/cassandra/data

# commit log.  when running on magnetic HDD, this should be a
# separate spindle than the data directories.
# If not set, the default directory is $CASSANDRA_HOME/data/commitlog.
commitlog_directory: /var/lib/cassandra/commitlog

# policy for data disk failures:
# die: shut down gossip and client transports and kill the JVM for any fs errors or
#      single-sstable errors, so the node can be replaced.
# stop_paranoid: shut down gossip and client transports even for single-sstable errors,
#                kill the JVM for errors during startup.
# stop: shut down gossip and client transports, leaving the node effectively dead, but
#       can still be inspected via JMX, kill the JVM for errors during startup.
# best_effort: stop using the failed disk and respond to requests based on
#              remaining available sstables.  This means you WILL see obsolete
#              data at CL.ONE!
# ignore: ignore fatal errors and let requests fail, as in pre-1.2 Cassandra
disk_failure_policy: stop

# policy for commit disk failures:
# die: shut down gossip and Thrift and kill the JVM, so the node can be replaced.
# stop: shut down gossip and Thrift, leaving the node effectively dead, but
#       can still be inspected via JMX.
# stop_commit: shutdown the commit log, letting writes collect but
#              continuing to service reads, as in pre-2.0.5 Cassandra
# ignore: ignore fatal errors and let the batches fail
commit_failure_policy: stop

# Maximum size of the key cache in memory.
#
# Each key cache hit saves 1 seek and each row cache hit saves 2 seeks at the
# minimum, sometimes more. The key cache is fairly tiny for the amount of
# time it saves, so it's worthwhile to use it at large numbers.
# The row cache saves even more time, but must contain the entire row,
# so it is extremely space-intensive. It's best to only use the
# row cache if you have hot rows or static rows.
#
# NOTE: if you reduce the size, you may not get you hottest keys loaded on startup.
#
# Default value is empty to make it "auto" (min(5% of Heap (in MB), 100MB)). Set to 0 to disable key cache.
key_cache_size_in_mb:

# Duration in seconds after which Cassandra should
# save the key cache. Caches are saved to saved_caches_directory as
# specified in this configuration file.
#
# Saved caches greatly improve cold-start speeds, and is relatively cheap in
# terms of I/O for the key cache. Row cache saving is much more expensive and
# has limited use.
#
# Default is 14400 or 4 hours.
key_cache_save_period: 14400

# Number of keys from the key cache to save
# Disabled by default, meaning all keys are going to be saved
# key_cache_keys_to_save: 100

# Maximum size of the row cache in memory.
# NOTE: if you reduce the size, you may not get you hottest keys loaded on startup.
#
# Default value is 0, to disable row caching.
row_cache_size_in_mb: 128

# Duration in seconds after which Cassandra should
# save the row cache. Caches are saved to saved_caches_directory as specified
# in this configuration file.
#
# Saved caches greatly improve cold-start speeds, and is relatively cheap in
# terms of I/O for the key cache. Row cache saving is much more expensive and
# has limited use.
#
# Default is 0 to disable saving the row cache.
row_cache_save_period: 14400

# Number of keys from the row cache to save
# Disabled by default, meaning all keys are going to be saved
# row_cache_keys_to_save: 100

# Maximum size of the counter cache in memory.
#
# Counter cache helps to reduce counter locks' contention for hot counter cells.
# In case of RF = 1 a counter cache hit will cause Cassandra to skip the read before
# write entirely. With RF > 1 a counter cache hit will still help to reduce the duration
# of the lock hold, helping with hot counter cell updates, but will not allow skipping
# the read entirely. Only the local (clock, count) tuple of a counter cell is kept
# in memory, not the whole counter, so it's relatively cheap.
#
# NOTE: if you reduce the size, you may not get you hottest keys loaded on startup.
#
# Default value is empty to make it "auto" (min(2.5% of Heap (in MB), 50MB)). Set to 0 to disable counter cache.
# NOTE: if you perform counter deletes and rely on low gcgs, you should disable the counter cache.
counter_cache_size_in_mb:

# Duration in seconds after which Cassandra should
# save the counter cache (keys only). Caches are saved to saved_caches_directory as
# specified in this configuration file.
#
# Default is 7200 or 2 hours.
counter_cache_save_period: 7200

# Number of keys from the counter cache to save
# Disabled by default, meaning all keys are going to be saved
# counter_cache_keys_to_save: 100

# The off-heap memory allocator.  Affects storage engine metadata as
# well as caches.  Experiments show that JEMAlloc saves some memory
# than the native GCC allocator (i.e., JEMalloc is more
# fragmentation-resistant).
#
# Supported values are: NativeAllocator, JEMallocAllocator
#
# If you intend to use JEMallocAllocator you have to install JEMalloc as library and
# modify cassandra-env.sh as directed in the file.
#
# Defaults to NativeAllocator
# memory_allocator: NativeAllocator

# saved caches
# If not set, the default directory is $CASSANDRA_HOME/data/saved_caches.
saved_caches_directory: /var/lib/cassandra/saved_caches

# commitlog_sync may be either "periodic" or "batch."
#
# When in batch mode, Cassandra won't ack writes until the commit log
# has been fsynced to disk.  It will wait
# commitlog_sync_batch_window_in_ms milliseconds between fsyncs.
# This window should be kept short because the writer threads will
# be unable to do extra work while waiting.  (You may need to increase
# concurrent_writes for the same reason.)
#
# commitlog_sync: batch
# commitlog_sync_batch_window_in_ms: 2
#
# the other option is "periodic" where writes may be acked immediately
# and the CommitLog is simply synced every commitlog_sync_period_in_ms
# milliseconds.
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

# The size of the individual commitlog file segments.  A commitlog
# segment may be archived, deleted, or recycled once all the data
# in it (potentially from each columnfamily in the system) has been
# flushed to sstables.
#
# The default size is 32, which is almost always fine, but if you are
# archiving commitlog segments (see commitlog_archiving.properties),
# then you probably want a finer granularity of archiving; 8 or 16 MB
# is reasonable.
commitlog_segment_size_in_mb: 32

# Reuse commit log files when possible. The default is false, and this
# feature will be removed entirely in future versions of Cassandra.
#commitlog_segment_recycling: false

# any class that implements the SeedProvider interface and has a
# constructor that takes a Map<String, String> of parameters will do.
seed_provider:
    # Addresses of hosts that are deemed contact points.
    # Cassandra nodes use this list of hosts to find each other and learn
    # the topology of the ring.  You must change this if you are running
    # multiple nodes!
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          # seeds is actually a comma-delimited list of addresses.
          # Ex: "<ip1>,<ip2>,<ip3>"
          - seeds: "10.2.25.13"

# For workloads with more data than can fit in memory, Cassandra's
# bottleneck will be reads that need to fetch data from
# disk. "concurrent_reads" should be set to (16 * number_of_drives) in
# order to allow the operations to enqueue low enough in the stack
# that the OS and drives can reorder them. Same applies to
# "concurrent_counter_writes", since counter writes read the current
# values before incrementing and writing them back.
#
# On the other hand, since writes are almost never IO bound, the ideal
# number of "concurrent_writes" is dependent on the number of cores in
# your system; (8 * number_of_cores) is a good rule of thumb.
concurrent_reads: 128
concurrent_writes: 128
concurrent_counter_writes: 32

# Total memory to use for sstable-reading buffers.  Defaults to
# the smaller of 1/4 of heap or 512MB.
# file_cache_size_in_mb: 512

# Total permitted memory to use for memtables. Cassandra will stop
# accepting writes when the limit is exceeded until a flush completes,
# and will trigger a flush based on memtable_cleanup_threshold
# If omitted, Cassandra will set both to 1/4 the size of the heap.
# memtable_heap_space_in_mb: 2048
# memtable_offheap_space_in_mb: 2048

# Ratio of occupied non-flushing memtable size to total permitted size
# that will trigger a flush of the largest memtable.  Lager mct will
# mean larger flushes and hence less compaction, but also less concurrent
# flush activity which can make it difficult to keep your disks fed
# under heavy write load.
#
# memtable_cleanup_threshold defaults to 1 / (memtable_flush_writers + 1)
memtable_cleanup_threshold: 0.2

# Specify the way Cassandra allocates and manages memtable memory.
# Options are:
#   heap_buffers:    on heap nio buffers
#   offheap_buffers: off heap (direct) nio buffers
#   offheap_objects: native memory, eliminating nio buffer heap overhead
memtable_allocation_type: offheap_objects

# Total space to use for commitlogs.  Since commitlog segments are
# mmapped, and hence use up address space, the default size is 32
# on 32-bit JVMs, and 8192 on 64-bit JVMs.
#
# If space gets above this value (it will round up to the next nearest
# segment multiple), Cassandra will flush every dirty CF in the oldest
# segment and remove it.  So a small total commitlog space will tend
# to cause more flush activity on less-active columnfamilies.
# commitlog_total_space_in_mb: 8192

# This sets the amount of memtable flush writer threads.  These will
# be blocked by disk io, and each one will hold a memtable in memory
# while blocked.
#
# memtable_flush_writers defaults to the smaller of (number of disks,
# number of cores), with a minimum of 2 and a maximum of 8.
#
# If your data directories are backed by SSD, you should increase this
# to the number of cores.
memtable_flush_writers: 4

# A fixed memory pool size in MB for for SSTable index summaries. If left
# empty, this will default to 5% of the heap size. If the memory usage of
# all index summaries exceeds this limit, SSTables with low read rates will
# shrink their index summaries in order to meet this limit.  However, this
# is a best-effort process. In extreme conditions Cassandra may need to use
# more than this amount of memory.
index_summary_capacity_in_mb:

# How frequently index summaries should be resampled.  This is done
# periodically to redistribute memory from the fixed-size pool to sstables
# proportional their recent read rates.  Setting to -1 will disable this
# process, leaving existing index summaries at their current sampling level.
index_summary_resize_interval_in_minutes: 60

# Whether to, when doing sequential writing, fsync() at intervals in
# order to force the operating system to flush the dirty
# buffers. Enable this to avoid sudden dirty buffer flushing from
# impacting read latencies. Almost always a good idea on SSDs; not
# necessarily on platters.
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240

# TCP port, for commands and data
# For security reasons, you should not expose this port to the internet.  Firewall it if needed.
storage_port: 7000

# SSL port, for encrypted communication.  Unused unless enabled in
# encryption_options
# For security reasons, you should not expose this port to the internet.  Firewall it if needed.
ssl_storage_port: 7001

# Address or interface to bind to and tell other Cassandra nodes to connect to.
# You _must_ change this if you want multiple nodes to be able to communicate!
#
# Set listen_address OR listen_interface, not both. Interfaces must correspond
# to a single address, IP aliasing is not supported.
#
# Leaving it blank leaves it up to InetAddress.getLocalHost(). This
# will always do the Right Thing _if_ the node is properly configured
# (hostname, name resolution, etc), and the Right Thing is to use the
# address associated with the hostname (it might not be).
#
# Setting listen_address to 0.0.0.0 is always wrong.
#
# If you choose to specify the interface by name and the interface has an ipv4 and an ipv6 address
# you can specify which should be chosen using listen_interface_prefer_ipv6. If false the first ipv4
# address will be used. If true the first ipv6 address will be used. Defaults to false preferring
# ipv4. If there is only one address it will be selected regardless of ipv4/ipv6.
listen_address: 10.2.25.13
# listen_interface: eth0
# listen_interface_prefer_ipv6: false

# Address to broadcast to other Cassandra nodes
# Leaving this blank will set it to the same value as listen_address
broadcast_address: 10.2.25.13

# Internode authentication backend, implementing IInternodeAuthenticator;
# used to allow/disallow connections from peer nodes.
# internode_authenticator: org.apache.cassandra.auth.AllowAllInternodeAuthenticator

# Whether to start the native transport server.
# Please note that the address on which the native transport is bound is the
# same as the rpc_address. The port however is different and specified below.
start_native_transport: true
# port for the CQL native transport to listen for clients on
# For security reasons, you should not expose this port to the internet.  Firewall it if needed.
native_transport_port: 9042
# The maximum threads for handling requests when the native transport is used.
# This is similar to rpc_max_threads though the default differs slightly (and
# there is no native_transport_min_threads, idle threads will always be stopped
# after 30 seconds).
# native_transport_max_threads: 128
#
# The maximum size of allowed frame. Frame (requests) larger than this will
# be rejected as invalid. The default is 256MB.
# native_transport_max_frame_size_in_mb: 256

# The maximum number of concurrent client connections.
# The default is -1, which means unlimited.
# native_transport_max_concurrent_connections: -1

# The maximum number of concurrent client connections per source ip.
# The default is -1, which means unlimited.
# native_transport_max_concurrent_connections_per_ip: -1

# Whether to start the thrift rpc server.
start_rpc: true

# The address or interface to bind the Thrift RPC service and native transport
# server to.
#
# Set rpc_address OR rpc_interface, not both. Interfaces must correspond
# to a single address, IP aliasing is not supported.
#
# Leaving rpc_address blank has the same effect as on listen_address
# (i.e. it will be based on the configured hostname of the node).
#
# Note that unlike listen_address, you can specify 0.0.0.0, but you must also
# set broadcast_rpc_address to a value other than 0.0.0.0.
#
# For security reasons, you should not expose this port to the internet.  Firewall it if needed.
#
# If you choose to specify the interface by name and the interface has an ipv4 and an ipv6 address
# you can specify which should be chosen using rpc_interface_prefer_ipv6. If false the first ipv4
# address will be used. If true the first ipv6 address will be used. Defaults to false preferring
# ipv4. If there is only one address it will be selected regardless of ipv4/ipv6.
rpc_address: 0.0.0.0
# rpc_interface: eth1
# rpc_interface_prefer_ipv6: false

# port for Thrift to listen for clients on
rpc_port: 9160

# RPC address to broadcast to drivers and other Cassandra nodes. This cannot
# be set to 0.0.0.0. If left blank, this will be set to the value of
# rpc_address. If rpc_address is set to 0.0.0.0, broadcast_rpc_address must
# be set.
broadcast_rpc_address: 10.2.25.13

# enable or disable keepalive on rpc/native connections
rpc_keepalive: true

# Cassandra provides two out-of-the-box options for the RPC Server:
#
# sync  -> One thread per thrift connection. For a very large number of clients, memory
#          will be your limiting factor. On a 64 bit JVM, 180KB is the minimum stack size
#          per thread, and that will correspond to your use of virtual memory (but physical memory
#          may be limited depending on use of stack space).
#
# hsha  -> Stands for "half synchronous, half asynchronous." All thrift clients are handled
#          asynchronously using a small number of threads that does not vary with the amount
#          of thrift clients (and thus scales well to many clients). The rpc requests are still
#          synchronous (one thread per active request). If hsha is selected then it is essential
#          that rpc_max_threads is changed from the default value of unlimited.
#
# The default is sync because on Windows hsha is about 30% slower.  On Linux,
# sync/hsha performance is about the same, with hsha of course using less memory.
#
# Alternatively,  can provide your own RPC server by providing the fully-qualified class name
# of an o.a.c.t.TServerFactory that can create an instance of it.
rpc_server_type: sync

# Uncomment rpc_min|max_thread to set request pool size limits.
#
# Regardless of your choice of RPC server (see above), the number of maximum requests in the
# RPC thread pool dictates how many concurrent requests are possible (but if you are using the sync
# RPC server, it also dictates the number of clients that can be connected at all).
#
# The default is unlimited and thus provides no protection against clients overwhelming the server. You are
# encouraged to set a maximum that makes sense for you in production, but do keep in mind that
# rpc_max_threads represents the maximum number of client requests this server may execute concurrently.
#
# rpc_min_threads: 16
# rpc_max_threads: 2048

# uncomment to set socket buffer sizes on rpc connections
# rpc_send_buff_size_in_bytes:
# rpc_recv_buff_size_in_bytes:

# Uncomment to set socket buffer size for internode communication
# Note that when setting this, the buffer size is limited by net.core.wmem_max
# and when not setting it it is defined by net.ipv4.tcp_wmem
# See:
# /proc/sys/net/core/wmem_max
# /proc/sys/net/core/rmem_max
# /proc/sys/net/ipv4/tcp_wmem
# /proc/sys/net/ipv4/tcp_wmem
# and: man tcp
# internode_send_buff_size_in_bytes:
# internode_recv_buff_size_in_bytes:

# Frame size for thrift (maximum message length).
thrift_framed_transport_size_in_mb: 15

# Set to true to have Cassandra create a hard link to each sstable
# flushed or streamed locally in a backups/ subdirectory of the
# keyspace data.  Removing these links is the operator's
# responsibility.
incremental_backups: false

# Whether or not to take a snapshot before each compaction.  Be
# careful using this option, since Cassandra won't clean up the
# snapshots for you.  Mostly useful if you're paranoid when there
# is a data format change.
snapshot_before_compaction: false

# Whether or not a snapshot is taken of the data before keyspace truncation
# or dropping of column families. The STRONGLY advised default of true
# should be used to provide data safety. If you set this flag to false, you will
# lose data on truncation or drop.
auto_snapshot: true

# When executing a scan, within or across a partition, we need to keep the
# tombstones seen in memory so we can return them to the coordinator, which
# will use them to make sure other replicas also know about the deleted rows.
# With workloads that generate a lot of tombstones, this can cause performance
# problems and even exaust the server heap.
# (http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets)
# Adjust the thresholds here if you understand the dangers and want to
# scan more tombstones anyway.  These thresholds may also be adjusted at runtime
# using the StorageService mbean.
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000

# Granularity of the collation index of rows within a partition.
# Increase if your rows are large, or if you have a very large
# number of rows per partition.  The competing goals are these:
#   1) a smaller granularity means more index entries are generated
#      and looking up rows withing the partition by collation column
#      is faster
#   2) but, Cassandra will keep the collation index in memory for hot
#      rows (as part of the key cache), so a larger granularity means
#      you can cache more hot rows
column_index_size_in_kb: 64

# Log WARN on any batch size exceeding this value. 5kb per batch by default.
# Caution should be taken on increasing the size of this threshold as it can lead to node instability.
batch_size_warn_threshold_in_kb: 64

# Log WARN on any batches not of type LOGGED than span across more partitions than this limit
unlogged_batch_across_partitions_warn_threshold: 10

# Number of simultaneous compactions to allow, NOT including
# validation "compactions" for anti-entropy repair.  Simultaneous
# compactions can help preserve read performance in a mixed read/write
# workload, by mitigating the tendency of small sstables to accumulate
# during a single long running compactions. The default is usually
# fine and if you experience problems with compaction running too
# slowly or too fast, you should look at
# compaction_throughput_mb_per_sec first.
#
# concurrent_compactors defaults to the smaller of (number of disks,
# number of cores), with a minimum of 2 and a maximum of 8.
#
# If your data directories are backed by SSD, you should increase this
# to the number of cores.
concurrent_compactors: 2

# Throttles compaction to the given total throughput across the entire
# system. The faster you insert data, the faster you need to compact in
# order to keep the sstable count down, but in general, setting this to
# 16 to 32 times the rate you are inserting data is more than sufficient.
# Setting this to 0 disables throttling. Note that this account for all types
# of compaction, including validation compaction.
compaction_throughput_mb_per_sec: 150

# Log a warning when compacting partitions larger than this value
compaction_large_partition_warning_threshold_mb: 100

# When compacting, the replacement sstable(s) can be opened before they
# are completely written, and used in place of the prior sstables for
# any range that has been written. This helps to smoothly transfer reads
# between the sstables, reducing page cache churn and keeping hot rows hot
sstable_preemptive_open_interval_in_mb: 50

# Throttles all outbound streaming file transfers on this node to the
# given total throughput in Mbps. This is necessary because Cassandra does
# mostly sequential IO when streaming data during bootstrap or repair, which
# can lead to saturating the network connection and degrading rpc performance.
# When unset, the default is 200 Mbps or 25 MB/s.
# stream_throughput_outbound_megabits_per_sec: 200

# Throttles all streaming file transfer between the datacenters,
# this setting allows users to throttle inter dc stream throughput in addition
# to throttling all network stream traffic as configured with
# stream_throughput_outbound_megabits_per_sec
# When unset, the default is 200 Mbps or 25 MB/s
# inter_dc_stream_throughput_outbound_megabits_per_sec: 200

# How long the coordinator should wait for read operations to complete
read_request_timeout_in_ms: 5000
# How long the coordinator should wait for seq or index scans to complete
range_request_timeout_in_ms: 10000
# How long the coordinator should wait for writes to complete
write_request_timeout_in_ms: 2000
# How long the coordinator should wait for counter writes to complete
counter_write_request_timeout_in_ms: 5000
# How long a coordinator should continue to retry a CAS operation
# that contends with other proposals for the same row
cas_contention_timeout_in_ms: 1000
# How long the coordinator should wait for truncates to complete
# (This can be much longer, because unless auto_snapshot is disabled
# we need to flush first so we can snapshot before removing the data.)
truncate_request_timeout_in_ms: 60000
# The default timeout for other, miscellaneous operations
request_timeout_in_ms: 10000

# Enable operation timeout information exchange between nodes to accurately
# measure request timeouts.  If disabled, replicas will assume that requests
# were forwarded to them instantly by the coordinator, which means that
# under overload conditions we will waste that much extra time processing
# already-timed-out requests.
#
# Warning: before enabling this property make sure to ntp is installed
# and the times are synchronized between the nodes.
cross_node_timeout: false

# Set socket timeout for streaming operation.
# The stream session is failed if no data/ack is received by any of the participants
# within that period, which means this should also be sufficient to stream a large
# sstable or rebuild table indexes.
# Default value is 86400000ms, which means stale streams timeout after 24 hours.
# A value of zero means stream sockets should never time out.
# streaming_socket_timeout_in_ms: 86400000

# phi value that must be reached for a host to be marked down.
# most users should never need to adjust this.
# phi_convict_threshold: 8

# endpoint_snitch -- Set this to a class that implements
# IEndpointSnitch.  The snitch has two functions:
# - it teaches Cassandra enough about your network topology to route
#   requests efficiently
# - it allows Cassandra to spread replicas around your cluster to avoid
#   correlated failures. It does this by grouping machines into
#   "datacenters" and "racks."  Cassandra will do its best not to have
#   more than one replica on the same "rack" (which may not actually
#   be a physical location)
#
# CASSANDRA WILL NOT ALLOW YOU TO SWITCH TO AN INCOMPATIBLE SNITCH
# ONCE DATA IS INSERTED INTO THE CLUSTER.  This would cause data loss.
# This means that if you start with the default SimpleSnitch, which
# locates every node on "rack1" in "datacenter1", your only options
# if you need to add another datacenter are GossipingPropertyFileSnitch
# (and the older PFS).  From there, if you want to migrate to an
# incompatible snitch like Ec2Snitch you can do it by adding new nodes
# under Ec2Snitch (which will locate them in a new "datacenter") and
# decommissioning the old ones.
#
# Out of the box, Cassandra provides
#  - SimpleSnitch:
#    Treats Strategy order as proximity. This can improve cache
#    locality when disabling read repair.  Only appropriate for
#    single-datacenter deployments.
#  - GossipingPropertyFileSnitch
#    This should be your go-to snitch for production use.  The rack
#    and datacenter for the local node are defined in
#    cassandra-rackdc.properties and propagated to other nodes via
#    gossip.  If cassandra-topology.properties exists, it is used as a
#    fallback, allowing migration from the PropertyFileSnitch.
#  - PropertyFileSnitch:
#    Proximity is determined by rack and data center, which are
#    explicitly configured in cassandra-topology.properties.
#  - Ec2Snitch:
#    Appropriate for EC2 deployments in a single Region. Loads Region
#    and Availability Zone information from the EC2 API. The Region is
#    treated as the datacenter, and the Availability Zone as the rack.
#    Only private IPs are used, so this will not work across multiple
#    Regions.
#  - Ec2MultiRegionSnitch:
#    Uses public IPs as broadcast_address to allow cross-region
#    connectivity.  (Thus, you should set seed addresses to the public
#    IP as well.) You will need to open the storage_port or
#    ssl_storage_port on the public IP firewall.  (For intra-Region
#    traffic, Cassandra will switch to the private IP after
#    establishing a connection.)
#  - RackInferringSnitch:
#    Proximity is determined by rack and data center, which are
#    assumed to correspond to the 3rd and 2nd octet of each node's IP
#    address, respectively.  Unless this happens to match your
#    deployment conventions, this is best used as an example of
#    writing a custom Snitch class and is provided in that spirit.
#
# You can use a custom Snitch by setting this to the full class name
# of the snitch, which will be assumed to be on your classpath.
endpoint_snitch: GossipingPropertyFileSnitch

# controls how often to perform the more expensive part of host score
# calculation
dynamic_snitch_update_interval_in_ms: 100
# controls how often to reset all host scores, allowing a bad host to
# possibly recover
dynamic_snitch_reset_interval_in_ms: 600000
# if set greater than zero and read_repair_chance is < 1.0, this will allow
# 'pinning' of replicas to hosts in order to increase cache capacity.
# The badness threshold will control how much worse the pinned host has to be
# before the dynamic snitch will prefer other replicas over it.  This is
# expressed as a double which represents a percentage.  Thus, a value of
# 0.2 means Cassandra would continue to prefer the static snitch values
# until the pinned host was 20% worse than the fastest.
dynamic_snitch_badness_threshold: 0.1

# request_scheduler -- Set this to a class that implements
# RequestScheduler, which will schedule incoming client requests
# according to the specific policy. This is useful for multi-tenancy
# with a single Cassandra cluster.
# NOTE: This is specifically for requests from the client and does
# not affect inter node communication.
# org.apache.cassandra.scheduler.NoScheduler - No scheduling takes place
# org.apache.cassandra.scheduler.RoundRobinScheduler - Round robin of
# client requests to a node with a separate queue for each
# request_scheduler_id. The scheduler is further customized by
# request_scheduler_options as described below.
request_scheduler: org.apache.cassandra.scheduler.NoScheduler

# Scheduler Options vary based on the type of scheduler
# NoScheduler - Has no options
# RoundRobin
#  - throttle_limit -- The throttle_limit is the number of in-flight
#                      requests per client.  Requests beyond
#                      that limit are queued up until
#                      running requests can complete.
#                      The value of 80 here is twice the number of
#                      concurrent_reads + concurrent_writes.
#  - default_weight -- default_weight is optional and allows for
#                      overriding the default which is 1.
#  - weights -- Weights are optional and will default to 1 or the
#               overridden default_weight. The weight translates into how
#               many requests are handled during each turn of the
#               RoundRobin, based on the scheduler id.
#
# request_scheduler_options:
#    throttle_limit: 80
#    default_weight: 5
#    weights:
#      Keyspace1: 1
#      Keyspace2: 5

# request_scheduler_id -- An identifier based on which to perform
# the request scheduling. Currently the only valid option is keyspace.
# request_scheduler_id: keyspace

# Enable or disable inter-node encryption
# Default settings are TLS v1, RSA 1024-bit keys (it is imperative that
# users generate their own keys) TLS_RSA_WITH_AES_128_CBC_SHA as the cipher
# suite for authentication, key exchange and encryption of the actual data transfers.
# Use the DHE/ECDHE ciphers if running in FIPS 140 compliant mode.
# NOTE: No custom encryption options are enabled at the moment
# The available internode options are : all, none, dc, rack
#
# If set to dc cassandra will encrypt the traffic between the DCs
# If set to rack cassandra will encrypt the traffic between the racks
#
# The passwords used in these options must match the passwords used when generating
# the keystore and truststore.  For instructions on generating these files, see:
# http://download.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#CreateKeystore
#
server_encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra
    # More advanced defaults below:
    # protocol: TLS
    # algorithm: SunX509
    # store_type: JKS
    # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]
    # require_client_auth: false

# enable or disable client/server encryption.
client_encryption_options:
    enabled: false
    # If enabled and optional is set to true encrypted and unencrypted connections are handled.
    optional: false
    keystore: conf/.keystore
    keystore_password: cassandra
    # require_client_auth: false
    # Set trustore and truststore_password if require_client_auth is true
    # truststore: conf/.truststore
    # truststore_password: cassandra
    # More advanced defaults below:
    # protocol: TLS
    # algorithm: SunX509
    # store_type: JKS
    # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

# internode_compression controls whether traffic between nodes is
# compressed.
# can be:  all  - all traffic is compressed
#          dc   - traffic between different datacenters is compressed
#          none - nothing is compressed.
internode_compression: dc

# Enable or disable tcp_nodelay for inter-dc communication.
# Disabling it will result in larger (but fewer) network packets being sent,
# reducing overhead from the TCP protocol itself, at the cost of increasing
# latency if you block for cross-datacenter responses.
inter_dc_tcp_nodelay: false

# GC Pauses greater than gc_warn_threshold_in_ms will be logged at WARN level
# Adjust the threshold based on your application throughput requirement
# By default, Cassandra logs GC Pauses greater than 200 ms at INFO level
# gc_warn_threshold_in_ms: 1000

otc_coalescing_strategy: TIMEHORIZON
otc_coalescing_window_us: 200
auto_bootstrap: true
-XX:+AlwaysPreTouch -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -XX:G1RSetUpdatingPauseTimePercent=5 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=4294967296 -XX:+ManagementServer -XX:MaxHeapSize=4294967296 -XX:+PerfDisableSharedMem -XX:+PrintCommandLineFlags -XX:+ResizeTLAB -XX:StringTableSize=1000003 -XX:ThreadPriorityPolicy=42 -XX:ThreadStackSize=256 -XX:-UseBiasedLocking -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseTLAB -XX:+UseThreadPriorities 
I> No access restrictor found, access to any MBean is allowed
Jolokia: Agent started with URL http://10.2.25.13:8778/jolokia/
2018-06-25 04:31:17.820:INFO:ipjsoejs.Server:jetty-8.y.z-SNAPSHOT
2018-06-25 04:31:17.875:INFO:ipjsoejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:1234
INFO  04:31:18 Configuration location: file:/etc/cassandra/cassandra.yaml
INFO  04:31:18 Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=64; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.2.25.13; broadcast_rpc_address=10.2.25.13; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; client_encryption_options=<REDACTED>; cluster_name=vince-cassandra-chart3; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=150; concurrent_compactors=2; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=128; concurrent_replicates=null; concurrent_writes=128; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@524d6d96; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=0; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=4096; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.2.25.13; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=4; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=offheap_objects; memtable_cleanup_threshold=0.2; memtable_flush_writers=4; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=null; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=14400; row_cache_size_in_mb=128; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.2.25.13}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@152aa092; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=0; write_request_timeout_in_ms=2000]
INFO  04:31:18 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  04:31:18 Global memtable on-heap threshold is enabled at 1024MB
INFO  04:31:18 Global memtable off-heap threshold is enabled at 1024MB
WARN  04:31:18 Only 30.692GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
WARN  04:31:18 memtable_cleanup_threshold has been deprecated and should be removed from cassandra.yaml
INFO  04:31:18 Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
INFO  04:31:18 Back-pressure is disabled with strategy null.
INFO  04:31:18 Unable to load cassandra-topology.properties; compatibility mode disabled
WARN  04:31:18 JMX settings in cassandra-env.sh have been bypassed as the JMX connector server is already initialized. Please refer to cassandra-env.(sh|ps1) for JMX configuration info
INFO  04:31:18 Hostname: cassandra-1.cassandra.vince-cassandra-chart3.svc.cluster.local
INFO  04:31:18 JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_171
INFO  04:31:18 Heap size: 4.000GiB/4.000GiB
INFO  04:31:18 Code Cache Non-heap memory: init = 2555904(2496K) used = 4135040(4038K) committed = 4194304(4096K) max = 251658240(245760K)
INFO  04:31:18 Metaspace Non-heap memory: init = 0(0K) used = 19594904(19135K) committed = 20185088(19712K) max = -1(-1K)
INFO  04:31:18 Compressed Class Space Non-heap memory: init = 0(0K) used = 2404576(2348K) committed = 2621440(2560K) max = 1073741824(1048576K)
INFO  04:31:18 G1 Eden Space Heap memory: init = 226492416(221184K) used = 125829120(122880K) committed = 226492416(221184K) max = -1(-1K)
INFO  04:31:18 G1 Survivor Space Heap memory: init = 0(0K) used = 0(0K) committed = 0(0K) max = -1(-1K)
INFO  04:31:18 G1 Old Gen Heap memory: init = 4068474880(3973120K) used = 0(0K) committed = 4068474880(3973120K) max = 4294967296(4194304K)
INFO  04:31:18 Classpath: /etc/cassandra:/usr/local/apache-cassandra/bin/../build/classes/main:/usr/local/apache-cassandra/bin/../build/classes/thrift:/usr/local/apache-cassandra/bin/../lib/HdrHistogram-2.1.9.jar:/usr/local/apache-cassandra/bin/../lib/ST4-4.0.8.jar:/usr/local/apache-cassandra/bin/../lib/airline-0.6.jar:/usr/local/apache-cassandra/bin/../lib/antlr-runtime-3.5.2.jar:/usr/local/apache-cassandra/bin/../lib/apache-cassandra-3.11.1.jar:/usr/local/apache-cassandra/bin/../lib/apache-cassandra-thrift-3.11.1.jar:/usr/local/apache-cassandra/bin/../lib/asm-5.0.4.jar:/usr/local/apache-cassandra/bin/../lib/caffeine-2.2.6.jar:/usr/local/apache-cassandra/bin/../lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/local/apache-cassandra/bin/../lib/commons-cli-1.1.jar:/usr/local/apache-cassandra/bin/../lib/commons-codec-1.9.jar:/usr/local/apache-cassandra/bin/../lib/commons-lang3-3.1.jar:/usr/local/apache-cassandra/bin/../lib/commons-math3-3.2.jar:/usr/local/apache-cassandra/bin/../lib/compress-lzf-0.8.4.jar:/usr/local/apache-cassandra/bin/../lib/concurrent-trees-2.4.0.jar:/usr/local/apache-cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/local/apache-cassandra/bin/../lib/disruptor-3.0.1.jar:/usr/local/apache-cassandra/bin/../lib/ecj-4.4.2.jar:/usr/local/apache-cassandra/bin/../lib/guava-18.0.jar:/usr/local/apache-cassandra/bin/../lib/high-scale-lib-1.0.6.jar:/usr/local/apache-cassandra/bin/../lib/hppc-0.5.4.jar:/usr/local/apache-cassandra/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/apache-cassandra/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/apache-cassandra/bin/../lib/jamm-0.3.0.jar:/usr/local/apache-cassandra/bin/../lib/javax.inject.jar:/usr/local/apache-cassandra/bin/../lib/jbcrypt-0.3m.jar:/usr/local/apache-cassandra/bin/../lib/jcl-over-slf4j-1.7.7.jar:/usr/local/apache-cassandra/bin/../lib/jctools-core-1.2.1.jar:/usr/local/apache-cassandra/bin/../lib/jflex-1.6.0.jar:/usr/local/apache-cassandra/bin/../lib/jna-4.2.2.jar:/usr/local/apache-cassandra/bin/../lib/joda-time-2.4.jar:/usr/local/apache-cassandra/bin/../lib/json-simple-1.1.jar:/usr/local/apache-cassandra/bin/../lib/jstackjunit-0.0.1.jar:/usr/local/apache-cassandra/bin/../lib/libthrift-0.9.2.jar:/usr/local/apache-cassandra/bin/../lib/log-encoder.jar:/usr/local/apache-cassandra/bin/../lib/log4j-over-slf4j-1.7.7.jar:/usr/local/apache-cassandra/bin/../lib/logback-classic-1.1.3.jar:/usr/local/apache-cassandra/bin/../lib/logback-core-1.1.3.jar:/usr/local/apache-cassandra/bin/../lib/lz4-1.3.0.jar:/usr/local/apache-cassandra/bin/../lib/metrics-core-3.1.0.jar:/usr/local/apache-cassandra/bin/../lib/metrics-jvm-3.1.0.jar:/usr/local/apache-cassandra/bin/../lib/metrics-logback-3.1.0.jar:/usr/local/apache-cassandra/bin/../lib/netty-all-4.0.44.Final.jar:/usr/local/apache-cassandra/bin/../lib/ohc-core-0.4.4.jar:/usr/local/apache-cassandra/bin/../lib/ohc-core-j8-0.4.4.jar:/usr/local/apache-cassandra/bin/../lib/reporter-config-base-3.0.3.jar:/usr/local/apache-cassandra/bin/../lib/reporter-config3-3.0.3.jar:/usr/local/apache-cassandra/bin/../lib/sigar-1.6.4.jar:/usr/local/apache-cassandra/bin/../lib/slf4j-api-1.7.7.jar:/usr/local/apache-cassandra/bin/../lib/snakeyaml-1.11.jar:/usr/local/apache-cassandra/bin/../lib/snappy-java-1.1.1.7.jar:/usr/local/apache-cassandra/bin/../lib/snowball-stemmer-1.3.0.581.1.jar:/usr/local/apache-cassandra/bin/../lib/stream-2.5.2.jar:/usr/local/apache-cassandra/bin/../lib/thrift-server-0.3.7.jar:/usr/local/apache-cassandra/bin/../lib/jsr223/*/*.jar:/usr/local/share/jolokia-agent.jar:/usr/local/share/prometheus-agent.jar:/usr/local/apache-cassandra/bin/../lib/jamm-0.3.0.jar
INFO  04:31:18 JVM Arguments: [-XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseG1GC, -XX:G1RSetUpdatingPauseTimePercent=5, -Dcassandra.max_queued_native_transport_requests=6144, -javaagent:/usr/local/share/jolokia-agent.jar=host=0.0.0.0, -javaagent:/usr/local/share/prometheus-agent.jar=0.0.0.0:1234:/etc/cassandra/prometheus.yaml, -XX:+PrintCommandLineFlags, -Dcassandra.logdir=/var/log/cassandra, -Dlogback.configurationFile=/etc/cassandra/logback-stdout.xml, -Dcassandra.migration_task_wait_in_seconds=1, -Dcassandra.ring_delay_ms=30000, -Xms4G, -Xmx4G, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/local/apache-cassandra/bin/../lib/jamm-0.3.0.jar, -Dcassandra.jmx.remote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Djava.library.path=/usr/local/apache-cassandra/bin/../lib/sigar-bin, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.local.only=false, -Dcom.sun.management.jmxremote.port=7199, -Dcom.sun.management.jmxremote.rmi.port=7199, -Djava.rmi.server.hostname=10.2.25.13, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]
WARN  04:31:18 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  04:31:18 jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
INFO  04:31:18 JMX is enabled to receive remote connections on port: 7199
WARN  04:31:18 Use of com.sun.management.jmxremote.port at startup is deprecated. Please use cassandra.jmx.remote.port instead.
WARN  04:31:18 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  04:31:18 Initializing SIGAR library
INFO  04:31:18 Checked OS settings and found them configured for optimal performance.
WARN  04:31:18 Maximum number of memory map areas per process (vm.max_map_count) 262144 is too low, recommended value: 1048575, you can change it with sysctl.
WARN  04:31:18 Directory /var/lib/cassandra/data doesn't exist
WARN  04:31:18 Directory /var/lib/cassandra/commitlog doesn't exist
WARN  04:31:18 Directory /var/lib/cassandra/saved_caches doesn't exist
WARN  04:31:18 Directory /var/lib/cassandra/hints doesn't exist
INFO  04:31:18 Overriding RING_DELAY to 30000ms
INFO  04:31:18 Initialized prepared statement caches with 16 MB (native) and 16 MB (Thrift)
INFO  04:31:19 Initializing system.IndexInfo
INFO  04:31:19 Initializing system.batches
INFO  04:31:19 Initializing system.paxos
INFO  04:31:19 Initializing system.local
INFO  04:31:19 Initializing system.peers
INFO  04:31:19 Initializing system.peer_events
INFO  04:31:19 Initializing system.range_xfers
INFO  04:31:19 Initializing system.compaction_history
INFO  04:31:19 Initializing system.sstable_activity
INFO  04:31:19 Initializing system.size_estimates
INFO  04:31:19 Initializing system.available_ranges
INFO  04:31:19 Initializing system.transferred_ranges
INFO  04:31:19 Initializing system.views_builds_in_progress
INFO  04:31:19 Initializing system.built_views
INFO  04:31:19 Initializing system.hints
INFO  04:31:19 Initializing system.batchlog
INFO  04:31:19 Initializing system.prepared_statements
INFO  04:31:19 Initializing system.schema_keyspaces
INFO  04:31:19 Initializing system.schema_columnfamilies
INFO  04:31:19 Initializing system.schema_columns
INFO  04:31:19 Initializing system.schema_triggers
INFO  04:31:19 Initializing system.schema_usertypes
INFO  04:31:19 Initializing system.schema_functions
INFO  04:31:19 Initializing system.schema_aggregates
INFO  04:31:19 Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  04:31:20 Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  04:31:20 Initializing system_schema.keyspaces
INFO  04:31:20 Initializing system_schema.tables
INFO  04:31:20 Initializing system_schema.columns
INFO  04:31:20 Initializing system_schema.triggers
INFO  04:31:20 Initializing system_schema.dropped_columns
INFO  04:31:20 Initializing system_schema.views
INFO  04:31:20 Initializing system_schema.types
INFO  04:31:20 Initializing system_schema.functions
INFO  04:31:20 Initializing system_schema.aggregates
INFO  04:31:20 Initializing system_schema.indexes
INFO  04:31:20 Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO  04:31:20 Initializing key cache with capacity of 100 MBs.
INFO  04:31:20 Initializing row cache with capacity of 128 MBs
INFO  04:31:20 OHC using Java8 Unsafe API
INFO  04:31:20 OHC using JNA OS native malloc/free
INFO  04:31:20 Initializing counter cache with capacity of 50 MBs
INFO  04:31:20 Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  04:31:20 Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap
INFO  04:31:21 Populating token metadata from system tables
INFO  04:31:21 Token metadata: 
INFO  04:31:21 Completed loading (0 ms; 1 keys) KeyCache cache
INFO  04:31:21 No commitlog files found; skipping replay
INFO  04:31:21 Populating token metadata from system tables
INFO  04:31:21 Token metadata: 
INFO  04:31:21 Preloaded 0 prepared statements
INFO  04:31:21 Cassandra version: 3.11.1
INFO  04:31:21 Thrift API version: 20.1.0
INFO  04:31:21 CQL supported versions: 3.4.4 (default: 3.4.4)
INFO  04:31:21 Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
INFO  04:31:21 Initializing index summary manager with a memory pool size of 204 MB and a resize interval of 60 minutes
INFO  04:31:21 Starting Messaging Service on /10.2.25.13:7000 (eth0)
WARN  04:31:21 No host ID found, created 36d5ecf5-7aa5-46bf-abed-a2c6c87b744a (Note: This should happen exactly once per node).
INFO  04:31:21 Loading persisted ring state
INFO  04:31:21 Starting up server gossip
INFO  04:31:21 This node will not auto bootstrap because it is configured to be a seed node.
INFO  04:31:21 Generated random tokens. tokens are [3413894215864813232, 3855987175149156191, -1095775926331436350, 1012513231931609798, -8011256876584177085, -3787278679490105099, 5218114314316090628, 8149102809669716557, -1171117166699793174, 7049387034561483428, 1067976389317038497, 3865108140212350722, 3803063857160365202, 163075974729341384, -2257875666139026307, 6273866067032649004, -704289054262218373, -5321133558441530457, 4537457561378625040, 868284053577715719, 8960662603765871347, 8841499579962700499, 1844226574236071779, -790528606534077518, 4862488131109735217, 8242444170427773522, -8821746003029640277, -2572735953223242378, 671357378151723663, 1399019572068635978, 4315952392572845761, -6797205563764685799, -1642865245648513288, -7027502332233635046, -7915507290860243159, -3229899425951837924, -8992014648060624868, -5881754894440465928, -6901580397273755394, -1119510585723029886, -8978603558284039706, 1031337551872455409, 1229309369000075095, -6699545262577669323, -6385989971900826753, -4160869164402117781, -1266252300327758095, 417476495673770887, -7935257305955435258, -6274280396599667975, -1747336984938185210, 604617627800673095, 2847004553260972381, -4233007324258983247, -1332914306385271879, -5228032367371711031, -878749555369985042, -3007461831444732295, -8415479136824651823, 4885409561426511546, -6561934435745816566, 8194187939309661212, -1694720961019246776, -9203553368407997324, 1584779180496916789, -6276742031918746559, -391363809136447270, -267014674575938739, 355095367613627084, 1469723189127711109, 3721297001711317114, -8824569635256197257, 2318818778282405853, 1850176953414935287, 3965756734421027296, 5643851085467245590, 1786691623938318016, 212266762478129204, 3975127058726983481, 7038062878690264807, -14800004541896053, 4341552454032019547, -909096958736934660, -452087630759322466, -2957932312316948778, 8287274024369720479, -9109367418001998119, 4584793572472066006, 1850706716292788449, -9074389601781892015, 1186929393077233382, -7113148664847368655, 2479025644028429066, -255914968965200800, -1946766450918507994, 7285755337503816720, -2978520040369279823, 1538244421690730075, 8021669357269042684, 758001873419702451, -1550972643189282689, -331701734514701702, -482104428939423367, 2673873122486460133, 4477806474593598641, -1092589736296292935, 2708539158089771370, 7892487509509214785, -8285654286989404472, -3180725789013264318, 7295649278852295782, 555116675006115949, -3334396285944897408, 6216727554614031484, 6628020675642007865, 5533909159422179717, -2784812312450225138, 8920384326062493071, 4081325799830033397, 7385548271058099321, -215469233477771585, -6112239581854398777, -3632761885421266219, -1812579633300969564, -2465098784951315415, 2018645010287828430, -3714356591286911511, -1382601928625875322, 858273302828806998, -8074650120785761401, 7607421796445010207, -3734401391325696979, 1609770099899993941, -6179835098758808192, 7105020359803471422, 121757796171771946, -7383386342175659153, -1112617202975232174, 2212114190630255475, 2163065071792921982, 2150705224686520278, 7142815409458368026, 7933439412305817498, -1318720717866276512, -1660237266272723434, -6272520906433615031, -4609679310390438479, 8543483886341569400, 3173502944238318586, 2520235722339917493, 9222153804448625357, -2272583955671810475, -318049119388458227, -4566866465390230693, 1415728047620175708, -1720054609013340918, 4720623162858138567, -6260123215685577265, -7939944214620170486, 8072414302513934338, -1704407030096441905, 752294573461932308, -8764085780404898072, 3442968262852677072, 2968180882273819079, 3642196832076216574, 3955877316819927745, -6055795597124767670, -215437503710158604, 283912657107995029, -779429467519117721, 1367932987508960119, 641870228057604471, 4066850428927995190, -7798969452163333704, 4264296371293000058, 2178551620441663262, 1193555633952293920, -3002395876727801518, 7679765562321501606, -1360320966562762616, 2014363940495296619, -7506730636458769499, -4003208511370200091, 1560459356016457883, -69099161774505741, -318481925831753240, -6152000973141868970, -5964513424051508294, 3535532525881628535, -5653276711046584364, -3380640632574158851, 4690371956701900107, 8203580698047545887, 7678631959050822206, 3071960945795705629, 496315076716812515, 5318126794289951839, -2828932419032926684, -5507377875574822517, -7719150213227176946, 8870603901931204663, -307975483495467188, -5099076774353572325, 7292507177362142360, -7767839479360526134, -867060852210832457, -5752419548961933697, 5656181134455395692, 6522439728609801189, -8089488940478072580, 4115624342399346063, -5216570991451431277, -2129874438658962895, 8263473782920686902, 1691732837672607391, 5538797710244583892, 7882847969479047653, -1870507187254498889, -3318323892450390788, 2163991365605803664, 4486870732791486347, 655506876033462554, 2284771957003302518, -7142588444152331791, -4481195296074926889, -5469298173213121171, -2099305222324101078, -5246840804824538749, 4282044229921266883, -4207060542059636469, 3176778826085555341, -5173739785032170964, 9164688112968671502, 1008002934081293057, -1321063084146362370, 4105130274800306997, -858145244216869767, 3970513877596685944, -8588211408336331732, -1102621813915521045, -3466387819379117428, 1421366134982350796, -2790831790482233981, 5868032873083366951, 777655239999766404, 362578342029912088, -6901943401308700691, -5554589713438036869, -6177974424109879234, 1010574863213075271, 5476400809453401433, 6491015089873971615, -6159214005510489622, -8163445961746777455, -4909747977195031936]
INFO  04:31:21 Create new Keyspace: KeyspaceMetadata{name=system_traces, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}, tables=[org.apache.cassandra.config.CFMetaData@324516a6[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,flags=[COMPOUND],params=TableParams{comment=tracing sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [client command coordinator duration request started_at parameters]],partitionKeyColumns=[session_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[client, command, session_id, coordinator, request, started_at, duration, parameters],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@1fa50099[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,flags=[COMPOUND],params=TableParams{comment=tracing events, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [activity source source_elapsed thread]],partitionKeyColumns=[session_id],clusteringColumns=[event_id],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[activity, event_id, session_id, source, thread, source_elapsed],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  04:31:21 Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO  04:31:21 Initializing system_traces.events
INFO  04:31:21 Initializing system_traces.sessions
INFO  04:31:21 Create new Keyspace: KeyspaceMetadata{name=system_distributed, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[org.apache.cassandra.config.CFMetaData@4eb4c9d1[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [coordinator exception_message exception_stacktrace finished_at parent_id range_begin range_end started_at status participants]],partitionKeyColumns=[keyspace_name, columnfamily_name],clusteringColumns=[id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[status, id, coordinator, finished_at, participants, exception_stacktrace, parent_id, range_end, range_begin, exception_message, keyspace_name, started_at, columnfamily_name],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@6d90ced1[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [exception_message exception_stacktrace finished_at keyspace_name started_at columnfamily_names options requested_ranges successful_ranges]],partitionKeyColumns=[parent_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,columnMetadata=[requested_ranges, exception_message, keyspace_name, successful_ranges, started_at, finished_at, options, exception_stacktrace, parent_id, columnfamily_names],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@4cf3f13[cfId=5582b59f-8e4e-35e1-b913-3acada51eb04,ksName=system_distributed,cfName=view_build_status,flags=[COMPOUND],params=TableParams{comment=Materialized View build status, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[] | [status]],partitionKeyColumns=[keyspace_name, view_name],clusteringColumns=[host_id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[view_name, status, keyspace_name, host_id],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  04:31:21 Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
INFO  04:31:21 Initializing system_distributed.parent_repair_history
INFO  04:31:21 Initializing system_distributed.repair_history
INFO  04:31:21 Initializing system_distributed.view_build_status
INFO  04:31:21 JOINING: Finish joining ring
INFO  04:31:21 Create new Keyspace: KeyspaceMetadata{name=system_auth, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[org.apache.cassandra.config.CFMetaData@57df42c7[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,flags=[COMPOUND],params=TableParams{comment=role definitions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [can_login is_superuser salted_hash member_of]],partitionKeyColumns=[role],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[salted_hash, member_of, role, can_login, is_superuser],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@6da96aa5[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,flags=[COMPOUND],params=TableParams{comment=role memberships lookup table, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[role],clusteringColumns=[member],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, member],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@19bb5a01[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,flags=[COMPOUND],params=TableParams{comment=permissions granted to db roles, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | [permissions]],partitionKeyColumns=[role],clusteringColumns=[resource],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, resource, permissions],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@61bb82c[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,flags=[COMPOUND],params=TableParams{comment=index of db roles with permissions granted on a resource, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@8ebffaa1, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[resource],clusteringColumns=[role],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[resource, role],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
INFO  04:31:22 Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
INFO  04:31:22 Initializing system_auth.resource_role_permissons_index
INFO  04:31:22 Initializing system_auth.role_members
INFO  04:31:22 Initializing system_auth.role_permissions
INFO  04:31:22 Initializing system_auth.roles
INFO  04:31:22 Waiting for gossip to settle...
INFO  04:31:30 No gossip backlog; proceeding
INFO  04:31:30 Netty using native Epoll event loop
INFO  04:31:30 Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO  04:31:30 Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO  04:31:30 Binding thrift service to /0.0.0.0:9160
INFO  04:31:30 Listening for thrift clients...
INFO  04:31:32 Created default superuser role 'cassandra'

[deimosfr]

Interesting, it's like if nodes were not able to talk each others.

Can you please paste your configuration (values.yaml file), I'll try.

On my side, I'm using Kubernetes 1.8. Do you have RBAC enabled?

[Vince-Cercury]

Yes I have RBAC enabled. Without helm chart, using the Cassandra Kubernetes example doc, it works fine. Usually it means the cassandra-1/2 are not able to see the seed node cassandra-0

here is my value. From master. I've changed cassandraNodeSelector and nodeSelector in two places, otherwise the pods would stay in Pending mode (I'm using kube-aws).

# Set where Cassandra local data should be located
cassandraNodeSelector:
  kubernetes.io/role: "node"

# Number of nodes
cassandraReplicaCount: 3
# Number of maximum dead nodes
cassandraMaxUnavailableNodes: 1

# Cassandra settings
cassandraImageName: "quay.io/mysocialapp/cassandra"
cassandraImageVersion: "3.11.1-8"
cassandraDC: "DC1"
cassandraConfig:
  concurrentWrites: 128
  concurrentReads: 128
  concurrentCompactors: 2
  batchSizeWarnThresholdInKb: 64
  batchSizeFailThresholdInKb: 640
  compactionThroughputMbPerSec: 150
  heapNewSize: 256M
  hintedHandoffThrottleInKb: 4096
  maxHeap: 4G
  maxHintsDeliveryThreads: 4
  memtableAllocationType: "offheap_objects"
  memtableFlushWriter: 4
  memtableCleanupThreshold: 0.2
  rowCacheSizeInMb: 128
  rowCacheSavePeriod: 14400

# Backups
cassandraBackup:
  enableBackups: false
  backupSchedule: "0 3 * * *"
  backupImageVersion: v0.1
  awsAccessKeyId: "xxx"
  awsSecretAccessKey: "yyy"
  awsPassphrase: "zzz"
  awsBucket: "bucket_name"
  #duplicityOptions: "--archive-dir /var/lib/cassandra/.duplicity --allow-source-mismatch --s3-european-buckets --s3-use-new-style --copy-links --num-retries 3 --s3-use-multiprocessing --s3-multipart-chunk-size 100 --volsize 1024 full . s3://s3.amazonaws.com/cassandra-backups/$CLUSTER_DOMAIN/$CASSANDRA_CLUSTER_NAME/$(hostname)"
  awsDestinationPath: "s3://s3-eu-west-1.amazonaws.com/${AWS_BUCKET}/$CLUSTER_DOMAIN/$CASSANDRA_CLUSTER_NAME/$(hostname)"
  restoreFolder: "/var/lib/cassandra/restore"

# Cassandra Exporter
cassandraExporter:
  enableExporter: false
  replicaCount: 1
  imageVersion: 1.0.2
  nodeSelector:
    kubernetes.io/role: "node"
  resources:
    limits:
      cpu: 300m
      memory: 500Mi
    requests:
      cpu: 300m
      memory: 500Mi
  config:
    host: 127.0.0.1:7199
    listenPort: 8080
    jvmOpts: ""

# Prometheus scraping
cassandraPrometheusScrap:
  enableScrap: false

# Alertmanager
cassandraAlertmanager:
  enableAlerts: false
  alertLabels:
    sloInterrupt:
      serviceLevel: objective
      severity: interrupt
      team: infra
      type: functional
    sloPage:
      serviceLevel: objective
      severity: page
      team: infra
      type: functional

# Cassandra Reaper Client register
cassandraReaperRegister:
  enableReaperRegister: false
  reaperServerServiceName: cassandra-reaper.svc

# Cassandra Reaper Server
cassandraReaper:
  enableReaper: false
  replicaCount: 1
  imageVersion: 1.1.0
  nodeSelector:
    kubernetes.io/role: "node"
  contactPoints: cassandra-0.cassandra
  jmxAuth:
    username: reaperUser
    password: reaperPass
  # clusterName:
  # keyspace: reaper_db
  envVariables:
    JAVA_OPTS: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -XX:+AlwaysPreTouch"
    REAPER_CASS_KEYSPACE: reaper_db
    REAPER_REPAIR_PARALELLISM: DATACENTER_AWARE
    REAPER_REPAIR_INTENSITY: 0.5
    REAPER_AUTO_SCHEDULING_ENABLED: true
    REAPER_SCHEDULE_DAYS_BETWEEN: 2
    REAPER_REPAIR_RUN_THREADS: 16
    REAPER_HANGING_REPAIR_TIMEOUT_MINS: 30
    REAPER_REPAIR_MANAGER_SCHEDULING_INTERVAL_SECONDS: 10
    REAPER_SEGMENT_COUNT: 200
    REAPER_LOGGING_ROOT_LEVEL: INFO
    REAPER_SERVER_ADMIN_PORT: 8081
    REAPER_SERVER_APP_PORT: 8080
    REAPER_METRICS_ENABLED: true
  resources:
    limits:
      cpu: 500m
      memory: 500Mi
    requests:
      cpu: 500m
      memory: 500Mi

[deimosfr]

Ok thanks, I don't see what's wrong here.

Give me a few days to test on 1.9 please

[deimosfr]

Hi,

I've added tests with 1.8 and 1.9 version and confirm it works with both version and the default values.

Have you changed things in the chart ? I mean other things than the values.yaml file ?

Thanks

[Vince-Cercury]

Hi @deimosfr I had made some change related to nodeSelector and to make it work with a different namespace. I ditched my changes and pulled from master just now, as updates were made.

The only change I kept in values.yaml in 3 places is:

    node-role.kubernetes.io/node: "true"

Replaced with

  kubernetes.io/role: "node"

Otherwise the containers wouldn't start. With kube-aws there is no such role.

I installed helm from master (after those 3 changes in values.yaml): helm install --namespace vince-helm kubernetes/

Unfortunately cassandra-0 is now in crashLoopBackoff mode. The pod events

  Normal   SuccessfulMountVolume  2m               kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  MountVolume.SetUp succeeded for volume "cassandra-data"
  Normal   SuccessfulMountVolume  2m               kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-lfjxw"
  Normal   SuccessfulMountVolume  2m               kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  MountVolume.SetUp succeeded for volume "cassandra"
  Normal   Pulling                2m               kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  pulling image "quay.io/mysocialapp/cassandra:3.11.1-9"
  Normal   Pulled                 1m               kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Successfully pulled image "quay.io/mysocialapp/cassandra:3.11.1-9"
  Normal   Created                1m (x3 over 1m)  kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Created container
  Normal   Started                1m (x3 over 1m)  kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Started container
  Normal   Pulled                 1m (x2 over 1m)  kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Container image "quay.io/mysocialapp/cassandra:3.11.1-9" already present on machine
  Warning  FailedPreStopHook      1m (x3 over 1m)  kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Exec lifecycle hook ([/usr/local/apache-cassandra/scripts/preStop.sh]) for Container "cassandra" in Pod "cassandra-0_vince-helm(177d0cc3-8624-11e8-8146-0ac31ae61cb8)" failed - error: command '/usr/local/apache-cassandra/scripts/preStop.sh' exited with 126: , message: "oci runtime error: exec failed: container \"cassandra\" does not exist\n\r\n"
  Normal   Killing                1m (x3 over 1m)  kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Killing container with id docker://cassandra:FailedPostStartHook
  Warning  FailedPostStartHook    1m (x3 over 1m)  kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Exec lifecycle hook ([/usr/local/apache-cassandra/scripts/postStart.sh]) for Container "cassandra" in Pod "cassandra-0_vince-helm(177d0cc3-8624-11e8-8146-0ac31ae61cb8)" failed - error: command '/usr/local/apache-cassandra/scripts/postStart.sh' exited with 137: , message: ""
  Warning  BackOff                1m (x3 over 1m)  kubelet, ip-10-10-84-85.ap-southeast-2.compute.internal  Back-off restarting failed container

Logs finish and then exit after:

otc_coalescing_strategy: TIMEHORIZON
otc_coalescing_window_us: 200
auto_bootstrap: true
-XX:+AlwaysPreTouch -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler -XX:G1RSetUpdatingPauseTimePercent=5 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8432648192 -XX:+ManagementServer -XX:MaxHeapSize=8432648192 -XX:MaxRAMFraction=2 -XX:+PerfDisableSharedMem -XX:+PrintCommandLineFlags -XX:+ResizeTLAB -XX:StringTableSize=1000003 -XX:ThreadPriorityPolicy=42 -XX:ThreadStackSize=256 -XX:+UnlockExperimentalVMOptions -XX:-UseBiasedLocking -XX:+UseCGroupMemoryLimitForHeap -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:+UseTLAB -XX:+UseThreadPriorities 

[deimosfr]

Hi,

Can you please please run a kubectl describe pod cassandra-0 ? I think it's an insufficient memory allocation and the output will confirm why it crashes (OOMKill).

Thanks

[Vince-Cercury]

I increased resource request to 4Gi after your comment to see if it would help. Here is the full describe pod output

Name:               cassandra-0
Namespace:          vince-test
Priority:           5
PriorityClassName:  <none>
Node:               ip-10-10-85-38.ap-southeast-2.compute.internal/10.10.85.38
Start Time:         Fri, 13 Jul 2018 20:24:19 +1000
Labels:             app=cassandra
                    chart=cassandra-0.9.2
                    controller-revision-hash=cassandra-587b767866
                    heritage=Tiller
                    release=gone-yak
                    statefulset.kubernetes.io/pod-name=cassandra-0
Annotations:        cni.projectcalico.org/podIP=10.2.2.16/32
Status:             Running
IP:                 10.2.2.16
Controlled By:      StatefulSet/cassandra
Containers:
  cassandra:
    Container ID:  docker://cdd7f25607f70059f9d3d3a6550271f9792575b79df62f124cc352377b936932
    Image:         quay.io/mysocialapp/cassandra:3.11.1-9
    Image ID:      docker-pullable://quay.io/mysocialapp/cassandra@sha256:bd2a54298d43880fd6a75471681d83c686cf96da228b81a0856b99f08549e7a2
    Ports:         7000/TCP, 7001/TCP, 7199/TCP, 9042/TCP
    Command:
      /bin/sh
      -c
      /usr/local/apache-cassandra/scripts/run_override.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 13 Jul 2018 20:28:32 +1000
      Finished:     Fri, 13 Jul 2018 20:28:35 +1000
    Ready:          False
    Restart Count:  1
    Limits:
      memory:  8000Mi
    Requests:
      memory:   4000Mi
    Readiness:  exec [/bin/bash -c /ready-probe.sh] delay=180s timeout=30s period=10s #success=1 #failure=3
    Environment:
      MAX_HEAP_SIZE:                                   
      HEAP_NEWSIZE:                                    256M
      CASSANDRA_NUM_TOKENS:                            256
      CASSANDRA_CLUSTER_NAME:                          vince-test
      CASSANDRA_RACK:                                  Rack1
      CASSANDRA_DC:                                    DC1
      CASSANDRA_LISTEN_ADDRESS:                        cassandra-0 (v1:metadata.name)
      CASSANDRA_BROADCAST_RPC_ADDRESS:                 cassandra-0 (v1:metadata.name)
      CASSANDRA_BROADCAST_ADDRESS:                     cassandra-0 (v1:metadata.name)
      POD_IP:                                           (v1:status.podIP)
      CASSANDRA_CONCURRENT_WRITES:                     128
      CASSANDRA_CONCURRENT_READS:                      128
      CASSANDRA_CONCURRENT_COMPACTORS:                 2
      CASSANDRA_YAML_batch_size_warn_threshold_in_kb:  64
      CASSANDRA_YAML_batch_size_fail_threshold_in_kb:  640
      CASSANDRA_COMPACTION_THROUGHPUT_MB_PER_SEC:      150
      CASSANDRA_YAML_hinted_handoff_throttle_in_kb:    4096
      CASSANDRA_YAML_max_hints_delivery_threads:       4
      CASSANDRA_MEMTABLE_ALLOCATION_TYPE:              offheap_objects
      CASSANDRA_MEMTABLE_FLUSH_WRITERS:                4
      CASSANDRA_MEMTABLE_CLEANUP_THRESHOLD:            0.2
      CASSANDRA_YAML_row_cache_size_in_mb:             128
      CASSANDRA_YAML_row_cache_save_period:            14400
    Mounts:
      /usr/local/apache-cassandra/scripts from cassandra (rw)
      /var/lib/cassandra from cassandra-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-97gzc (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  cassandra-data:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/data/vince-test/cassandra
    HostPathType:  
  cassandra:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      cassandra
    Optional:  false
  default-token-97gzc:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-97gzc
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/role=node
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                From                                                     Message
  ----     ------                 ----               ----                                                     -------
  Warning  FailedScheduling       4m (x12 over 5m)   default-scheduler                                        0/6 nodes are available: 2 MatchNodeSelector, 2 PodToleratesNodeTaints, 4 NodeNotReady.
  Normal   NotTriggerScaleUp      4m                 cluster-autoscaler                                       pod didn't trigger scale-up (it wouldn't fit if a new node is added)
  Warning  FailedScheduling       4m (x8 over 4m)    default-scheduler                                        0/7 nodes are available: 2 MatchNodeSelector, 2 PodToleratesNodeTaints, 5 NodeNotReady.
  Normal   Scheduled              4m                 default-scheduler                                        Successfully assigned cassandra-0 to ip-10-10-85-38.ap-southeast-2.compute.internal
  Normal   SuccessfulMountVolume  4m                 kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  MountVolume.SetUp succeeded for volume "cassandra-data"
  Normal   SuccessfulMountVolume  4m                 kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  MountVolume.SetUp succeeded for volume "cassandra"
  Normal   SuccessfulMountVolume  4m                 kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  MountVolume.SetUp succeeded for volume "default-token-97gzc"
  Normal   Pulling                4m                 kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  pulling image "quay.io/mysocialapp/cassandra:3.11.1-9"
  Normal   Pulled                 23s                kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Successfully pulled image "quay.io/mysocialapp/cassandra:3.11.1-9"
  Warning  FailedPostStartHook    16s (x2 over 20s)  kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Exec lifecycle hook ([/usr/local/apache-cassandra/scripts/postStart.sh]) for Container "cassandra" in Pod "cassandra-0_vince-test(d2f458fd-8686-11e8-adc4-0a6277d9e188)" failed - error: command '/usr/local/apache-cassandra/scripts/postStart.sh' exited with 137: nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
, message: "Waiting node to be ready\nnodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.\n"
  Warning  FailedPreStopHook  16s (x2 over 20s)  kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Exec lifecycle hook ([/usr/local/apache-cassandra/scripts/preStop.sh]) for Container "cassandra" in Pod "cassandra-0_vince-test(d2f458fd-8686-11e8-adc4-0a6277d9e188)" failed - error: command '/usr/local/apache-cassandra/scripts/preStop.sh' exited with 126: , message: "oci runtime error: exec failed: container \"cassandra\" does not exist\n\r\n"
  Normal   Killing            16s (x2 over 20s)  kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Killing container with id docker://cassandra:FailedPostStartHook
  Warning  BackOff            15s (x2 over 16s)  kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Back-off restarting failed container
  Normal   Pulled             2s (x2 over 19s)   kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Container image "quay.io/mysocialapp/cassandra:3.11.1-9" already present on machine
  Normal   Created            2s (x3 over 22s)   kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Created container
  Normal   Started            1s (x3 over 22s)   kubelet, ip-10-10-85-38.ap-southeast-2.compute.internal  Started container

[deimosfr]

It's crital clear here:

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 13 Jul 2018 20:28:32 +1000
      Finished:     Fri, 13 Jul 2018 20:28:35 +1000

Is this ok with 4G now ?

Once it will be ok, regarding AWS, don't you you have firewalls or stuffs like that denying ports access between nodes?

[Vince-Cercury]

That was with 4GB. It's strange, I've used Cassandra with 4G and never had any issue. I can try with more on Monday

[Vince-Cercury]

Good news! I got it working with:

  limits:
    memory: 20000Mi
  requests:
    memory: 8000Mi

I set the memory limit very high on purpose to rule this issue out. So I think we have to set the right values in values.yaml so that anyone trying out this chart for the first time is not confused about why it does not work. If it didn't work for me, it will likely fail for others.

So e have to figure out the minimum memory request and limit and commit that in values.yaml

I will continue playing with this chart. ( next I need to enable Ec2Snitch)

[deimosfr]

On my side, with 2.5G of RAM it works (per node). Do you created/imported data or is the cluster completely empty?

By the way...good news :)

[Vince-Cercury]

completely empty.

[deimosfr]

Really strange :/. I'm wondering if there is not a different behavior inside a VM on EC2. Can you please tell me the minimum of working RAM for you please ?

Silly question: can you confirm that you're not running the 3 instances in the same EC2 VM ? Thanks

[Vince-Cercury]

I'm running on EC2 M4.2xlarge each. Each of the 2 have 32GB memory and 8 CPU. It is strange. It should definitely work with less than 6GB of memory. I've been using Cassandra with 4GB with the official Cassandra example

[deimosfr]

And I can confirm me too. Also, with the vagrant tests inside the repo, I can confirm it.

[Vince-Cercury]

More testing:

WORKS:

• very very high memory limit (20GB)

  memory: 20000Mi
  requests:
  memory: 8000Mi

• high resource memory limit (10GB)

  cassandraResources:
  limits:
  memory: 5000Mi
  requests:
  memory: 2400Mi

• 9GB resource limit

  limits:
  memory: 9000Mi
  requests:
  memory: 2400Mi

• no memory limit

  #  limits:
  #    memory: 2400Mi
  requests:
  memory: 2400Mi

DOES NOT WORKS

• original setup

  cassandraResources:
  limits:
  memory: 2400Mi
  requests:
  memory: 2400Mi

• slightly higher resource memory limit (5GB)

  cassandraResources:
  limits:
  memory: 5000Mi
  requests:
  memory: 2400Mi

• slightly higher resource memory limit (8GB)

  cassandraResources:
  limits:
  memory: 8000Mi
  requests:
  memory: 2400Mi

What I can conclude from my tests:

• the problem is with memory limit, not memory request
• the problem happens whenever the memory limit is <9GB regardless of memory request.

I had a look at memory utilisation, and found that the cassandra-0 pods consumes 8.5GB of memory every time it starts See the memory utilisation graph: https://ibb.co/mQyr1y (each spike is a different experiment, and I shut it down quickly to try something else)

I did a describe of the statefulset and found this:

        env:
        - name: MAX_HEAP_SIZE
        - name: HEAP_NEWSIZE
          value: 256M

It looks like MAX_HEAP_SIZE does not have any value set. What that means is that Cassandra will try to calculate the HEAP size depending on hardware available. As I run on large EC2 instances, it must use a formula to set the HEAP size at 8.5GB. And it hits the memory limit.

Next I looked at the statefulset template to find:

            - name: MAX_HEAP_SIZE
              value: {{ .Values.cassandraConfig.maxHeap }}

cassandraConfig.maxHeap does not exist in https://github.com/MySocialApp/kubernetes-helm-chart-cassandra/blob/master/kubernetes/values.yaml

That's probably why it fails.

So would the fix be to set a default value recommended value for cassandraConfig.maxHeap in values.yaml?

Ideally, as seen done by other CHART, it would be good to have two values.yaml

• testing-values.yaml (for testing)
• production-min-values.yaml

I understand it depends on usage but that could be helpful for beginners.

[deimosfr]

Thanks for your tests. I think I have an idea. I'm currently working on memory assignments on Java in containers and I'm wondering if you enabled this:

cassandraJvmOptions: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2"

[Vince-Cercury]

I don't remember enabling this.

But setting cassandraConfig.maxHeap value to something lower than the resources.limit made it work. I think it would be good to have cassandraConfig.maxHeap set to 2048Mb in values.yaml to ensure it's lower than the 2400Mb resource memory limit

[deimosfr]

Can you confirm you're using 3.11.1-9 image version ? I remember this issue where XMX was automatically set to 8G if CASSANDRA_MAX_HEAP was not set

[Vince-Cercury]

I'm indeed using cassandraImageVersion: "3.11.1-9" (default in values.yaml)

[deimosfr]

Sorry for the late answer, you can find an update here: https://github.com/MySocialApp/kubernetes-helm-chart-cassandra/blob/master/kubernetes/templates/statefulset.yaml#L45

Can you please make a try ? Hope this will solve the problem you encountered.

[deimosfr]

Hi, any news ?