Closed mmanciop closed 3 years ago
Cassandra units occasionally faily to get Cassandra up and running after the application joins a prometheus_scrape relation.
prometheus_scrape
Juju status:
Model Controller Cloud/Region Version SLA Timestamp lma development microk8s/localhost 2.9.9 unsupported 12:35:44+02:00 App Version Status Scale Charm Store Channel Rev OS Address Message alertmanager active 1 alertmanager-k8s charmhub edge 2 kubernetes 10.152.183.195 cassandra active 1 cassandra-k8s charmhub edge 2 kubernetes 10.152.183.124 grafana active 1 grafana-k8s charmhub edge 7 kubernetes 10.152.183.29 prometheus active 1 prometheus-k8s charmhub edge 4 kubernetes 10.152.183.244 Unit Workload Agent Address Ports Message alertmanager/0* active idle 10.1.151.74 cassandra/0* active idle 10.1.151.80 grafana/0* active idle 10.1.151.77 prometheus/0* active idle 10.1.151.78 Relation provider Requirer Interface Type Message alertmanager:alerting prometheus:alertmanager alertmanager_dispatch regular alertmanager:replicas alertmanager:replicas alertmanager_replica peer cassandra:cassandra-peers cassandra:cassandra-peers cassandra_peers peer grafana:grafana-peers grafana:grafana-peers grafana_peers peer grafana:grafana-source prometheus:grafana-source grafana_datasource regular prometheus:monitoring cassandra:monitoring prometheus_scrape regular
The cassandra container has no Cassandra process running, because it fails to bootstrap, see logs:
cassandra
michele@boombox:~$ microk8s.kubectl logs cassandra-0 -n lma -c cassandra 2021-07-30T10:30:09.478Z [pebble] Started daemon. 2021-07-30T10:30:35.117Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra.yaml 222.738µs 200 2021-07-30T10:30:35.158Z [pebble] POST /v1/files 3.177701ms 200 2021-07-30T10:30:35.158Z [pebble] GET /v1/plan?format=yaml 61.332µs 200 2021-07-30T10:30:35.159Z [pebble] POST /v1/layers 138.781µs 200 2021-07-30T10:30:35.163Z [pebble] GET /v1/services?names=cassandra 54.407µs 200 2021-07-30T10:30:35.166Z [pebble] POST /v1/services 2.909499ms 202 2021-07-30T10:30:37.005Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra.yaml 380.593µs 200 2021-07-30T10:30:37.011Z [pebble] GET /v1/plan?format=yaml 138.732µs 200 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns; 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I 2021-07-30T10:30:37.274Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I 2021-07-30T10:30:37.275Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I 2021-07-30T10:30:37.275Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I 2021-07-30T10:30:37.275Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I 2021-07-30T10:30:37.275Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B 2021-07-30T10:30:37.564Z [cassandra] INFO [main] 2021-07-30 10:30:37,562 YamlConfigurationLoader.java:92 - Configuration location: file:/etc/cassandra/cassandra.yaml 2021-07-30T10:30:37.677Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra.yaml 116.003µs 200 2021-07-30T10:30:37.682Z [pebble] GET /v1/plan?format=yaml 127.168µs 200 2021-07-30T10:30:37.706Z [cassandra] INFO [main] 2021-07-30 10:30:37,705 Config.java:539 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=<REDACTED>; cluster_name=juju-cluster-cassandra; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@525b461a; disk_access_mode=auto; disk_failure_policy=ignore; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_drop_compact_storage=false; enable_materialized_views=true; enable_sasi_indexes=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=<REDACTED>; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=0; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=true; internode_authenticator=null; internode_compression=none; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.1.151.79; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_flush_in_batches_legacy=true; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=-2147483648; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; repair_session_max_tree_depth=18; request_scheduler=null; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=null; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=null; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.1.151.79}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_on_duplicate_row_detection=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@58c1c010; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=0; write_request_timeout_in_ms=2000] 2021-07-30T10:30:37.706Z [cassandra] INFO [main] 2021-07-30 10:30:37,706 DatabaseDescriptor.java:381 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap 2021-07-30T10:30:37.706Z [cassandra] INFO [main] 2021-07-30 10:30:37,706 DatabaseDescriptor.java:439 - Global memtable on-heap threshold is enabled at 1969MB 2021-07-30T10:30:37.706Z [cassandra] INFO [main] 2021-07-30 10:30:37,706 DatabaseDescriptor.java:443 - Global memtable off-heap threshold is enabled at 1969MB 2021-07-30T10:30:37.799Z [cassandra] INFO [main] 2021-07-30 10:30:37,798 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000. 2021-07-30T10:30:37.799Z [cassandra] INFO [main] 2021-07-30 10:30:37,799 DatabaseDescriptor.java:775 - Back-pressure is disabled with strategy null. 2021-07-30T10:30:37.804Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra.yaml 82.637µs 200 2021-07-30T10:30:37.808Z [pebble] GET /v1/plan?format=yaml 156.254µs 200 2021-07-30T10:30:37.886Z [cassandra] INFO [main] 2021-07-30 10:30:37,885 GossipingPropertyFileSnitch.java:64 - Loaded cassandra-topology.properties for compatibility 2021-07-30T10:30:37.922Z [cassandra] INFO [main] 2021-07-30 10:30:37,922 JMXServerUtils.java:253 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi 2021-07-30T10:30:37.925Z [cassandra] INFO [main] 2021-07-30 10:30:37,925 CassandraDaemon.java:490 - Hostname: cassandra-0.cassandra-endpoints.lma.svc.cluster.local 2021-07-30T10:30:37.925Z [cassandra] INFO [main] 2021-07-30 10:30:37,925 CassandraDaemon.java:497 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_292 2021-07-30T10:30:37.926Z [cassandra] INFO [main] 2021-07-30 10:30:37,926 CassandraDaemon.java:498 - Heap size: 7.691GiB/7.691GiB 2021-07-30T10:30:37.926Z [cassandra] INFO [main] 2021-07-30 10:30:37,926 CassandraDaemon.java:503 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4539776(4433K) committed = 4587520(4480K) max = 251658240(245760K) 2021-07-30T10:30:37.926Z [cassandra] INFO [main] 2021-07-30 10:30:37,926 CassandraDaemon.java:503 - Metaspace Non-heap memory: init = 0(0K) used = 20734664(20248K) committed = 21364736(20864K) max = -1(-1K) 2021-07-30T10:30:37.926Z [cassandra] INFO [main] 2021-07-30 10:30:37,926 CassandraDaemon.java:503 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2520744(2461K) committed = 2752512(2688K) max = 1073741824(1048576K) 2021-07-30T10:30:37.926Z [cassandra] INFO [main] 2021-07-30 10:30:37,926 CassandraDaemon.java:503 - Par Eden Space Heap memory: init = 671088640(655360K) used = 161069288(157294K) committed = 671088640(655360K) max = 671088640(655360K) 2021-07-30T10:30:37.927Z [cassandra] INFO [main] 2021-07-30 10:30:37,926 CassandraDaemon.java:503 - Par Survivor Space Heap memory: init = 83886080(81920K) used = 0(0K) committed = 83886080(81920K) max = 83886080(81920K) 2021-07-30T10:30:37.927Z [cassandra] INFO [main] 2021-07-30 10:30:37,927 CassandraDaemon.java:503 - CMS Old Gen Heap memory: init = 7503609856(7327744K) used = 0(0K) committed = 7503609856(7327744K) max = 7503609856(7327744K) 2021-07-30T10:30:37.927Z [cassandra] INFO [main] 2021-07-30 10:30:37,927 CassandraDaemon.java:505 - Classpath: /etc/cassandra:/opt/cassandra/build/classes/main:/opt/cassandra/build/classes/thrift:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.6.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-3.11.11.jar:/opt/cassandra/lib/apache-cassandra-thrift-3.11.11.jar:/opt/cassandra/lib/asm-5.0.4.jar:/opt/cassandra/lib/caffeine-2.2.6.jar:/opt/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.1.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/compress-lzf-0.8.4.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/cassandra/lib/disruptor-3.0.1.jar:/opt/cassandra/lib/ecj-4.4.2.jar:/opt/cassandra/lib/guava-18.0.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.5.4.jar:/opt/cassandra/lib/jackson-annotations-2.9.10.jar:/opt/cassandra/lib/jackson-core-2.9.10.jar:/opt/cassandra/lib/jackson-databind-2.9.10.8.jar:/opt/cassandra/lib/jamm-0.3.0.jar:/opt/cassandra/lib/javax.inject-1.jar:/opt/cassandra/lib/jbcrypt-0.3m.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/cassandra/lib/jctools-core-1.2.1.jar:/opt/cassandra/lib/jflex-1.6.0.jar:/opt/cassandra/lib/jna-4.2.2.jar:/opt/cassandra/lib/joda-time-2.4.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/libthrift-0.9.2.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/cassandra/lib/logback-classic-1.1.3.jar:/opt/cassandra/lib/logback-core-1.1.3.jar:/opt/cassandra/lib/lz4-1.3.0.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/netty-all-4.0.44.Final.jar:/opt/cassandra/lib/ohc-core-0.4.4.jar:/opt/cassandra/lib/ohc-core-j8-0.4.4.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/slf4j-api-1.7.7.jar:/opt/cassandra/lib/snakeyaml-1.11.jar:/opt/cassandra/lib/snappy-java-1.1.1.7.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/thrift-server-0.3.7.jar:/opt/cassandra/lib/jsr223/*/*.jar::/opt/cassandra/lib/jamm-0.3.0.jar 2021-07-30T10:30:37.927Z [cassandra] INFO [main] 2021-07-30 10:30:37,927 CassandraDaemon.java:507 - JVM Arguments: [-Xloggc:/opt/cassandra/logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms7955M, -Xmx7955M, -Xmn800M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/local/lib/libjemalloc.so, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes] 2021-07-30T10:30:37.961Z [cassandra] WARN [main] 2021-07-30 10:30:37,961 NativeLibrary.java:189 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root. 2021-07-30T10:30:37.961Z [cassandra] INFO [main] 2021-07-30 10:30:37,961 StartupChecks.java:140 - jemalloc seems to be preloaded from /usr/local/lib/libjemalloc.so 2021-07-30T10:30:37.961Z [cassandra] WARN [main] 2021-07-30 10:30:37,961 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. 2021-07-30T10:30:37.962Z [cassandra] INFO [main] 2021-07-30 10:30:37,962 SigarLibrary.java:44 - Initializing SIGAR library 2021-07-30T10:30:37.967Z [cassandra] INFO [main] 2021-07-30 10:30:37,967 SigarLibrary.java:53 - Could not initialize SIGAR library No such file or directory 2021-07-30T10:30:37.967Z [cassandra] INFO [main] 2021-07-30 10:30:37,967 SigarLibrary.java:185 - Sigar could not be initialized, test for checking degraded mode omitted. 2021-07-30T10:30:37.968Z [cassandra] WARN [main] 2021-07-30 10:30:37,967 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl. 2021-07-30T10:30:37.971Z [cassandra] WARN [main] 2021-07-30 10:30:37,971 StartupChecks.java:332 - Directory /opt/cassandra/data/data doesn't exist 2021-07-30T10:30:37.973Z [cassandra] WARN [main] 2021-07-30 10:30:37,973 StartupChecks.java:332 - Directory /opt/cassandra/data/commitlog doesn't exist 2021-07-30T10:30:37.974Z [cassandra] WARN [main] 2021-07-30 10:30:37,974 StartupChecks.java:332 - Directory /opt/cassandra/data/saved_caches doesn't exist 2021-07-30T10:30:37.974Z [cassandra] WARN [main] 2021-07-30 10:30:37,974 StartupChecks.java:332 - Directory /opt/cassandra/data/hints doesn't exist 2021-07-30T10:30:38.000Z [cassandra] INFO [main] 2021-07-30 10:30:38,000 QueryProcessor.java:116 - Initialized prepared statement caches with 30 MB (native) and 30 MB (Thrift) 2021-07-30T10:30:38.237Z [cassandra] INFO [main] 2021-07-30 10:30:38,236 ColumnFamilyStore.java:432 - Initializing system.IndexInfo 2021-07-30T10:30:38.599Z [cassandra] INFO [main] 2021-07-30 10:30:38,598 ColumnFamilyStore.java:432 - Initializing system.batches 2021-07-30T10:30:38.602Z [cassandra] INFO [main] 2021-07-30 10:30:38,602 ColumnFamilyStore.java:432 - Initializing system.paxos 2021-07-30T10:30:38.608Z [cassandra] INFO [main] 2021-07-30 10:30:38,608 ColumnFamilyStore.java:432 - Initializing system.local 2021-07-30T10:30:38.611Z [cassandra] INFO [main] 2021-07-30 10:30:38,611 ColumnFamilyStore.java:432 - Initializing system.peers 2021-07-30T10:30:38.615Z [cassandra] INFO [main] 2021-07-30 10:30:38,614 ColumnFamilyStore.java:432 - Initializing system.peer_events 2021-07-30T10:30:38.618Z [cassandra] INFO [main] 2021-07-30 10:30:38,618 ColumnFamilyStore.java:432 - Initializing system.range_xfers 2021-07-30T10:30:38.621Z [cassandra] INFO [main] 2021-07-30 10:30:38,620 ColumnFamilyStore.java:432 - Initializing system.compaction_history 2021-07-30T10:30:38.624Z [cassandra] INFO [main] 2021-07-30 10:30:38,624 ColumnFamilyStore.java:432 - Initializing system.sstable_activity 2021-07-30T10:30:38.628Z [cassandra] INFO [main] 2021-07-30 10:30:38,628 ColumnFamilyStore.java:432 - Initializing system.size_estimates 2021-07-30T10:30:38.631Z [cassandra] INFO [main] 2021-07-30 10:30:38,631 ColumnFamilyStore.java:432 - Initializing system.available_ranges 2021-07-30T10:30:38.634Z [cassandra] INFO [main] 2021-07-30 10:30:38,634 ColumnFamilyStore.java:432 - Initializing system.transferred_ranges 2021-07-30T10:30:38.638Z [cassandra] INFO [main] 2021-07-30 10:30:38,638 ColumnFamilyStore.java:432 - Initializing system.views_builds_in_progress 2021-07-30T10:30:38.641Z [cassandra] INFO [main] 2021-07-30 10:30:38,641 ColumnFamilyStore.java:432 - Initializing system.built_views 2021-07-30T10:30:38.644Z [cassandra] INFO [main] 2021-07-30 10:30:38,644 ColumnFamilyStore.java:432 - Initializing system.hints 2021-07-30T10:30:38.648Z [cassandra] INFO [main] 2021-07-30 10:30:38,648 ColumnFamilyStore.java:432 - Initializing system.batchlog 2021-07-30T10:30:38.652Z [cassandra] INFO [main] 2021-07-30 10:30:38,652 ColumnFamilyStore.java:432 - Initializing system.prepared_statements 2021-07-30T10:30:38.655Z [cassandra] INFO [main] 2021-07-30 10:30:38,654 ColumnFamilyStore.java:432 - Initializing system.schema_keyspaces 2021-07-30T10:30:38.658Z [cassandra] INFO [main] 2021-07-30 10:30:38,658 ColumnFamilyStore.java:432 - Initializing system.schema_columnfamilies 2021-07-30T10:30:38.661Z [cassandra] INFO [main] 2021-07-30 10:30:38,661 ColumnFamilyStore.java:432 - Initializing system.schema_columns 2021-07-30T10:30:38.664Z [cassandra] INFO [main] 2021-07-30 10:30:38,664 ColumnFamilyStore.java:432 - Initializing system.schema_triggers 2021-07-30T10:30:38.667Z [cassandra] INFO [main] 2021-07-30 10:30:38,667 ColumnFamilyStore.java:432 - Initializing system.schema_usertypes 2021-07-30T10:30:38.670Z [cassandra] INFO [main] 2021-07-30 10:30:38,670 ColumnFamilyStore.java:432 - Initializing system.schema_functions 2021-07-30T10:30:38.673Z [cassandra] INFO [main] 2021-07-30 10:30:38,673 ColumnFamilyStore.java:432 - Initializing system.schema_aggregates 2021-07-30T10:30:38.677Z [cassandra] INFO [main] 2021-07-30 10:30:38,677 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized 2021-07-30T10:30:38.723Z [cassandra] INFO [main] 2021-07-30 10:30:38,722 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds 2021-07-30T10:30:38.733Z [cassandra] INFO [main] 2021-07-30 10:30:38,733 ColumnFamilyStore.java:432 - Initializing system_schema.keyspaces 2021-07-30T10:30:38.737Z [cassandra] INFO [main] 2021-07-30 10:30:38,736 ColumnFamilyStore.java:432 - Initializing system_schema.tables 2021-07-30T10:30:38.739Z [cassandra] INFO [main] 2021-07-30 10:30:38,739 ColumnFamilyStore.java:432 - Initializing system_schema.columns 2021-07-30T10:30:38.742Z [cassandra] INFO [main] 2021-07-30 10:30:38,742 ColumnFamilyStore.java:432 - Initializing system_schema.triggers 2021-07-30T10:30:38.744Z [cassandra] INFO [main] 2021-07-30 10:30:38,744 ColumnFamilyStore.java:432 - Initializing system_schema.dropped_columns 2021-07-30T10:30:38.747Z [cassandra] INFO [main] 2021-07-30 10:30:38,747 ColumnFamilyStore.java:432 - Initializing system_schema.views 2021-07-30T10:30:38.751Z [cassandra] INFO [main] 2021-07-30 10:30:38,750 ColumnFamilyStore.java:432 - Initializing system_schema.types 2021-07-30T10:30:38.754Z [cassandra] INFO [main] 2021-07-30 10:30:38,754 ColumnFamilyStore.java:432 - Initializing system_schema.functions 2021-07-30T10:30:38.757Z [cassandra] INFO [main] 2021-07-30 10:30:38,757 ColumnFamilyStore.java:432 - Initializing system_schema.aggregates 2021-07-30T10:30:38.760Z [cassandra] INFO [main] 2021-07-30 10:30:38,760 ColumnFamilyStore.java:432 - Initializing system_schema.indexes 2021-07-30T10:30:38.762Z [cassandra] INFO [main] 2021-07-30 10:30:38,762 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized 2021-07-30T10:30:38.994Z [cassandra] INFO [MemtableFlushWriter:1] 2021-07-30 10:30:38,994 CacheService.java:100 - Initializing key cache with capacity of 100 MBs. 2021-07-30T10:30:38.997Z [cassandra] INFO [MemtableFlushWriter:1] 2021-07-30 10:30:38,997 CacheService.java:122 - Initializing row cache with capacity of 0 MBs 2021-07-30T10:30:38.998Z [cassandra] INFO [MemtableFlushWriter:1] 2021-07-30 10:30:38,997 CacheService.java:151 - Initializing counter cache with capacity of 50 MBs 2021-07-30T10:30:38.998Z [cassandra] INFO [MemtableFlushWriter:1] 2021-07-30 10:30:38,998 CacheService.java:162 - Scheduling counter cache save to every 7200 seconds (going to save all keys). 2021-07-30T10:30:39.233Z [cassandra] INFO [CompactionExecutor:2] 2021-07-30 10:30:39,232 BufferPool.java:234 - Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap 2021-07-30T10:30:39.310Z [cassandra] INFO [main] 2021-07-30 10:30:39,310 StorageService.java:681 - Populating token metadata from system tables 2021-07-30T10:30:39.327Z [cassandra] INFO [main] 2021-07-30 10:30:39,326 StorageService.java:688 - Token metadata: 2021-07-30T10:30:39.336Z [cassandra] INFO [pool-3-thread-1] 2021-07-30 10:30:39,336 AutoSavingCache.java:174 - Completed loading (0 ms; 5 keys) KeyCache cache 2021-07-30T10:30:39.344Z [cassandra] INFO [main] 2021-07-30 10:30:39,344 CommitLog.java:142 - No commitlog files found; skipping replay 2021-07-30T10:30:39.344Z [cassandra] INFO [main] 2021-07-30 10:30:39,344 StorageService.java:681 - Populating token metadata from system tables 2021-07-30T10:30:39.348Z [cassandra] INFO [main] 2021-07-30 10:30:39,347 StorageService.java:688 - Token metadata: 2021-07-30T10:30:39.382Z [cassandra] INFO [main] 2021-07-30 10:30:39,382 QueryProcessor.java:163 - Preloaded 0 prepared statements 2021-07-30T10:30:39.382Z [cassandra] INFO [main] 2021-07-30 10:30:39,382 StorageService.java:699 - Cassandra version: 3.11.11 2021-07-30T10:30:39.382Z [cassandra] INFO [main] 2021-07-30 10:30:39,382 StorageService.java:700 - Thrift API version: 20.1.0 2021-07-30T10:30:39.383Z [cassandra] INFO [main] 2021-07-30 10:30:39,383 StorageService.java:701 - CQL supported versions: 3.4.4 (default: 3.4.4) 2021-07-30T10:30:39.383Z [cassandra] INFO [main] 2021-07-30 10:30:39,383 StorageService.java:703 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4) 2021-07-30T10:30:39.398Z [cassandra] INFO [main] 2021-07-30 10:30:39,398 IndexSummaryManager.java:87 - Initializing index summary manager with a memory pool size of 393 MB and a resize interval of 60 minutes 2021-07-30T10:30:39.404Z [cassandra] INFO [main] 2021-07-30 10:30:39,403 MessagingService.java:750 - Starting Messaging Service on /10.1.151.79:7000 (eth0) 2021-07-30T10:30:39.406Z [cassandra] WARN [main] 2021-07-30 10:30:39,406 SystemKeyspace.java:1142 - No host ID found, created 2322533e-eaed-40a2-bf73-0d41a0681baf (Note: This should happen exactly once per node). 2021-07-30T10:30:39.410Z [cassandra] INFO [main] 2021-07-30 10:30:39,409 StorageService.java:631 - Unable to gossip with any peers but continuing anyway since node is in its own seed list 2021-07-30T10:30:39.416Z [cassandra] INFO [main] 2021-07-30 10:30:39,415 StorageService.java:785 - Loading persisted ring state 2021-07-30T10:30:39.416Z [cassandra] INFO [main] 2021-07-30 10:30:39,416 StorageService.java:916 - Starting up server gossip 2021-07-30T10:30:39.455Z [cassandra] INFO [main] 2021-07-30 10:30:39,455 StorageService.java:999 - This node will not auto bootstrap because it is configured to be a seed node. 2021-07-30T10:30:39.458Z [cassandra] INFO [main] 2021-07-30 10:30:39,458 BootStrapper.java:228 - Generated random tokens. tokens are [-7515025171885157771, -3481117073832839324, -5793872627564805570, 8082324176470954814, 3886611687267486770, 1751319676869757007, -3920030009084385463, 3567610181979181292, 2915724356772550082, 3197509724053885565, 5799015000292704924, -6984247978537458053, -5229127153337318899, 829929245006856329, -2264529389713285448, -3691688312689471112, -5827548807028509232, -4091389403461996801, 6029683724817958766, 4859379886870114462, -2617307422951713869, -3450641287542398333, 4017648167500991543, 698543379993770662, 1362181508894792019, 5323816708881365232, -2306211076432515366, 7941488083743814363, -3813396041477850505, 8216512424402815401, -6980188723717057047, 8017192731196945382, 4980532483221922936, 8948107076505255637, 8061016511513833223, -5962414159588833507, -1881180874138899733, 6241722215677341057, -2966316568503459156, 6728505851579814861, -7623069530130233180, -5250500431365446737, -8909553939896956750, 7252457102545716293, -2410013359603297370, -791694258500179854, -5150969540634437361, 4503321729228818, 4919805081900531823, -2449147699705287194, 952027041395573779, -420913782980519114, 6014946379934081082, 7748592787010436373, -2569687181033384564, 7533147522863845829, 6441747903748719380, 1629596584634906584, -5763939245711622705, -6267972153616014253, 4165801152689474384, -4280531430930878783, 4463191194939930175, 3336965917361609482, -4710933175275751369, -4095485983293146203, -3446117672392968123, -8215641521816068398, -4786830916656724114, -8117915210699576967, 4500988289786013465, 6832556149427221829, -5329242249210555585, 8717575083472601373, -7436477051335907044, 5348845922057358638, 3062674649788658550, -9145443241267791133, -5351106398419910935, -8866709393112997817, -7716744961899065703, -125212629665291188, 3015133447037397888, 1084528721627586525, 2142562692999398729, -349908626783588159, 55662006772186396, 4676486170577655167, 1693679554962858816, 3407234247437496715, -6104896112650439576, -9191344013086076927, 8400052647090090791, -1561150731128299653, -6372137469794023149, 7349269048571349671, -7894768813353222894, -3376160057072329495, 2080954615527300905, 689829941308378866, 8573354300400501385, 8345999131076610417, 8294668850668785928, 4354417228014801016, 6000854885689895444, 8829219794978044743, 9124646087708853811, 1414871190981790658, 8539319733114051054, -9180992805235843225, 1084217748153477755, 92856920715862908, 2608319070247798896, 1799333518458501754, -4854644171315516084, 5449173391719789019, 2475094123546077137, 5264500555757927788, -46307689422446766, 193736986497447054, 7424100322874441182, -3669108620437093813, 9004589953743268408, 916115775814806353, -8611792544064646509, -1624652661543119313, 4235648202961166681, 8294868448254797863, 747454481042772217, 2448326727875220959, -6915625528332171768, 8794125398495102178, 4851801699707585034, -9173347592744757736, -4317081658022990018, 878210159094624426, 861868433848994978, 1811202562653561884, 7059735326008244219, -6743795629079693815, -3263951448441190044, 130027830926478386, -3433432895522945092, -3348437105133465261, 1431919903271756851, -7748852675870803666, -8315463037473688645, -1206314848545569706, -3545210509716217784, -4585603204205770970, 4002406989521184178, 3283952850905062466, 5791913373894084883, 135462329316659654, 579798863142072439, -7479867906948377142, -4366929052595726918, 5176100133020652469, 2833473401838284805, 6109248761816926319, -7526294261526658747, -4923115726835688581, 3949110589017730961, -7027824060760470004, -152567521884204594, 4368557060256884188, -5212912688960708589, 7584666537748149348, 262093727423294193, -6022757259324666706, -5284497744379185809, -5604529569955527427, -4677642665103896835, -673764527754066556, 3726637870108897253, 7113156902033174039, 8223598570428936138, -2934852260688185812, 5001118483672744480, -8717965300787693934, 4465604061516531085, 5851833035065417885, -7326125949717518018, 1920276654888917900, -7898368501714637181, -1834382385289919095, 3232809120353825329, -7091783245990453519, -3970572536195423764, 2730188220868571127, -8018421687832555074, -3249040369195698501, 7356452942405299612, 8153314412998449209, 722857411574816605, 2644120677517161002, -6924770800200015140, -7915410071986537370, -4377688885776503927, 469040546150550118, 8320390870243833543, -3241488587498082962, -4835324358866835821, 8323124307805603704, 8234939623124657471, 7495971710562444632, -405175226770157550, 7434471186604178118, 4410098101972899574, -7450215445184400355, -8895957255903069706, -2393718669172034265, -4630448113780350768, 6274172258210083564, 1085618434295798108, 312889125745091813, -9125295161721286773, 2992983658272054565, -7158210432813539790, 157530492989567314, -9004795682803585730, -3613155462790511036, -792686323644714908, -6141140892155271018, 8712250261787132029, 8098267121502562248, -6935519018090499698, -4002818402397768502, -6134212548868422964, -6800780991386989877, 1193801360071855658, 6969956984698222892, 2927522416186852762, -7259870775603953164, 6402345350814219397, -5644920650162570965, 835823516481365383, 8413487862110917778, 6530446225672813383, 3281403157694207403, -8139362464337785119, 2097045715693173419, -2941254540162986547, 1730914033967537050, 1906144025826873940, 4129051106057288005, -1622255861844762745, 6815511165905413296, 4613993373924225442, -6445760139270514337, 146044185055253026, 6578038749734134642, 8664869084764918252, 3469515990041151303, 5316366030137533645, -1034734204606963819] 2021-07-30T10:30:39.728Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,728 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized 2021-07-30T10:30:39.729Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,729 ColumnFamilyStore.java:432 - Initializing system_traces.events 2021-07-30T10:30:39.734Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,734 ColumnFamilyStore.java:432 - Initializing system_traces.sessions 2021-07-30T10:30:39.739Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,739 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized 2021-07-30T10:30:39.740Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,739 ColumnFamilyStore.java:432 - Initializing system_distributed.parent_repair_history 2021-07-30T10:30:39.744Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,744 ColumnFamilyStore.java:432 - Initializing system_distributed.repair_history 2021-07-30T10:30:39.748Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,748 ColumnFamilyStore.java:432 - Initializing system_distributed.view_build_status 2021-07-30T10:30:39.752Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,752 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized 2021-07-30T10:30:39.753Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,753 ColumnFamilyStore.java:432 - Initializing system_auth.resource_role_permissons_index 2021-07-30T10:30:39.757Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,757 ColumnFamilyStore.java:432 - Initializing system_auth.role_members 2021-07-30T10:30:39.761Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,761 ColumnFamilyStore.java:432 - Initializing system_auth.role_permissions 2021-07-30T10:30:39.764Z [cassandra] INFO [MigrationStage:1] 2021-07-30 10:30:39,764 ColumnFamilyStore.java:432 - Initializing system_auth.roles 2021-07-30T10:30:39.782Z [cassandra] INFO [main] 2021-07-30 10:30:39,781 StorageService.java:1564 - JOINING: Finish joining ring 2021-07-30T10:30:39.857Z [cassandra] INFO [main] 2021-07-30 10:30:39,857 AuthCache.java:177 - (Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000) 2021-07-30T10:30:39.858Z [cassandra] INFO [main] 2021-07-30 10:30:39,858 Gossiper.java:1832 - Waiting for gossip to settle... 2021-07-30T10:30:42.886Z [cassandra] INFO [ScheduledTasks:1] 2021-07-30 10:30:42,886 TokenMetadata.java:520 - Updating topology for all endpoints that have changed 2021-07-30T10:30:47.859Z [cassandra] INFO [main] 2021-07-30 10:30:47,859 Gossiper.java:1863 - No gossip backlog; proceeding 2021-07-30T10:30:47.991Z [cassandra] INFO [main] 2021-07-30 10:30:47,991 NativeTransportService.java:68 - Netty using native Epoll event loop 2021-07-30T10:30:48.015Z [cassandra] INFO [main] 2021-07-30 10:30:48,015 Server.java:158 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a] 2021-07-30T10:30:48.015Z [cassandra] INFO [main] 2021-07-30 10:30:48,015 Server.java:159 - Starting listening for CQL clients on /10.1.151.79:9042 (unencrypted)... 2021-07-30T10:30:48.046Z [cassandra] INFO [main] 2021-07-30 10:30:48,046 ThriftServer.java:116 - Binding thrift service to /10.1.151.79:9160 2021-07-30T10:30:48.050Z [cassandra] INFO [main] 2021-07-30 10:30:48,049 CassandraDaemon.java:650 - Startup complete 2021-07-30T10:30:48.050Z [cassandra] INFO [Thread-2] 2021-07-30 10:30:48,050 ThriftServer.java:133 - Listening for thrift clients... 2021-07-30T10:30:49.940Z [cassandra] INFO [OptionalTasks:1] 2021-07-30 10:30:49,940 CassandraRoleManager.java:372 - Created default superuser role 'cassandra' 2021-07-30T10:31:32.007Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra.yaml 97.253µs 200 2021-07-30T10:31:32.010Z [pebble] GET /v1/plan?format=yaml 75.532µs 200 2021-07-30T10:31:32.160Z [cassandra] INFO [Native-Transport-Requests-1] 2021-07-30 10:31:32,160 AuthCache.java:177 - (Re)initializing PermissionsCache (validity period/update interval/max entries) (2000/2000/1000) 2021-07-30T10:31:32.340Z [cassandra] INFO [Native-Transport-Requests-1] 2021-07-30 10:31:32,340 AuthCache.java:177 - (Re)initializing RolesCache (validity period/update interval/max entries) (2000/2000/1000) 2021-07-30T10:31:32.856Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra.yaml 82.503µs 200 2021-07-30T10:31:32.859Z [pebble] GET /v1/plan?format=yaml 87.075µs 200 2021-07-30T10:31:32.876Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra-env.sh 121.241µs 200 2021-07-30T10:31:32.880Z [pebble] POST /v1/files 3.048708ms 200 2021-07-30T10:31:32.881Z [pebble] GET /v1/files?action=list&path=%2Fopt%2Fcassandra%2Flib%2Fprometheus_exporter_javaagent.jar 212.297µs 404 2021-07-30T10:31:36.322Z [pebble] POST /v1/files 6.22562ms 200 2021-07-30T10:31:36.327Z [pebble] GET /v1/services?names=cassandra 45.845µs 200 2021-07-30T10:31:36.332Z [pebble] POST /v1/services 4.595217ms 202 2021-07-30T10:31:36.337Z [cassandra] INFO [StorageServiceShutdownHook] 2021-07-30 10:31:36,337 HintsService.java:209 - Paused hints dispatch 2021-07-30T10:31:36.338Z [cassandra] INFO [StorageServiceShutdownHook] 2021-07-30 10:31:36,338 ThriftServer.java:139 - Stop listening to thrift clients 2021-07-30T10:31:36.339Z [cassandra] INFO [StorageServiceShutdownHook] 2021-07-30 10:31:36,339 Server.java:179 - Stop listening for CQL clients 2021-07-30T10:31:36.339Z [cassandra] INFO [StorageServiceShutdownHook] 2021-07-30 10:31:36,339 Gossiper.java:1683 - Announcing shutdown 2021-07-30T10:31:36.341Z [cassandra] INFO [StorageServiceShutdownHook] 2021-07-30 10:31:36,341 StorageService.java:2480 - Node /10.1.151.79 state jump to shutdown 2021-07-30T10:31:38.342Z [cassandra] INFO [StorageServiceShutdownHook] 2021-07-30 10:31:38,341 MessagingService.java:985 - Waiting for messaging service to quiesce 2021-07-30T10:31:38.343Z [cassandra] INFO [ACCEPT-/10.1.151.79] 2021-07-30 10:31:38,342 MessagingService.java:1346 - MessagingService has terminated the accept() thread 2021-07-30T10:31:38.456Z [cassandra] INFO [StorageServiceShutdownHook] 2021-07-30 10:31:38,456 HintsService.java:209 - Paused hints dispatch 2021-07-30T10:31:39.171Z [pebble] POST /v1/services 3.878526ms 202 2021-07-30T10:31:40.184Z [pebble] GET /v1/services?names=cassandra 23.122µs 200 2021-07-30T10:31:41.099Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns; 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I 2021-07-30T10:31:41.100Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B 2021-07-30T10:31:41.264Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra-env.sh 139.917µs 200 2021-07-30T10:31:41.268Z [pebble] POST /v1/files 3.090573ms 200 2021-07-30T10:31:41.268Z [pebble] GET /v1/files?action=list&path=%2Fopt%2Fcassandra%2Flib%2Fprometheus_exporter_javaagent.jar 236.244µs 200 2021-07-30T10:31:41.273Z [pebble] GET /v1/services?names=cassandra 33.797µs 200 2021-07-30T10:31:41.278Z [pebble] POST /v1/services 4.609902ms 202 2021-07-30T10:31:41.791Z [pebble] POST /v1/services 4.826506ms 202 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns; 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I 2021-07-30T10:31:43.709Z [cassandra] CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B 2021-07-30T10:31:43.983Z [pebble] GET /v1/files?action=read&path=%2Fetc%2Fcassandra%2Fcassandra.yaml 139.811µs 200 2021-07-30T10:31:43.987Z [pebble] GET /v1/plan?format=yaml 123.786µs 200 2021-07-30T10:31:44.431Z [cassandra] INFO [main] 2021-07-30 10:31:44,426 Server.java:93 - cassandra-exporter server has started. Listening on http://0.0.0.0:9500. 2021-07-30T10:31:44.462Z [cassandra] Exception in thread "main" java.lang.reflect.InvocationTargetException 2021-07-30T10:31:44.463Z [cassandra] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2021-07-30T10:31:44.463Z [cassandra] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2021-07-30T10:31:44.463Z [cassandra] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2021-07-30T10:31:44.463Z [cassandra] at java.lang.reflect.Method.invoke(Method.java:498) 2021-07-30T10:31:44.463Z [cassandra] at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386) 2021-07-30T10:31:44.463Z [cassandra] at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401) 2021-07-30T10:31:44.463Z [cassandra] Caused by: picocli.CommandLine$ExecutionException: Error while calling command (com.zegelin.cassandra.exporter.Agent@2f9f7dcf): java.net.BindException: Address already in use 2021-07-30T10:31:44.463Z [cassandra] at picocli.CommandLine.execute(CommandLine.java:1016) 2021-07-30T10:31:44.463Z [cassandra] at picocli.CommandLine.access$900(CommandLine.java:142) 2021-07-30T10:31:44.463Z [cassandra] at picocli.CommandLine$RunLast.handle(CommandLine.java:1199) 2021-07-30T10:31:44.463Z [cassandra] at picocli.CommandLine$RunLast.handle(CommandLine.java:1167) 2021-07-30T10:31:44.463Z [cassandra] at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1075) 2021-07-30T10:31:44.463Z [cassandra] at picocli.CommandLine.parseWithHandlers(CommandLine.java:1358) 2021-07-30T10:31:44.463Z [cassandra] at com.zegelin.cassandra.exporter.Agent.premain(Agent.java:52) 2021-07-30T10:31:44.463Z [cassandra] ... 6 more 2021-07-30T10:31:44.463Z [cassandra] Caused by: java.net.BindException: Address already in use 2021-07-30T10:31:44.463Z [cassandra] at sun.nio.ch.Net.bind0(Native Method) 2021-07-30T10:31:44.463Z [cassandra] at sun.nio.ch.Net.bind(Net.java:461) 2021-07-30T10:31:44.463Z [cassandra] at sun.nio.ch.Net.bind(Net.java:453) 2021-07-30T10:31:44.463Z [cassandra] at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:355) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) 2021-07-30T10:31:44.463Z [cassandra] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) 2021-07-30T10:31:44.463Z [cassandra] at java.lang.Thread.run(Thread.java:748) 2021-07-30T10:31:44.463Z [cassandra] FATAL ERROR in native method: processing of -javaagent failed
juju debug-log:
juju debug-log
unit-prometheus-0: 12:31:32 INFO juju.worker.uniter.operation ran "monitoring-relation-created" hook (via hook dispatching script: dispatch) unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Control connection created unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Initializing connection for host 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending initial options message for new connection (140650088563088) to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Received options response on new connection (140650088563088) from 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: No available compression types supported on both ends. locally supported: odict_keys([]). remotely supported: ['snappy', 'lz4'] unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending StartupMessage on <LibevConnection(140650088563088) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sent StartupMessage on <LibevConnection(140650088563088) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Got AuthenticateMessage on new connection (140650088563088) from 10.1.151.79:9042: org.apache.cassandra.auth.PasswordAuthenticator unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending SASL-based auth response on <LibevConnection(140650088563088) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Connection <LibevConnection(140650088563088) 10.1.151.79:9042> successfully authenticated unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Finished initializing connection for host 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Added pool for host 10.1.151.79:9042 to session unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Operator Framework 1.2.0+21.gb1a8fff up and running. unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Not starting MonitorReporter thread for Insights; not supported by server version 3.11.11 on ControlConnection host 10.1.151.79:9042 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Legacy hooks/monitoring-relation-joined does not exist. unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Started Session with client_id 28f1e946-1547-4ea8-8914-fcc49dc4a8d6 and session_id 44e2760e-46eb-49b8-8d3d-fd943059b1f7 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Using local storage: /var/lib/juju/agents/unit-prometheus-0/charm/.unit-state.db already exists unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Initializing Charm unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Starting new HTTP connection (1): localhost:9090 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: http://localhost:9090 "GET /api/v1/status/buildinfo HTTP/1.1" 200 209 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Starting new HTTP connection (1): localhost:9090 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: http://localhost:9090 "GET /api/v1/status/buildinfo HTTP/1.1" 200 209 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Emitting Juju event monitoring_relation_joined. unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Providing for joined consumer : {'provides': {'prometheus': '2.25.2'}, 'ready': False} unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Shutting down Cluster Scheduler unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Shutting down control connection unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closing connection (140650105407472) to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closed socket to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closing connection (140650088563088) to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closed socket to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Connecting to cluster, contact points: ['10.1.151.79']; protocol version: 4 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Host 10.1.151.79:9042 is now marked up unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: [control connection] Opening new connection to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending initial options message for new connection (140650088398464) to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Received options response on new connection (140650088398464) from 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: No available compression types supported on both ends. locally supported: odict_keys([]). remotely supported: ['snappy', 'lz4'] unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending StartupMessage on <LibevConnection(140650088398464) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sent StartupMessage on <LibevConnection(140650088398464) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Got AuthenticateMessage on new connection (140650088398464) from 10.1.151.79:9042: org.apache.cassandra.auth.PasswordAuthenticator unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending SASL-based auth response on <LibevConnection(140650088398464) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Connection <LibevConnection(140650088398464) 10.1.151.79:9042> successfully authenticated unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: [control connection] Established new connection <LibevConnection(140650088398464) 10.1.151.79:9042>, registering watchers and refreshing schema and topology unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: [control connection] Refreshing node list and token map using preloaded results unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: [control connection] Finished fetching ring info unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: [control connection] Rebuilding token map due to topology changes unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Control connection created unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Initializing connection for host 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending initial options message for new connection (140650071044688) to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Received options response on new connection (140650071044688) from 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: No available compression types supported on both ends. locally supported: odict_keys([]). remotely supported: ['snappy', 'lz4'] unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending StartupMessage on <LibevConnection(140650071044688) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sent StartupMessage on <LibevConnection(140650071044688) 10.1.151.79:9042> unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Got AuthenticateMessage on new connection (140650071044688) from 10.1.151.79:9042: org.apache.cassandra.auth.PasswordAuthenticator unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Sending SASL-based auth response on <LibevConnection(140650071044688) 10.1.151.79:9042> unit-prometheus-0: 12:31:32 INFO juju.worker.uniter.operation ran "monitoring-relation-joined" hook (via hook dispatching script: dispatch) unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Connection <LibevConnection(140650071044688) 10.1.151.79:9042> successfully authenticated unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Finished initializing connection for host 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Added pool for host 10.1.151.79:9042 to session unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Not starting MonitorReporter thread for Insights; not supported by server version 3.11.11 on ControlConnection host 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Started Session with client_id b13333f3-f36b-461e-9014-a59515b6e315 and session_id cbfcec0a-8fd5-4667-a948-366a334badc2 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Operator Framework 1.2.0+21.gb1a8fff up and running. unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Legacy hooks/monitoring-relation-changed does not exist. unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Using local storage: /var/lib/juju/agents/unit-prometheus-0/charm/.unit-state.db already exists unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Initializing Charm unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Starting new HTTP connection (1): localhost:9090 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Shutting down Cluster Scheduler unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Shutting down control connection unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: http://localhost:9090 "GET /api/v1/status/buildinfo HTTP/1.1" 200 209 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closing connection (140650088398464) to 10.1.151.79:9042 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Starting new HTTP connection (1): localhost:9090 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closed socket to 10.1.151.79:9042 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: http://localhost:9090 "GET /api/v1/status/buildinfo HTTP/1.1" 200 209 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closing connection (140650071044688) to 10.1.151.79:9042 unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Closed socket to 10.1.151.79:9042 unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Emitting Juju event monitoring_relation_changed. unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Provider is ready unit-prometheus-0: 12:31:32 INFO unit.prometheus/0.juju-log monitoring:5: Configuring Prometheus unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Notifying Consumer : {'provides': {'cassandra': '3.11'}, 'ready': True} unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: No alertmanagers available unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Prometheus config : {'global': {'scrape_interval': '1m', 'scrape_timeout': '10s', 'evaluation_interval': '1m'}, 'scrape_configs': [{'job_name': 'prometheus', 'scrape_interval': '5s', 'scrape_timeout': '5s', 'metrics_path': '/metrics', 'honor_timestamps': True, 'scheme': 'http', 'static_configs': [{'targets': ['localhost:9090']}]}]} unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Pod spec set successfully. unit-prometheus-0: 12:31:32 INFO unit.prometheus/0.juju-log monitoring:5: Pushed new configuration unit-prometheus-0: 12:31:32 DEBUG unit.prometheus/0.juju-log monitoring:5: Building pebble layer unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Re-emitting <PebbleReadyEvent via CassandraOperatorCharm/on/cassandra_pebble_ready[19]>. unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Pod spec set successfully. unit-cassandra-0: 12:31:32 DEBUG unit.cassandra/0.juju-log monitoring:5: Pushing Prometheus exporter to container to /opt/cassandra/lib/prometheus_exporter_javaagent.jar unit-prometheus-0: 12:31:34 INFO unit.prometheus/0.juju-log monitoring:5: Prometheus started unit-prometheus-0: 12:31:34 INFO juju.worker.uniter.operation ran "monitoring-relation-changed" hook (via hook dispatching script: dispatch) unit-cassandra-0: 12:31:36 INFO unit.cassandra/0.juju-log monitoring:5: Restarting cassandra unit-cassandra-0: 12:31:40 DEBUG unit.cassandra/0.juju-log monitoring:5: Emitting Juju event monitoring_relation_created. unit-cassandra-0: 12:31:40 INFO juju.worker.uniter.operation ran "monitoring-relation-created" hook (via hook dispatching script: dispatch) unit-cassandra-0: 12:31:40 DEBUG unit.cassandra/0.juju-log monitoring:5: Operator Framework 1.2.0+21.gb1a8fff up and running. unit-cassandra-0: 12:31:40 DEBUG unit.cassandra/0.juju-log monitoring:5: Legacy hooks/monitoring-relation-changed does not exist. unit-cassandra-0: 12:31:40 DEBUG unit.cassandra/0.juju-log monitoring:5: Emitting Juju event monitoring_relation_changed. unit-cassandra-0: 12:31:40 DEBUG unit.cassandra/0.juju-log monitoring:5: Got data from provider : {'provider_data': '{"provides": {"prometheus": "2.25.2"}, "ready": false}'} unit-cassandra-0: 12:31:40 INFO juju.worker.uniter.operation ran "monitoring-relation-changed" hook (via hook dispatching script: dispatch) unit-cassandra-0: 12:31:41 DEBUG unit.cassandra/0.juju-log monitoring:5: Operator Framework 1.2.0+21.gb1a8fff up and running. unit-cassandra-0: 12:31:41 DEBUG unit.cassandra/0.juju-log monitoring:5: Legacy hooks/monitoring-relation-joined does not exist. unit-cassandra-0: 12:31:41 DEBUG unit.cassandra/0.juju-log monitoring:5: Emitting Juju event monitoring_relation_joined. unit-cassandra-0: 12:31:41 INFO unit.cassandra/0.juju-log monitoring:5: Restarting cassandra unit-cassandra-0: 12:31:43 INFO juju.worker.uniter.operation ran "monitoring-relation-joined" hook (via hook dispatching script: dispatch) unit-cassandra-0: 12:31:43 DEBUG unit.cassandra/0.juju-log monitoring:5: Operator Framework 1.2.0+21.gb1a8fff up and running. unit-cassandra-0: 12:31:43 DEBUG unit.cassandra/0.juju-log monitoring:5: Legacy hooks/monitoring-relation-changed does not exist. unit-cassandra-0: 12:31:43 DEBUG unit.cassandra/0.juju-log monitoring:5: Emitting Juju event monitoring_relation_changed. unit-cassandra-0: 12:31:43 DEBUG unit.cassandra/0.juju-log monitoring:5: Got data from provider : {'provider_data': '{"provides": {"prometheus": "2.25.2"}, "ready": false}'} unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Operator Framework 1.2.0+21.gb1a8fff up and running. unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Legacy hooks/monitoring-relation-changed does not exist. unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Using local storage: /var/lib/juju/agents/unit-prometheus-0/charm/.unit-state.db already exists unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Initializing Charm unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Starting new HTTP connection (1): localhost:9090 unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: http://localhost:9090 "GET /api/v1/status/buildinfo HTTP/1.1" 200 209 unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Starting new HTTP connection (1): localhost:9090 unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: http://localhost:9090 "GET /api/v1/status/buildinfo HTTP/1.1" 200 209 unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Emitting Juju event monitoring_relation_changed. unit-prometheus-0: 12:31:43 INFO unit.prometheus/0.juju-log monitoring:5: Configuring Prometheus unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: No alertmanagers available unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Prometheus config : {'global': {'scrape_interval': '1m', 'scrape_timeout': '10s', 'evaluation_interval': '1m'}, 'scrape_configs': [{'job_name': 'prometheus', 'scrape_interval': '5s', 'scrape_timeout': '5s', 'metrics_path': '/metrics', 'honor_timestamps': True, 'scheme': 'http', 'static_configs': [{'targets': ['localhost:9090']}]}, {'job_name': 'juju_lma_80cd576_cassandra_prometheus_scrape', 'metrics_path': '/metrics', 'static_configs': [{'labels': {'juju_model': 'lma', 'juju_model_uuid': '80cd576a-bf19-4bd7-8377-99e42ffeeaf4', 'juju_application': 'cassandra', 'juju_unit': 'cassandra-0'}, 'targets': ['10.1.151.79:9500']}], 'relabel_configs': [{'source_labels': ['juju_model', 'juju_model_uuid', 'juju_application', 'juju_unit'], 'separator': '_', 'target_label': 'instance', 'regex': '(.*)'}]}]} unit-prometheus-0: 12:31:43 INFO unit.prometheus/0.juju-log monitoring:5: Pushed new configuration unit-prometheus-0: 12:31:43 DEBUG unit.prometheus/0.juju-log monitoring:5: Building pebble layer unit-cassandra-0: 12:31:43 INFO juju.worker.uniter.operation ran "monitoring-relation-changed" hook (via hook dispatching script: dispatch) unit-cassandra-0: 12:31:43 DEBUG unit.cassandra/0.juju-log cassandra-peers:4: Operator Framework 1.2.0+21.gb1a8fff up and running. unit-cassandra-0: 12:31:43 DEBUG unit.cassandra/0.juju-log cassandra-peers:4: Legacy hooks/cassandra-peers-relation-changed does not exist. unit-cassandra-0: 12:31:43 DEBUG unit.cassandra/0.juju-log cassandra-peers:4: Emitting Juju event cassandra_peers_relation_changed. unit-cassandra-0: 12:31:44 DEBUG unit.cassandra/0.juju-log cassandra-peers:4: Pod spec set successfully. unit-cassandra-0: 12:31:44 INFO juju.worker.uniter.operation ran "cassandra-peers-relation-changed" hook (via hook dispatching script: dispatch)
Also, we should get rid of that PodSpec reference ;-)
PodSpec
Solved in the last edge
edge
mmanciop
commented
3 years ago mmanciop
commented
3 years ago
Cassandra units occasionally faily to get Cassandra up and running after the application joins a
prometheus_scraperelation.Juju status:
The
cassandracontainer has no Cassandra process running, because it fails to bootstrap, see logs:juju debug-log:Also, we should get rid of that
PodSpecreference ;-)