Could not read commit log descriptor in file /var/lib/cassandra/commitlog/CommitLog-6-1646630759828.log

[JBjumper]

cassandra 3.11 running with error Exiting due to error while processing commit log during initialization. org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Could not read commit log descriptor in file /var/lib/cassandra/commitlog/CommitLog-6-1646630759828.log， i want to know the reason for the error.

error details:

 ERROR [main] 2022-03-28 06:45:42,388 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Could not read commit log descriptor in file /var/lib/cassandra/commitlog/CommitLog-6-1646630759828.log
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:153) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:84) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:140) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:177) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:158) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:325) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600) [apache-cassandra-3.11.0.jar:3.11.0]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.11.0.jar:3.11.0]

[wglambert]

Are you mounting /var/lib/cassandra as a bind-mount volume? What are the permissions like in that commitlog folder

I'm not able to reproduce any issues with running cassandra

$ docker run -d --name cassandra cassandra
3c2559513457004faace953ff927d033eb67567830874cbddb08980c8908fa1b

$ docker exec cassandra ls -l /var/lib/cassandra/commitlog/
total 96
-rw-r--r-- 1 cassandra cassandra 33554432 Mar 30 16:13 CommitLog-7-1648656465078.log
-rw-r--r-- 1 cassandra cassandra 33554432 Mar 30 16:07 CommitLog-7-1648656465079.log

docker logs

```console OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. get_mempolicy: Operation not permitted CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset(Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns; CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I CompileCommand: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom(Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V CompileCommand: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents()Z CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop()V CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition()V CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush(I)V CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow(JI)V CompileCommand: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly(I)J CompileCommand: exclude org/apache/cassandra/utils/JVMStabilityInspector.forceHeapSpaceOomMaybe(Ljava/lang/OutOfMemoryError;)V CompileCommand: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody(Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/rows/SerializationHelper;Lorg/apache/cassandra/io/util/DataOutputPlus;)V CompileCommand: inline org/apache/cassandra/io/util/Memory.checkBounds(JJ)V CompileCommand: inline org/apache/cassandra/io/util/SafeMemory.checkBounds(JJ)V CompileCommand: inline org/apache/cassandra/net/FrameDecoderWith8bHeader.decode(Ljava/util/Collection;Lorg/apache/cassandra/net/ShareableBytes;I)V CompileCommand: inline org/apache/cassandra/service/reads/repair/RowIteratorMergeListener.applyToPartition(ILjava/util/function/Consumer;)V CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I CompileCommand: inline org/apache/cassandra/utils/BloomFilter.indexes(Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J CompileCommand: inline org/apache/cassandra/utils/BloomFilter.setIndexes(JJIJ[J)V CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare(Ljava/nio/ByteBuffer;[B)I CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare([BLjava/nio/ByteBuffer;)I CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/lang/Object;JI)I CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompileCommand: inline org/apache/cassandra/utils/memory/BufferPool$LocalPool.tryGetInternal(IZ)Ljava/nio/ByteBuffer; CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.encodeUnsignedVInt(JI)[B CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.encodeUnsignedVInt(JI[B)V CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/io/DataOutput;)V CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/nio/ByteBuffer;)V CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt(JLjava/io/DataOutput;)V INFO [main] 2022-03-30 16:07:44,186 YamlConfigurationLoader.java:97 - Configuration location: file:/etc/cassandra/cassandra.yaml INFO [main] 2022-03-30 16:07:44,676 Config.java:706 - Node configuration:[allocate_tokens_for_keyspace=null; allocate_tokens_for_local_replication_factor=3; allow_extra_insecure_udfs=false; allow_insecure_udfs=false; audit_logging_options=AuditLogOptions{enabled=false, logger='BinAuditLogger', included_keyspaces='', excluded_keyspaces='system,system_schema,system_virtual_schema', included_categories='', excluded_categories='', included_users='', excluded_users='', audit_logs_dir='/opt/cassandra/logs/audit/', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_optimise_full_repair_streams=false; auto_optimise_inc_repair_streams=false; auto_optimise_preview_repair_streams=false; auto_snapshot=true; autocompaction_on_startup_enabled=true; automatic_sstable_upgrade=false; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; block_for_peers_in_remote_dcs=false; block_for_peers_timeout_in_secs=10; broadcast_address=172.17.0.5; broadcast_rpc_address=172.17.0.5; buffer_pool_use_heap_if_exhausted=false; cache_load_timeout_seconds=30; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_group_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=64; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_builders=1; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_validations=0; concurrent_writes=32; consecutive_message_errors_threshold=1; corrupted_tombstone_strategy=disabled; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=true; data_file_directories=[Ljava.lang.String;@10027fc9; diagnostic_events_enabled=false; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=1.0; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_drop_compact_storage=false; enable_materialized_views=false; enable_sasi_indexes=false; enable_scripted_user_defined_functions=false; enable_transient_replication=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; endpoint_snitch=SimpleSnitch; file_cache_enabled=false; file_cache_round_up=null; file_cache_size_in_mb=null; flush_compression=fast; force_new_prepared_statement_behaviour=false; full_query_logging_options=FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; ideal_consistency_level=null; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_range_tombstone_list_allocation_size=1; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_application_receive_queue_capacity_in_bytes=4194304; internode_application_receive_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_receive_queue_reserve_global_capacity_in_bytes=536870912; internode_application_send_queue_capacity_in_bytes=4194304; internode_application_send_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_send_queue_reserve_global_capacity_in_bytes=536870912; internode_authenticator=null; internode_compression=dc; internode_max_message_size_in_bytes=null; internode_socket_receive_buffer_size_in_bytes=0; internode_socket_send_buffer_size_in_bytes=0; internode_streaming_tcp_user_timeout_in_ms=300000; internode_tcp_connect_timeout_in_ms=2000; internode_tcp_user_timeout_in_ms=30000; key_cache_keys_to_save=2147483647; key_cache_migrate_during_compaction=true; key_cache_save_period=14400; key_cache_size_in_mb=null; keyspace_count_warn_threshold=40; listen_address=172.17.0.5; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; local_system_data_file_directory=null; max_concurrent_automatic_sstable_upgrades=1; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_allow_older_protocols=true; native_transport_flush_in_batches_legacy=false; native_transport_idle_timeout_in_ms=0; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=null; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; native_transport_receive_queue_capacity_in_bytes=1048576; network_authorizer=AllowAllNetworkAuthorizer; networking_cache_size_in_mb=null; num_tokens=16; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; periodic_commitlog_sync_lag_block_in_ms=null; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; range_tombstone_list_growth_factor=1.5; read_request_timeout_in_ms=5000; reject_repair_compaction_threshold=2147483647; repair_command_pool_full_strategy=queue; repair_command_pool_size=0; repair_session_max_tree_depth=null; repair_session_space_in_mb=null; repaired_data_tracking_for_partition_reads_enabled=false; repaired_data_tracking_for_range_reads_enabled=false; report_unconfirmed_repaired_data_mismatches=false; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.17.0.5}; server_encryption_options=; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_links_per_second=0; snapshot_on_duplicate_row_detection=false; snapshot_on_repaired_data_mismatch=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; storage_port=7000; stream_entire_sstables=true; stream_throughput_outbound_megabits_per_sec=200; streaming_connections_per_host=1; streaming_keep_alive_period_in_secs=300; table_count_warn_threshold=150; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@54afd745; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; use_offheap_merkle_trees=true; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; validation_preview_purge_head_start_in_sec=3600; windows_timer_interval=1; write_request_timeout_in_ms=2000] INFO [main] 2022-03-30 16:07:44,680 DatabaseDescriptor.java:416 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO [main] 2022-03-30 16:07:44,681 DatabaseDescriptor.java:477 - Global memtable on-heap threshold is enabled at 251MB INFO [main] 2022-03-30 16:07:44,681 DatabaseDescriptor.java:481 - Global memtable off-heap threshold is enabled at 251MB WARN [main] 2022-03-30 16:07:44,701 DatabaseDescriptor.java:628 - Only 4.562GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots INFO [main] 2022-03-30 16:07:44,876 JMXServerUtils.java:271 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi INFO [main] 2022-03-30 16:07:44,896 CassandraDaemon.java:625 - Hostname: 3c2559513457:7000:7001 INFO [main] 2022-03-30 16:07:44,898 CassandraDaemon.java:632 - JVM vendor/version: OpenJDK 64-Bit Server VM/11.0.14.1 INFO [main] 2022-03-30 16:07:44,898 CassandraDaemon.java:633 - Heap size: 1004.000MiB/1004.000MiB INFO [main] 2022-03-30 16:07:44,901 CassandraDaemon.java:638 - CodeHeap 'non-nmethods' Non-heap memory: init = 2555904(2496K) used = 1270912(1241K) committed = 2555904(2496K) max = 5828608(5692K) INFO [main] 2022-03-30 16:07:44,932 CassandraDaemon.java:638 - Metaspace Non-heap memory: init = 0(0K) used = 27506440(26861K) committed = 28188672(27528K) max = -1(-1K) INFO [main] 2022-03-30 16:07:44,933 CassandraDaemon.java:638 - CodeHeap 'profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 3474432(3393K) committed = 3538944(3456K) max = 122912768(120032K) INFO [main] 2022-03-30 16:07:44,933 CassandraDaemon.java:638 - Compressed Class Space Non-heap memory: init = 0(0K) used = 3112408(3039K) committed = 3346432(3268K) max = 1073741824(1048576K) INFO [main] 2022-03-30 16:07:44,933 CassandraDaemon.java:638 - Par Eden Space Heap memory: init = 167772160(163840K) used = 87331328(85284K) committed = 167772160(163840K) max = 167772160(163840K) INFO [main] 2022-03-30 16:07:44,934 CassandraDaemon.java:638 - Par Survivor Space Heap memory: init = 20971520(20480K) used = 0(0K) committed = 20971520(20480K) max = 20971520(20480K) INFO [main] 2022-03-30 16:07:44,934 CassandraDaemon.java:638 - CodeHeap 'non-profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 652672(637K) committed = 2555904(2496K) max = 122916864(120036K) INFO [main] 2022-03-30 16:07:44,940 CassandraDaemon.java:638 - CMS Old Gen Heap memory: init = 864026624(843776K) used = 0(0K) committed = 864026624(843776K) max = 864026624(843776K) INFO [main] 2022-03-30 16:07:44,941 CassandraDaemon.java:640 - Classpath: /etc/cassandra:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.8.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-4.0.3.jar:/opt/cassandra/lib/asm-7.1.jar:/opt/cassandra/lib/caffeine-2.5.6.jar:/opt/cassandra/lib/cassandra-driver-core-3.11.0-shaded.jar:/opt/cassandra/lib/chronicle-bytes-2.20.111.jar:/opt/cassandra/lib/chronicle-core-2.20.126.jar:/opt/cassandra/lib/chronicle-queue-5.20.123.jar:/opt/cassandra/lib/chronicle-threads-2.20.111.jar:/opt/cassandra/lib/chronicle-wire-2.20.117.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.11.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/ecj-4.6.1.jar:/opt/cassandra/lib/guava-27.0-jre.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.8.1.jar:/opt/cassandra/lib/j2objc-annotations-1.3.jar:/opt/cassandra/lib/jackson-annotations-2.12.5.jar:/opt/cassandra/lib/jackson-core-2.12.5.jar:/opt/cassandra/lib/jackson-databind-2.12.5.jar:/opt/cassandra/lib/jamm-0.3.2.jar:/opt/cassandra/lib/java-cup-runtime-11b-20160615.jar:/opt/cassandra/lib/javax.inject-1.jar:/opt/cassandra/lib/jbcrypt-0.4.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.25.jar:/opt/cassandra/lib/jcommander-1.30.jar:/opt/cassandra/lib/jctools-core-3.1.0.jar:/opt/cassandra/lib/jflex-1.8.2.jar:/opt/cassandra/lib/jna-5.6.0.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/jvm-attach-api-1.5.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.25.jar:/opt/cassandra/lib/logback-classic-1.2.9.jar:/opt/cassandra/lib/logback-core-1.2.9.jar:/opt/cassandra/lib/lz4-java-1.8.0.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/mxdump-0.14.jar:/opt/cassandra/lib/netty-all-4.1.58.Final.jar:/opt/cassandra/lib/netty-tcnative-boringssl-static-2.0.36.Final.jar:/opt/cassandra/lib/ohc-core-0.5.1.jar:/opt/cassandra/lib/ohc-core-j8-0.5.1.jar:/opt/cassandra/lib/psjava-0.1.19.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/sjk-cli-0.14.jar:/opt/cassandra/lib/sjk-core-0.14.jar:/opt/cassandra/lib/sjk-json-0.14.jar:/opt/cassandra/lib/sjk-stacktrace-0.14.jar:/opt/cassandra/lib/slf4j-api-1.7.25.jar:/opt/cassandra/lib/snakeyaml-1.26.jar:/opt/cassandra/lib/snappy-java-1.1.2.6.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/zstd-jni-1.5.0-4.jar:/opt/cassandra/lib/jsr223/*/*.jar: INFO [main] 2022-03-30 16:07:44,942 CassandraDaemon.java:642 - JVM Arguments: [-ea, -da:net.openhft..., -XX:+UseThreadPriorities, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -Djdk.attach.allowAttachSelf=true, --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, --add-exports=java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.server=ALL-UNNAMED, --add-exports=java.sql/java.sql=ALL-UNNAMED, --add-opens=java.base/java.lang.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED, --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED, --add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED, --add-opens=java.base/jdk.internal.math=ALL-UNNAMED, --add-opens=java.base/jdk.internal.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.util.jar=ALL-UNNAMED, --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED, -Dio.netty.tryReflectionSetAccessible=true, -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/opt/cassandra/logs/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760, -Xms1024M, -Xmx1024M, -Xmn200M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.2.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/local/lib/libjemalloc.so, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes] WARN [main] 2022-03-30 16:07:45,036 NativeLibrary.java:201 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK. INFO [main] 2022-03-30 16:07:45,086 MonotonicClock.java:202 - Scheduling approximate time conversion task with an interval of 10000 milliseconds INFO [main] 2022-03-30 16:07:45,094 MonotonicClock.java:338 - Scheduling approximate time-check task with a precision of 2 milliseconds INFO [main] 2022-03-30 16:07:45,097 StartupChecks.java:147 - jemalloc seems to be preloaded from /usr/local/lib/libjemalloc.so WARN [main] 2022-03-30 16:07:45,097 StartupChecks.java:187 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. INFO [main] 2022-03-30 16:07:45,099 SigarLibrary.java:44 - Initializing SIGAR library WARN [main] 2022-03-30 16:07:45,125 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true WARN [main] 2022-03-30 16:07:45,131 StartupChecks.java:329 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl. INFO [main] 2022-03-30 16:07:45,658 Keyspace.java:386 - Creating replication strategy system params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}} INFO [main] 2022-03-30 16:07:45,698 ColumnFamilyStore.java:385 - Initializing system.IndexInfo INFO [main] 2022-03-30 16:07:47,416 ColumnFamilyStore.java:385 - Initializing system.batches INFO [main] 2022-03-30 16:07:47,448 ColumnFamilyStore.java:385 - Initializing system.paxos INFO [main] 2022-03-30 16:07:47,478 ColumnFamilyStore.java:385 - Initializing system.local INFO [main] 2022-03-30 16:07:47,502 ColumnFamilyStore.java:385 - Initializing system.peers_v2 INFO [main] 2022-03-30 16:07:47,528 ColumnFamilyStore.java:385 - Initializing system.peers INFO [main] 2022-03-30 16:07:47,543 ColumnFamilyStore.java:385 - Initializing system.peer_events_v2 INFO [main] 2022-03-30 16:07:47,550 ColumnFamilyStore.java:385 - Initializing system.peer_events INFO [main] 2022-03-30 16:07:47,556 ColumnFamilyStore.java:385 - Initializing system.compaction_history INFO [main] 2022-03-30 16:07:47,564 ColumnFamilyStore.java:385 - Initializing system.sstable_activity INFO [main] 2022-03-30 16:07:47,584 ColumnFamilyStore.java:385 - Initializing system.size_estimates INFO [main] 2022-03-30 16:07:47,601 ColumnFamilyStore.java:385 - Initializing system.table_estimates INFO [main] 2022-03-30 16:07:47,615 ColumnFamilyStore.java:385 - Initializing system.available_ranges_v2 INFO [main] 2022-03-30 16:07:47,633 ColumnFamilyStore.java:385 - Initializing system.available_ranges INFO [main] 2022-03-30 16:07:47,645 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges_v2 INFO [main] 2022-03-30 16:07:47,663 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges INFO [main] 2022-03-30 16:07:47,675 ColumnFamilyStore.java:385 - Initializing system.view_builds_in_progress INFO [main] 2022-03-30 16:07:47,687 ColumnFamilyStore.java:385 - Initializing system.built_views INFO [main] 2022-03-30 16:07:47,699 ColumnFamilyStore.java:385 - Initializing system.prepared_statements INFO [main] 2022-03-30 16:07:47,709 ColumnFamilyStore.java:385 - Initializing system.repairs INFO [main] 2022-03-30 16:07:47,887 QueryProcessor.java:115 - Initialized prepared statement caches with 10 MB INFO [main] 2022-03-30 16:07:48,056 Keyspace.java:386 - Creating replication strategy system_schema params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}} INFO [main] 2022-03-30 16:07:48,063 ColumnFamilyStore.java:385 - Initializing system_schema.keyspaces INFO [main] 2022-03-30 16:07:48,072 ColumnFamilyStore.java:385 - Initializing system_schema.tables INFO [main] 2022-03-30 16:07:48,079 ColumnFamilyStore.java:385 - Initializing system_schema.columns INFO [main] 2022-03-30 16:07:48,087 ColumnFamilyStore.java:385 - Initializing system_schema.triggers INFO [main] 2022-03-30 16:07:48,095 ColumnFamilyStore.java:385 - Initializing system_schema.dropped_columns INFO [main] 2022-03-30 16:07:48,104 ColumnFamilyStore.java:385 - Initializing system_schema.views INFO [main] 2022-03-30 16:07:48,114 ColumnFamilyStore.java:385 - Initializing system_schema.types INFO [main] 2022-03-30 16:07:48,122 ColumnFamilyStore.java:385 - Initializing system_schema.functions INFO [main] 2022-03-30 16:07:48,132 ColumnFamilyStore.java:385 - Initializing system_schema.aggregates INFO [main] 2022-03-30 16:07:48,139 ColumnFamilyStore.java:385 - Initializing system_schema.indexes INFO [main] 2022-03-30 16:07:48,229 SystemKeyspaceMigrator40.java:76 - system.peers_v2 table was empty, migrating legacy system.peers, if this fails you should fix the issue and then truncate system.peers_v2 to have it try again. INFO [main] 2022-03-30 16:07:48,238 SystemKeyspaceMigrator40.java:100 - Migrating rows from legacy system.peers to system.peers_v2 INFO [main] 2022-03-30 16:07:48,243 SystemKeyspaceMigrator40.java:119 - Migrated 0 rows from legacy system.peers to system.peers_v2 INFO [main] 2022-03-30 16:07:48,244 SystemKeyspaceMigrator40.java:129 - system.peer_events_v2 table was empty, migrating legacy system.peer_events to system.peer_events_v2 INFO [main] 2022-03-30 16:07:48,246 SystemKeyspaceMigrator40.java:152 - Migrated 0 rows from legacy system.peer_events to system.peer_events_v2 INFO [main] 2022-03-30 16:07:48,246 SystemKeyspaceMigrator40.java:162 - system.transferred_ranges_v2 table was empty, migrating legacy system.transferred_ranges to system.transferred_ranges_v2 INFO [main] 2022-03-30 16:07:48,249 SystemKeyspaceMigrator40.java:190 - Migrated 0 rows from legacy system.transferred_ranges to system.transferred_ranges_v2 INFO [main] 2022-03-30 16:07:48,249 SystemKeyspaceMigrator40.java:200 - system.available_ranges_v2 table was empty, migrating legacy system.available_ranges to system.available_ranges_v2 INFO [main] 2022-03-30 16:07:48,250 SystemKeyspaceMigrator40.java:226 - Migrated 0 rows from legacy system.available_ranges to system.available_ranges_v2 INFO [main] 2022-03-30 16:07:48,251 StorageService.java:837 - Populating token metadata from system tables INFO [main] 2022-03-30 16:07:48,278 StorageService.java:831 - Token metadata: INFO [main] 2022-03-30 16:07:48,331 CacheService.java:103 - Initializing key cache with capacity of 50 MBs. INFO [main] 2022-03-30 16:07:48,347 CacheService.java:125 - Initializing row cache with capacity of 0 MBs INFO [main] 2022-03-30 16:07:48,350 CacheService.java:154 - Initializing counter cache with capacity of 25 MBs INFO [main] 2022-03-30 16:07:48,352 CacheService.java:165 - Scheduling counter cache save to every 7200 seconds (going to save all keys). INFO [main] 2022-03-30 16:07:48,398 CommitLog.java:168 - No commitlog files found; skipping replay INFO [main] 2022-03-30 16:07:48,399 StorageService.java:837 - Populating token metadata from system tables INFO [main] 2022-03-30 16:07:48,442 StorageService.java:831 - Token metadata: INFO [main] 2022-03-30 16:07:48,715 ColumnFamilyStore.java:2252 - Truncating system.size_estimates INFO [main] 2022-03-30 16:07:48,779 ColumnFamilyStore.java:2289 - Truncating system.size_estimates with truncatedAt=1648656468727 INFO [main] 2022-03-30 16:07:48,825 ColumnFamilyStore.java:878 - Enqueuing flush of local: 2.198KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:48,916 Memtable.java:469 - Writing Memtable-local@1875021063(0.429KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:48,921 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big-Data.db (0.224KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=31207) INFO [MemtableFlushWriter:2] 2022-03-30 16:07:49,007 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_8b50f9c0-b043-11ec-8e41-d522e6a825ce.log INFO [main] 2022-03-30 16:07:49,025 ColumnFamilyStore.java:2316 - Truncate of system.size_estimates is complete INFO [main] 2022-03-30 16:07:49,026 ColumnFamilyStore.java:2252 - Truncating system.table_estimates INFO [main] 2022-03-30 16:07:49,030 ColumnFamilyStore.java:2289 - Truncating system.table_estimates with truncatedAt=1648656469029 INFO [main] 2022-03-30 16:07:49,033 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.575KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:07:49,047 Memtable.java:469 - Writing Memtable-local@1694855549(0.091KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:07:49,049 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big-Data.db (0.063KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=31318) INFO [MemtableFlushWriter:2] 2022-03-30 16:07:49,060 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_8b708fb0-b043-11ec-8e41-d522e6a825ce.log INFO [main] 2022-03-30 16:07:49,063 ColumnFamilyStore.java:2316 - Truncate of system.table_estimates is complete INFO [main] 2022-03-30 16:07:49,069 QueryProcessor.java:167 - Preloaded 0 prepared statements INFO [main] 2022-03-30 16:07:49,070 StorageService.java:735 - Cassandra version: 4.0.3 INFO [main] 2022-03-30 16:07:49,070 StorageService.java:736 - CQL version: 3.4.5 INFO [main] 2022-03-30 16:07:49,071 StorageService.java:737 - Native protocol supported versions: 3/v3, 4/v4, 5/v5, 6/v6-beta (default: 5/v5) INFO [main] 2022-03-30 16:07:49,108 IndexSummaryManager.java:84 - Initializing index summary manager with a memory pool size of 50 MB and a resize interval of 60 minutes INFO [main] 2022-03-30 16:07:49,111 StorageService.java:754 - Loading persisted ring state INFO [main] 2022-03-30 16:07:49,112 StorageService.java:837 - Populating token metadata from system tables INFO [main] 2022-03-30 16:07:49,135 BufferPools.java:49 - Global buffer pool limit is 251.000MiB for chunk-cache and 62.000MiB for networking INFO [main] 2022-03-30 16:07:49,324 InboundConnectionInitiator.java:127 - Listening on address: (/172.17.0.5:7000), nic: eth0, encryption: unencrypted WARN [main] 2022-03-30 16:07:49,385 SystemKeyspace.java:1130 - No host ID found, created f9560c41-2844-4e41-a695-1c78370bb139 (Note: This should happen exactly once per node). INFO [main] 2022-03-30 16:07:49,413 StorageService.java:652 - Unable to gossip with any peers but continuing anyway since node is in its own seed list INFO [main] 2022-03-30 16:07:49,443 StorageService.java:959 - Starting up server gossip INFO [main] 2022-03-30 16:07:49,452 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.583KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:49,463 Memtable.java:469 - Writing Memtable-local@571545974(0.079KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:49,463 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=31482) INFO [MemtableFlushWriter:1] 2022-03-30 16:07:49,473 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_8bb0a5f0-b043-11ec-8e41-d522e6a825ce.log INFO [main] 2022-03-30 16:07:49,704 StorageService.java:1049 - This node will not auto bootstrap because it is configured to be a seed node. INFO [main] 2022-03-30 16:08:19,717 Gossiper.java:2214 - Waiting for gossip to settle... INFO [main] 2022-03-30 16:08:27,720 Gossiper.java:2245 - No gossip backlog; proceeding INFO [main] 2022-03-30 16:08:27,721 Gossiper.java:2214 - Waiting for gossip to settle... INFO [main] 2022-03-30 16:08:35,723 Gossiper.java:2245 - No gossip backlog; proceeding INFO [main] 2022-03-30 16:08:35,730 NetworkTopologyStrategy.java:88 - Configured datacenter replicas are datacenter1:rf(3) INFO [main] 2022-03-30 16:08:35,734 TokenAllocatorFactory.java:44 - Using ReplicationAwareTokenAllocator. INFO [main] 2022-03-30 16:08:35,778 TokenAllocation.java:106 - Selected tokens [7159693272382515772, -78643292598952927, -4888993251840852361, 4268648890998440934, -7486162000054009734, -2014317050782326943, 2433125553643663124, -9076166552567908372, 6080320464934914114, -3110358951017265232, -5867138704356966436, 1361624937303630311, 8403213278063107744, -923019940090114683, 3450976958386090139, 5344404772224020473] INFO [MigrationStage:1] 2022-03-30 16:08:35,847 ColumnFamilyStore.java:878 - Enqueuing flush of columns: 200.082KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,857 Memtable.java:469 - Writing Memtable-columns@1807913901(41.994KiB serialized bytes, 277 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,870 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb-1-big-Data.db (20.847KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42100) INFO [MemtableFlushWriter:2] 2022-03-30 16:08:35,880 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_a757a880-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:35,883 ColumnFamilyStore.java:878 - Enqueuing flush of dropped_columns: 1.079KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,891 Memtable.java:469 - Writing Memtable-dropped_columns@2012660903(0.120KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,892 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb-1-big-Data.db (0.097KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42100) INFO [MemtableFlushWriter:1] 2022-03-30 16:08:35,904 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb_txn_flush_a75cffb0-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:35,907 ColumnFamilyStore.java:878 - Enqueuing flush of triggers: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,914 Memtable.java:469 - Writing Memtable-triggers@1878858253(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,915 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42100) INFO [MemtableFlushWriter:2] 2022-03-30 16:08:35,924 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb_txn_flush_a760d040-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:35,927 ColumnFamilyStore.java:878 - Enqueuing flush of types: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,932 Memtable.java:469 - Writing Memtable-types@2048419531(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,933 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) INFO [MemtableFlushWriter:1] 2022-03-30 16:08:35,942 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb_txn_flush_a763dd80-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:35,949 ColumnFamilyStore.java:878 - Enqueuing flush of functions: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,956 Memtable.java:469 - Writing Memtable-functions@721394070(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,956 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) INFO [MemtableFlushWriter:2] 2022-03-30 16:08:35,969 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb_txn_flush_a76711d0-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:35,972 ColumnFamilyStore.java:878 - Enqueuing flush of aggregates: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,978 Memtable.java:469 - Writing Memtable-aggregates@1568510948(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,983 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) INFO [MemtableFlushWriter:1] 2022-03-30 16:08:35,995 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb_txn_flush_a76a9440-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:35,998 ColumnFamilyStore.java:878 - Enqueuing flush of indexes: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,006 Memtable.java:469 - Writing Memtable-indexes@1690274605(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,007 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) INFO [MemtableFlushWriter:2] 2022-03-30 16:08:36,016 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb_txn_flush_a76eb2f0-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:36,019 ColumnFamilyStore.java:878 - Enqueuing flush of tables: 92.656KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,025 Memtable.java:469 - Writing Memtable-tables@1997400980(28.858KiB serialized bytes, 42 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,029 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb-1-big-Data.db (17.879KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) INFO [MemtableFlushWriter:1] 2022-03-30 16:08:36,045 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb_txn_flush_a771c030-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:36,048 ColumnFamilyStore.java:878 - Enqueuing flush of views: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,054 Memtable.java:469 - Writing Memtable-views@463418348(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,055 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) INFO [MemtableFlushWriter:2] 2022-03-30 16:08:36,065 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb_txn_flush_a7762d00-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:36,068 ColumnFamilyStore.java:878 - Enqueuing flush of keyspaces: 3.243KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,075 Memtable.java:469 - Writing Memtable-keyspaces@908057134(0.673KiB serialized bytes, 7 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,076 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb-1-big-Data.db (0.556KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) INFO [MemtableFlushWriter:1] 2022-03-30 16:08:36,089 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb_txn_flush_a7796150-b043-11ec-8e41-d522e6a825ce.log INFO [MigrationStage:1] 2022-03-30 16:08:36,162 Keyspace.java:386 - Creating replication strategy system_auth params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}} INFO [MigrationStage:1] 2022-03-30 16:08:36,167 ColumnFamilyStore.java:385 - Initializing system_auth.network_permissions INFO [MigrationStage:1] 2022-03-30 16:08:36,176 ColumnFamilyStore.java:385 - Initializing system_auth.resource_role_permissons_index INFO [MigrationStage:1] 2022-03-30 16:08:36,182 ColumnFamilyStore.java:385 - Initializing system_auth.role_members INFO [MigrationStage:1] 2022-03-30 16:08:36,189 ColumnFamilyStore.java:385 - Initializing system_auth.role_permissions INFO [MigrationStage:1] 2022-03-30 16:08:36,195 ColumnFamilyStore.java:385 - Initializing system_auth.roles INFO [MigrationStage:1] 2022-03-30 16:08:36,206 Keyspace.java:386 - Creating replication strategy system_distributed params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}} INFO [MigrationStage:1] 2022-03-30 16:08:36,209 ColumnFamilyStore.java:385 - Initializing system_distributed.parent_repair_history INFO [MigrationStage:1] 2022-03-30 16:08:36,218 ColumnFamilyStore.java:385 - Initializing system_distributed.repair_history INFO [MigrationStage:1] 2022-03-30 16:08:36,224 ColumnFamilyStore.java:385 - Initializing system_distributed.view_build_status INFO [MigrationStage:1] 2022-03-30 16:08:36,231 Keyspace.java:386 - Creating replication strategy system_traces params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}} INFO [MigrationStage:1] 2022-03-30 16:08:36,234 ColumnFamilyStore.java:385 - Initializing system_traces.events INFO [MigrationStage:1] 2022-03-30 16:08:36,242 ColumnFamilyStore.java:385 - Initializing system_traces.sessions INFO [main] 2022-03-30 16:08:36,289 StorageService.java:1633 - JOINING: Finish joining ring INFO [main] 2022-03-30 16:08:36,295 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.604KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,302 Memtable.java:469 - Writing Memtable-local@473268192(0.084KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,312 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db (0.058KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42264) INFO [MemtableFlushWriter:2] 2022-03-30 16:08:36,330 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_a79c0480-b043-11ec-8e41-d522e6a825ce.log INFO [main] 2022-03-30 16:08:36,351 ColumnFamilyStore.java:878 - Enqueuing flush of local: 2.853KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,363 Memtable.java:469 - Writing Memtable-local@252234776(0.592KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,365 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db (0.360KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42676) INFO [MemtableFlushWriter:1] 2022-03-30 16:08:36,375 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_a7a50530-b043-11ec-8e41-d522e6a825ce.log INFO [main] 2022-03-30 16:08:36,397 StorageService.java:2784 - Node /172.17.0.5:7000 state jump to NORMAL INFO [main] 2022-03-30 16:08:36,403 Gossiper.java:2214 - Waiting for gossip to settle... INFO [main] 2022-03-30 16:08:44,405 Gossiper.java:2245 - No gossip backlog; proceeding INFO [main] 2022-03-30 16:08:44,491 NativeTransportService.java:130 - Using Netty Version: [netty-buffer=netty-buffer-4.1.58.Final.10b03e6, netty-codec=netty-codec-4.1.58.Final.10b03e6, netty-codec-dns=netty-codec-dns-4.1.58.Final.10b03e6, netty-codec-haproxy=netty-codec-haproxy-4.1.58.Final.10b03e6, netty-codec-http=netty-codec-http-4.1.58.Final.10b03e6, netty-codec-http2=netty-codec-http2-4.1.58.Final.10b03e6, netty-codec-memcache=netty-codec-memcache-4.1.58.Final.10b03e6, netty-codec-mqtt=netty-codec-mqtt-4.1.58.Final.10b03e6, netty-codec-redis=netty-codec-redis-4.1.58.Final.10b03e6, netty-codec-smtp=netty-codec-smtp-4.1.58.Final.10b03e6, netty-codec-socks=netty-codec-socks-4.1.58.Final.10b03e6, netty-codec-stomp=netty-codec-stomp-4.1.58.Final.10b03e6, netty-codec-xml=netty-codec-xml-4.1.58.Final.10b03e6, netty-common=netty-common-4.1.58.Final.10b03e6, netty-handler=netty-handler-4.1.58.Final.10b03e6, netty-handler-proxy=netty-handler-proxy-4.1.58.Final.10b03e6, netty-resolver=netty-resolver-4.1.58.Final.10b03e6, netty-resolver-dns=netty-resolver-dns-4.1.58.Final.10b03e6, netty-resolver-dns-native-macos=netty-resolver-dns-native-macos-4.1.58.Final.10b03e65f1, netty-transport=netty-transport-4.1.58.Final.10b03e6, netty-transport-native-epoll=netty-transport-native-epoll-4.1.58.Final.10b03e6, netty-transport-native-kqueue=netty-transport-native-kqueue-4.1.58.Final.10b03e65f1, netty-transport-native-unix-common=netty-transport-native-unix-common-4.1.58.Final.10b03e6, netty-transport-rxtx=netty-transport-rxtx-4.1.58.Final.10b03e6, netty-transport-sctp=netty-transport-sctp-4.1.58.Final.10b03e6, netty-transport-udt=netty-transport-udt-4.1.58.Final.10b03e6] INFO [CompactionExecutor:2] 2022-03-30 16:08:44,502 CompactionTask.java:150 - Compacting (ac7ca4f0-b043-11ec-8e41-d522e6a825ce) [/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db:level=0, ] INFO [main] 2022-03-30 16:08:44,519 NativeTransportService.java:68 - Netty using native Epoll event loop INFO [main] 2022-03-30 16:08:44,599 PipelineConfigurator.java:124 - Using Netty Version: [netty-buffer=netty-buffer-4.1.58.Final.10b03e6, netty-codec=netty-codec-4.1.58.Final.10b03e6, netty-codec-dns=netty-codec-dns-4.1.58.Final.10b03e6, netty-codec-haproxy=netty-codec-haproxy-4.1.58.Final.10b03e6, netty-codec-http=netty-codec-http-4.1.58.Final.10b03e6, netty-codec-http2=netty-codec-http2-4.1.58.Final.10b03e6, netty-codec-memcache=netty-codec-memcache-4.1.58.Final.10b03e6, netty-codec-mqtt=netty-codec-mqtt-4.1.58.Final.10b03e6, netty-codec-redis=netty-codec-redis-4.1.58.Final.10b03e6, netty-codec-smtp=netty-codec-smtp-4.1.58.Final.10b03e6, netty-codec-socks=netty-codec-socks-4.1.58.Final.10b03e6, netty-codec-stomp=netty-codec-stomp-4.1.58.Final.10b03e6, netty-codec-xml=netty-codec-xml-4.1.58.Final.10b03e6, netty-common=netty-common-4.1.58.Final.10b03e6, netty-handler=netty-handler-4.1.58.Final.10b03e6, netty-handler-proxy=netty-handler-proxy-4.1.58.Final.10b03e6, netty-resolver=netty-resolver-4.1.58.Final.10b03e6, netty-resolver-dns=netty-resolver-dns-4.1.58.Final.10b03e6, netty-resolver-dns-native-macos=netty-resolver-dns-native-macos-4.1.58.Final.10b03e65f1, netty-transport=netty-transport-4.1.58.Final.10b03e6, netty-transport-native-epoll=netty-transport-native-epoll-4.1.58.Final.10b03e6, netty-transport-native-kqueue=netty-transport-native-kqueue-4.1.58.Final.10b03e65f1, netty-transport-native-unix-common=netty-transport-native-unix-common-4.1.58.Final.10b03e6, netty-transport-rxtx=netty-transport-rxtx-4.1.58.Final.10b03e6, netty-transport-sctp=netty-transport-sctp-4.1.58.Final.10b03e6, netty-transport-udt=netty-transport-udt-4.1.58.Final.10b03e6] INFO [main] 2022-03-30 16:08:44,600 PipelineConfigurator.java:125 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)... INFO [main] 2022-03-30 16:08:44,614 CassandraDaemon.java:782 - Startup complete INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,638 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big INFO [CompactionExecutor:2] 2022-03-30 16:08:44,639 CompactionTask.java:241 - Compacted (ac7ca4f0-b043-11ec-8e41-d522e6a825ce) 5 sstables to [/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big,] to level=0. 0.773KiB to 0.629KiB (~81% of original) in 127ms. Read Throughput = 6.060KiB/s, Write Throughput = 4.927KiB/s, Row Throughput = ~2/s. 5 total partitions merged to 1. Partition merge counts were {5:1, } INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,642 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,645 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,648 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,650 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big INFO [OptionalTasks:1] 2022-03-30 16:08:54,599 CassandraRoleManager.java:339 - Created default superuser role 'cassandra' ```

[JBjumper]

Are you mounting /var/lib/cassandra as a bind-mount volume? What are the permissions like in that commitlog folder

I'm not able to reproduce any issues with running cassandra

$ docker run -d --name cassandra cassandra
3c2559513457004faace953ff927d033eb67567830874cbddb08980c8908fa1b

$ docker exec cassandra ls -l /var/lib/cassandra/commitlog/
total 96
-rw-r--r-- 1 cassandra cassandra 33554432 Mar 30 16:13 CommitLog-7-1648656465078.log
-rw-r--r-- 1 cassandra cassandra 33554432 Mar 30 16:07 CommitLog-7-1648656465079.log

> docker logs

``` > OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. > get_mempolicy: Operation not permitted > CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset(Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns; > CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V > CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize(Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I > CompileCommand: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom(Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V > CompileCommand: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents()Z > CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop()V > CompileCommand: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition()V > CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush(I)V > CompileCommand: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow(JI)V > CompileCommand: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly(I)J > CompileCommand: exclude org/apache/cassandra/utils/JVMStabilityInspector.forceHeapSpaceOomMaybe(Ljava/lang/OutOfMemoryError;)V > CompileCommand: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody(Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/rows/SerializationHelper;Lorg/apache/cassandra/io/util/DataOutputPlus;)V > CompileCommand: inline org/apache/cassandra/io/util/Memory.checkBounds(JJ)V > CompileCommand: inline org/apache/cassandra/io/util/SafeMemory.checkBounds(JJ)V > CompileCommand: inline org/apache/cassandra/net/FrameDecoderWith8bHeader.decode(Ljava/util/Collection;Lorg/apache/cassandra/net/ShareableBytes;I)V > CompileCommand: inline org/apache/cassandra/service/reads/repair/RowIteratorMergeListener.applyToPartition(ILjava/util/function/Consumer;)V > CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I > CompileCommand: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan(Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I > CompileCommand: inline org/apache/cassandra/utils/BloomFilter.indexes(Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J > CompileCommand: inline org/apache/cassandra/utils/BloomFilter.setIndexes(JJIJ[J)V > CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare(Ljava/nio/ByteBuffer;[B)I > CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compare([BLjava/nio/ByteBuffer;)I > CompileCommand: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I > CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/lang/Object;JI)I > CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/lang/Object;JILjava/nio/ByteBuffer;)I > CompileCommand: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I > CompileCommand: inline org/apache/cassandra/utils/memory/BufferPool$LocalPool.tryGetInternal(IZ)Ljava/nio/ByteBuffer; > CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.encodeUnsignedVInt(JI)[B > CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.encodeUnsignedVInt(JI[B)V > CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/io/DataOutput;)V > CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeUnsignedVInt(JLjava/nio/ByteBuffer;)V > CompileCommand: inline org/apache/cassandra/utils/vint/VIntCoding.writeVInt(JLjava/io/DataOutput;)V > INFO [main] 2022-03-30 16:07:44,186 YamlConfigurationLoader.java:97 - Configuration location: file:/etc/cassandra/cassandra.yaml > INFO [main] 2022-03-30 16:07:44,676 Config.java:706 - Node configuration:[allocate_tokens_for_keyspace=null; allocate_tokens_for_local_replication_factor=3; allow_extra_insecure_udfs=false; allow_insecure_udfs=false; audit_logging_options=AuditLogOptions{enabled=false, logger='BinAuditLogger', included_keyspaces='', excluded_keyspaces='system,system_schema,system_virtual_schema', included_categories='', excluded_categories='', included_users='', excluded_users='', audit_logs_dir='/opt/cassandra/logs/audit/', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_optimise_full_repair_streams=false; auto_optimise_inc_repair_streams=false; auto_optimise_preview_repair_streams=false; auto_snapshot=true; autocompaction_on_startup_enabled=true; automatic_sstable_upgrade=false; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; block_for_peers_in_remote_dcs=false; block_for_peers_timeout_in_secs=10; broadcast_address=172.17.0.5; broadcast_rpc_address=172.17.0.5; buffer_pool_use_heap_if_exhausted=false; cache_load_timeout_seconds=30; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_group_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=64; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_builders=1; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_validations=0; concurrent_writes=32; consecutive_message_errors_threshold=1; corrupted_tombstone_strategy=disabled; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=true; data_file_directories=[Ljava.lang.String;@10027fc9; diagnostic_events_enabled=false; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=1.0; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_drop_compact_storage=false; enable_materialized_views=false; enable_sasi_indexes=false; enable_scripted_user_defined_functions=false; enable_transient_replication=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; endpoint_snitch=SimpleSnitch; file_cache_enabled=false; file_cache_round_up=null; file_cache_size_in_mb=null; flush_compression=fast; force_new_prepared_statement_behaviour=false; full_query_logging_options=FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; ideal_consistency_level=null; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_range_tombstone_list_allocation_size=1; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_application_receive_queue_capacity_in_bytes=4194304; internode_application_receive_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_receive_queue_reserve_global_capacity_in_bytes=536870912; internode_application_send_queue_capacity_in_bytes=4194304; internode_application_send_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_send_queue_reserve_global_capacity_in_bytes=536870912; internode_authenticator=null; internode_compression=dc; internode_max_message_size_in_bytes=null; internode_socket_receive_buffer_size_in_bytes=0; internode_socket_send_buffer_size_in_bytes=0; internode_streaming_tcp_user_timeout_in_ms=300000; internode_tcp_connect_timeout_in_ms=2000; internode_tcp_user_timeout_in_ms=30000; key_cache_keys_to_save=2147483647; key_cache_migrate_during_compaction=true; key_cache_save_period=14400; key_cache_size_in_mb=null; keyspace_count_warn_threshold=40; listen_address=172.17.0.5; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; local_system_data_file_directory=null; max_concurrent_automatic_sstable_upgrades=1; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_allow_older_protocols=true; native_transport_flush_in_batches_legacy=false; native_transport_idle_timeout_in_ms=0; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=null; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; native_transport_receive_queue_capacity_in_bytes=1048576; network_authorizer=AllowAllNetworkAuthorizer; networking_cache_size_in_mb=null; num_tokens=16; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; periodic_commitlog_sync_lag_block_in_ms=null; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; range_tombstone_list_growth_factor=1.5; read_request_timeout_in_ms=5000; reject_repair_compaction_threshold=2147483647; repair_command_pool_full_strategy=queue; repair_command_pool_size=0; repair_session_max_tree_depth=null; repair_session_space_in_mb=null; repaired_data_tracking_for_partition_reads_enabled=false; repaired_data_tracking_for_range_reads_enabled=false; report_unconfirmed_repaired_data_mismatches=false; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.17.0.5}; server_encryption_options=; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_links_per_second=0; snapshot_on_duplicate_row_detection=false; snapshot_on_repaired_data_mismatch=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; storage_port=7000; stream_entire_sstables=true; stream_throughput_outbound_megabits_per_sec=200; streaming_connections_per_host=1; streaming_keep_alive_period_in_secs=300; table_count_warn_threshold=150; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@54afd745; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; use_offheap_merkle_trees=true; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; validation_preview_purge_head_start_in_sec=3600; windows_timer_interval=1; write_request_timeout_in_ms=2000] > INFO [main] 2022-03-30 16:07:44,680 DatabaseDescriptor.java:416 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap > INFO [main] 2022-03-30 16:07:44,681 DatabaseDescriptor.java:477 - Global memtable on-heap threshold is enabled at 251MB > INFO [main] 2022-03-30 16:07:44,681 DatabaseDescriptor.java:481 - Global memtable off-heap threshold is enabled at 251MB > WARN [main] 2022-03-30 16:07:44,701 DatabaseDescriptor.java:628 - Only 4.562GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots > INFO [main] 2022-03-30 16:07:44,876 JMXServerUtils.java:271 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi > INFO [main] 2022-03-30 16:07:44,896 CassandraDaemon.java:625 - Hostname: 3c2559513457:7000:7001 > INFO [main] 2022-03-30 16:07:44,898 CassandraDaemon.java:632 - JVM vendor/version: OpenJDK 64-Bit Server VM/11.0.14.1 > INFO [main] 2022-03-30 16:07:44,898 CassandraDaemon.java:633 - Heap size: 1004.000MiB/1004.000MiB > INFO [main] 2022-03-30 16:07:44,901 CassandraDaemon.java:638 - CodeHeap 'non-nmethods' Non-heap memory: init = 2555904(2496K) used = 1270912(1241K) committed = 2555904(2496K) max = 5828608(5692K) > INFO [main] 2022-03-30 16:07:44,932 CassandraDaemon.java:638 - Metaspace Non-heap memory: init = 0(0K) used = 27506440(26861K) committed = 28188672(27528K) max = -1(-1K) > INFO [main] 2022-03-30 16:07:44,933 CassandraDaemon.java:638 - CodeHeap 'profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 3474432(3393K) committed = 3538944(3456K) max = 122912768(120032K) > INFO [main] 2022-03-30 16:07:44,933 CassandraDaemon.java:638 - Compressed Class Space Non-heap memory: init = 0(0K) used = 3112408(3039K) committed = 3346432(3268K) max = 1073741824(1048576K) > INFO [main] 2022-03-30 16:07:44,933 CassandraDaemon.java:638 - Par Eden Space Heap memory: init = 167772160(163840K) used = 87331328(85284K) committed = 167772160(163840K) max = 167772160(163840K) > INFO [main] 2022-03-30 16:07:44,934 CassandraDaemon.java:638 - Par Survivor Space Heap memory: init = 20971520(20480K) used = 0(0K) committed = 20971520(20480K) max = 20971520(20480K) > INFO [main] 2022-03-30 16:07:44,934 CassandraDaemon.java:638 - CodeHeap 'non-profiled nmethods' Non-heap memory: init = 2555904(2496K) used = 652672(637K) committed = 2555904(2496K) max = 122916864(120036K) > INFO [main] 2022-03-30 16:07:44,940 CassandraDaemon.java:638 - CMS Old Gen Heap memory: init = 864026624(843776K) used = 0(0K) committed = 864026624(843776K) max = 864026624(843776K) > INFO [main] 2022-03-30 16:07:44,941 CassandraDaemon.java:640 - Classpath: /etc/cassandra:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.8.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-4.0.3.jar:/opt/cassandra/lib/asm-7.1.jar:/opt/cassandra/lib/caffeine-2.5.6.jar:/opt/cassandra/lib/cassandra-driver-core-3.11.0-shaded.jar:/opt/cassandra/lib/chronicle-bytes-2.20.111.jar:/opt/cassandra/lib/chronicle-core-2.20.126.jar:/opt/cassandra/lib/chronicle-queue-5.20.123.jar:/opt/cassandra/lib/chronicle-threads-2.20.111.jar:/opt/cassandra/lib/chronicle-wire-2.20.117.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.11.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/ecj-4.6.1.jar:/opt/cassandra/lib/guava-27.0-jre.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.8.1.jar:/opt/cassandra/lib/j2objc-annotations-1.3.jar:/opt/cassandra/lib/jackson-annotations-2.12.5.jar:/opt/cassandra/lib/jackson-core-2.12.5.jar:/opt/cassandra/lib/jackson-databind-2.12.5.jar:/opt/cassandra/lib/jamm-0.3.2.jar:/opt/cassandra/lib/java-cup-runtime-11b-20160615.jar:/opt/cassandra/lib/javax.inject-1.jar:/opt/cassandra/lib/jbcrypt-0.4.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.25.jar:/opt/cassandra/lib/jcommander-1.30.jar:/opt/cassandra/lib/jctools-core-3.1.0.jar:/opt/cassandra/lib/jflex-1.8.2.jar:/opt/cassandra/lib/jna-5.6.0.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/jvm-attach-api-1.5.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.25.jar:/opt/cassandra/lib/logback-classic-1.2.9.jar:/opt/cassandra/lib/logback-core-1.2.9.jar:/opt/cassandra/lib/lz4-java-1.8.0.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/mxdump-0.14.jar:/opt/cassandra/lib/netty-all-4.1.58.Final.jar:/opt/cassandra/lib/netty-tcnative-boringssl-static-2.0.36.Final.jar:/opt/cassandra/lib/ohc-core-0.5.1.jar:/opt/cassandra/lib/ohc-core-j8-0.5.1.jar:/opt/cassandra/lib/psjava-0.1.19.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/sjk-cli-0.14.jar:/opt/cassandra/lib/sjk-core-0.14.jar:/opt/cassandra/lib/sjk-json-0.14.jar:/opt/cassandra/lib/sjk-stacktrace-0.14.jar:/opt/cassandra/lib/slf4j-api-1.7.25.jar:/opt/cassandra/lib/snakeyaml-1.26.jar:/opt/cassandra/lib/snappy-java-1.1.2.6.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/zstd-jni-1.5.0-4.jar:/opt/cassandra/lib/jsr223/*/*.jar: > INFO [main] 2022-03-30 16:07:44,942 CassandraDaemon.java:642 - JVM Arguments: [-ea, -da:net.openhft..., -XX:+UseThreadPriorities, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -Djdk.attach.allowAttachSelf=true, --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, --add-exports=java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.server=ALL-UNNAMED, --add-exports=java.sql/java.sql=ALL-UNNAMED, --add-opens=java.base/java.lang.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED, --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED, --add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED, --add-opens=java.base/jdk.internal.math=ALL-UNNAMED, --add-opens=java.base/jdk.internal.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.util.jar=ALL-UNNAMED, --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED, -Dio.netty.tryReflectionSetAccessible=true, -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/opt/cassandra/logs/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760, -Xms1024M, -Xmx1024M, -Xmn200M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.2.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/local/lib/libjemalloc.so, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes] > WARN [main] 2022-03-30 16:07:45,036 NativeLibrary.java:201 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK. > INFO [main] 2022-03-30 16:07:45,086 MonotonicClock.java:202 - Scheduling approximate time conversion task with an interval of 10000 milliseconds > INFO [main] 2022-03-30 16:07:45,094 MonotonicClock.java:338 - Scheduling approximate time-check task with a precision of 2 milliseconds > INFO [main] 2022-03-30 16:07:45,097 StartupChecks.java:147 - jemalloc seems to be preloaded from /usr/local/lib/libjemalloc.so > WARN [main] 2022-03-30 16:07:45,097 StartupChecks.java:187 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. > INFO [main] 2022-03-30 16:07:45,099 SigarLibrary.java:44 - Initializing SIGAR library > WARN [main] 2022-03-30 16:07:45,125 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false, Address space adequate? : true, nofile limit adequate? : true, nproc limit adequate? : true > WARN [main] 2022-03-30 16:07:45,131 StartupChecks.java:329 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl. > INFO [main] 2022-03-30 16:07:45,658 Keyspace.java:386 - Creating replication strategy system params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}} > INFO [main] 2022-03-30 16:07:45,698 ColumnFamilyStore.java:385 - Initializing system.IndexInfo > INFO [main] 2022-03-30 16:07:47,416 ColumnFamilyStore.java:385 - Initializing system.batches > INFO [main] 2022-03-30 16:07:47,448 ColumnFamilyStore.java:385 - Initializing system.paxos > INFO [main] 2022-03-30 16:07:47,478 ColumnFamilyStore.java:385 - Initializing system.local > INFO [main] 2022-03-30 16:07:47,502 ColumnFamilyStore.java:385 - Initializing system.peers_v2 > INFO [main] 2022-03-30 16:07:47,528 ColumnFamilyStore.java:385 - Initializing system.peers > INFO [main] 2022-03-30 16:07:47,543 ColumnFamilyStore.java:385 - Initializing system.peer_events_v2 > INFO [main] 2022-03-30 16:07:47,550 ColumnFamilyStore.java:385 - Initializing system.peer_events > INFO [main] 2022-03-30 16:07:47,556 ColumnFamilyStore.java:385 - Initializing system.compaction_history > INFO [main] 2022-03-30 16:07:47,564 ColumnFamilyStore.java:385 - Initializing system.sstable_activity > INFO [main] 2022-03-30 16:07:47,584 ColumnFamilyStore.java:385 - Initializing system.size_estimates > INFO [main] 2022-03-30 16:07:47,601 ColumnFamilyStore.java:385 - Initializing system.table_estimates > INFO [main] 2022-03-30 16:07:47,615 ColumnFamilyStore.java:385 - Initializing system.available_ranges_v2 > INFO [main] 2022-03-30 16:07:47,633 ColumnFamilyStore.java:385 - Initializing system.available_ranges > INFO [main] 2022-03-30 16:07:47,645 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges_v2 > INFO [main] 2022-03-30 16:07:47,663 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges > INFO [main] 2022-03-30 16:07:47,675 ColumnFamilyStore.java:385 - Initializing system.view_builds_in_progress > INFO [main] 2022-03-30 16:07:47,687 ColumnFamilyStore.java:385 - Initializing system.built_views > INFO [main] 2022-03-30 16:07:47,699 ColumnFamilyStore.java:385 - Initializing system.prepared_statements > INFO [main] 2022-03-30 16:07:47,709 ColumnFamilyStore.java:385 - Initializing system.repairs > INFO [main] 2022-03-30 16:07:47,887 QueryProcessor.java:115 - Initialized prepared statement caches with 10 MB > INFO [main] 2022-03-30 16:07:48,056 Keyspace.java:386 - Creating replication strategy system_schema params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}} > INFO [main] 2022-03-30 16:07:48,063 ColumnFamilyStore.java:385 - Initializing system_schema.keyspaces > INFO [main] 2022-03-30 16:07:48,072 ColumnFamilyStore.java:385 - Initializing system_schema.tables > INFO [main] 2022-03-30 16:07:48,079 ColumnFamilyStore.java:385 - Initializing system_schema.columns > INFO [main] 2022-03-30 16:07:48,087 ColumnFamilyStore.java:385 - Initializing system_schema.triggers > INFO [main] 2022-03-30 16:07:48,095 ColumnFamilyStore.java:385 - Initializing system_schema.dropped_columns > INFO [main] 2022-03-30 16:07:48,104 ColumnFamilyStore.java:385 - Initializing system_schema.views > INFO [main] 2022-03-30 16:07:48,114 ColumnFamilyStore.java:385 - Initializing system_schema.types > INFO [main] 2022-03-30 16:07:48,122 ColumnFamilyStore.java:385 - Initializing system_schema.functions > INFO [main] 2022-03-30 16:07:48,132 ColumnFamilyStore.java:385 - Initializing system_schema.aggregates > INFO [main] 2022-03-30 16:07:48,139 ColumnFamilyStore.java:385 - Initializing system_schema.indexes > INFO [main] 2022-03-30 16:07:48,229 SystemKeyspaceMigrator40.java:76 - system.peers_v2 table was empty, migrating legacy system.peers, if this fails you should fix the issue and then truncate system.peers_v2 to have it try again. > INFO [main] 2022-03-30 16:07:48,238 SystemKeyspaceMigrator40.java:100 - Migrating rows from legacy system.peers to system.peers_v2 > INFO [main] 2022-03-30 16:07:48,243 SystemKeyspaceMigrator40.java:119 - Migrated 0 rows from legacy system.peers to system.peers_v2 > INFO [main] 2022-03-30 16:07:48,244 SystemKeyspaceMigrator40.java:129 - system.peer_events_v2 table was empty, migrating legacy system.peer_events to system.peer_events_v2 > INFO [main] 2022-03-30 16:07:48,246 SystemKeyspaceMigrator40.java:152 - Migrated 0 rows from legacy system.peer_events to system.peer_events_v2 > INFO [main] 2022-03-30 16:07:48,246 SystemKeyspaceMigrator40.java:162 - system.transferred_ranges_v2 table was empty, migrating legacy system.transferred_ranges to system.transferred_ranges_v2 > INFO [main] 2022-03-30 16:07:48,249 SystemKeyspaceMigrator40.java:190 - Migrated 0 rows from legacy system.transferred_ranges to system.transferred_ranges_v2 > INFO [main] 2022-03-30 16:07:48,249 SystemKeyspaceMigrator40.java:200 - system.available_ranges_v2 table was empty, migrating legacy system.available_ranges to system.available_ranges_v2 > INFO [main] 2022-03-30 16:07:48,250 SystemKeyspaceMigrator40.java:226 - Migrated 0 rows from legacy system.available_ranges to system.available_ranges_v2 > INFO [main] 2022-03-30 16:07:48,251 StorageService.java:837 - Populating token metadata from system tables > INFO [main] 2022-03-30 16:07:48,278 StorageService.java:831 - Token metadata: > INFO [main] 2022-03-30 16:07:48,331 CacheService.java:103 - Initializing key cache with capacity of 50 MBs. > INFO [main] 2022-03-30 16:07:48,347 CacheService.java:125 - Initializing row cache with capacity of 0 MBs > INFO [main] 2022-03-30 16:07:48,350 CacheService.java:154 - Initializing counter cache with capacity of 25 MBs > INFO [main] 2022-03-30 16:07:48,352 CacheService.java:165 - Scheduling counter cache save to every 7200 seconds (going to save all keys). > INFO [main] 2022-03-30 16:07:48,398 CommitLog.java:168 - No commitlog files found; skipping replay > INFO [main] 2022-03-30 16:07:48,399 StorageService.java:837 - Populating token metadata from system tables > INFO [main] 2022-03-30 16:07:48,442 StorageService.java:831 - Token metadata: > INFO [main] 2022-03-30 16:07:48,715 ColumnFamilyStore.java:2252 - Truncating system.size_estimates > INFO [main] 2022-03-30 16:07:48,779 ColumnFamilyStore.java:2289 - Truncating system.size_estimates with truncatedAt=1648656468727 > INFO [main] 2022-03-30 16:07:48,825 ColumnFamilyStore.java:878 - Enqueuing flush of local: 2.198KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:48,916 Memtable.java:469 - Writing Memtable-local@1875021063(0.429KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:48,921 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big-Data.db (0.224KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=31207) > INFO [MemtableFlushWriter:2] 2022-03-30 16:07:49,007 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_8b50f9c0-b043-11ec-8e41-d522e6a825ce.log > INFO [main] 2022-03-30 16:07:49,025 ColumnFamilyStore.java:2316 - Truncate of system.size_estimates is complete > INFO [main] 2022-03-30 16:07:49,026 ColumnFamilyStore.java:2252 - Truncating system.table_estimates > INFO [main] 2022-03-30 16:07:49,030 ColumnFamilyStore.java:2289 - Truncating system.table_estimates with truncatedAt=1648656469029 > INFO [main] 2022-03-30 16:07:49,033 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.575KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:07:49,047 Memtable.java:469 - Writing Memtable-local@1694855549(0.091KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:07:49,049 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big-Data.db (0.063KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=31318) > INFO [MemtableFlushWriter:2] 2022-03-30 16:07:49,060 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_8b708fb0-b043-11ec-8e41-d522e6a825ce.log > INFO [main] 2022-03-30 16:07:49,063 ColumnFamilyStore.java:2316 - Truncate of system.table_estimates is complete > INFO [main] 2022-03-30 16:07:49,069 QueryProcessor.java:167 - Preloaded 0 prepared statements > INFO [main] 2022-03-30 16:07:49,070 StorageService.java:735 - Cassandra version: 4.0.3 > INFO [main] 2022-03-30 16:07:49,070 StorageService.java:736 - CQL version: 3.4.5 > INFO [main] 2022-03-30 16:07:49,071 StorageService.java:737 - Native protocol supported versions: 3/v3, 4/v4, 5/v5, 6/v6-beta (default: 5/v5) > INFO [main] 2022-03-30 16:07:49,108 IndexSummaryManager.java:84 - Initializing index summary manager with a memory pool size of 50 MB and a resize interval of 60 minutes > INFO [main] 2022-03-30 16:07:49,111 StorageService.java:754 - Loading persisted ring state > INFO [main] 2022-03-30 16:07:49,112 StorageService.java:837 - Populating token metadata from system tables > INFO [main] 2022-03-30 16:07:49,135 BufferPools.java:49 - Global buffer pool limit is 251.000MiB for chunk-cache and 62.000MiB for networking > INFO [main] 2022-03-30 16:07:49,324 InboundConnectionInitiator.java:127 - Listening on address: (/172.17.0.5:7000), nic: eth0, encryption: unencrypted > WARN [main] 2022-03-30 16:07:49,385 SystemKeyspace.java:1130 - No host ID found, created f9560c41-2844-4e41-a695-1c78370bb139 (Note: This should happen exactly once per node). > INFO [main] 2022-03-30 16:07:49,413 StorageService.java:652 - Unable to gossip with any peers but continuing anyway since node is in its own seed list > INFO [main] 2022-03-30 16:07:49,443 StorageService.java:959 - Starting up server gossip > INFO [main] 2022-03-30 16:07:49,452 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.583KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:49,463 Memtable.java:469 - Writing Memtable-local@571545974(0.079KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:07:49,463 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=31482) > INFO [MemtableFlushWriter:1] 2022-03-30 16:07:49,473 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_8bb0a5f0-b043-11ec-8e41-d522e6a825ce.log > INFO [main] 2022-03-30 16:07:49,704 StorageService.java:1049 - This node will not auto bootstrap because it is configured to be a seed node. > INFO [main] 2022-03-30 16:08:19,717 Gossiper.java:2214 - Waiting for gossip to settle... > INFO [main] 2022-03-30 16:08:27,720 Gossiper.java:2245 - No gossip backlog; proceeding > INFO [main] 2022-03-30 16:08:27,721 Gossiper.java:2214 - Waiting for gossip to settle... > INFO [main] 2022-03-30 16:08:35,723 Gossiper.java:2245 - No gossip backlog; proceeding > INFO [main] 2022-03-30 16:08:35,730 NetworkTopologyStrategy.java:88 - Configured datacenter replicas are datacenter1:rf(3) > INFO [main] 2022-03-30 16:08:35,734 TokenAllocatorFactory.java:44 - Using ReplicationAwareTokenAllocator. > INFO [main] 2022-03-30 16:08:35,778 TokenAllocation.java:106 - Selected tokens [7159693272382515772, -78643292598952927, -4888993251840852361, 4268648890998440934, -7486162000054009734, -2014317050782326943, 2433125553643663124, -9076166552567908372, 6080320464934914114, -3110358951017265232, -5867138704356966436, 1361624937303630311, 8403213278063107744, -923019940090114683, 3450976958386090139, 5344404772224020473] > INFO [MigrationStage:1] 2022-03-30 16:08:35,847 ColumnFamilyStore.java:878 - Enqueuing flush of columns: 200.082KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,857 Memtable.java:469 - Writing Memtable-columns@1807913901(41.994KiB serialized bytes, 277 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,870 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb-1-big-Data.db (20.847KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42100) > INFO [MemtableFlushWriter:2] 2022-03-30 16:08:35,880 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_a757a880-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:35,883 ColumnFamilyStore.java:878 - Enqueuing flush of dropped_columns: 1.079KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,891 Memtable.java:469 - Writing Memtable-dropped_columns@2012660903(0.120KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,892 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb-1-big-Data.db (0.097KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42100) > INFO [MemtableFlushWriter:1] 2022-03-30 16:08:35,904 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb_txn_flush_a75cffb0-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:35,907 ColumnFamilyStore.java:878 - Enqueuing flush of triggers: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,914 Memtable.java:469 - Writing Memtable-triggers@1878858253(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,915 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42100) > INFO [MemtableFlushWriter:2] 2022-03-30 16:08:35,924 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb_txn_flush_a760d040-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:35,927 ColumnFamilyStore.java:878 - Enqueuing flush of types: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,932 Memtable.java:469 - Writing Memtable-types@2048419531(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,933 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) > INFO [MemtableFlushWriter:1] 2022-03-30 16:08:35,942 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb_txn_flush_a763dd80-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:35,949 ColumnFamilyStore.java:878 - Enqueuing flush of functions: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,956 Memtable.java:469 - Writing Memtable-functions@721394070(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:35,956 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) > INFO [MemtableFlushWriter:2] 2022-03-30 16:08:35,969 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb_txn_flush_a76711d0-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:35,972 ColumnFamilyStore.java:878 - Enqueuing flush of aggregates: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,978 Memtable.java:469 - Writing Memtable-aggregates@1568510948(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:35,983 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) > INFO [MemtableFlushWriter:1] 2022-03-30 16:08:35,995 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb_txn_flush_a76a9440-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:35,998 ColumnFamilyStore.java:878 - Enqueuing flush of indexes: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,006 Memtable.java:469 - Writing Memtable-indexes@1690274605(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,007 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) > INFO [MemtableFlushWriter:2] 2022-03-30 16:08:36,016 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb_txn_flush_a76eb2f0-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:36,019 ColumnFamilyStore.java:878 - Enqueuing flush of tables: 92.656KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,025 Memtable.java:469 - Writing Memtable-tables@1997400980(28.858KiB serialized bytes, 42 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,029 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb-1-big-Data.db (17.879KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) > INFO [MemtableFlushWriter:1] 2022-03-30 16:08:36,045 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb_txn_flush_a771c030-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:36,048 ColumnFamilyStore.java:878 - Enqueuing flush of views: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,054 Memtable.java:469 - Writing Memtable-views@463418348(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,055 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) > INFO [MemtableFlushWriter:2] 2022-03-30 16:08:36,065 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb_txn_flush_a7762d00-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:36,068 ColumnFamilyStore.java:878 - Enqueuing flush of keyspaces: 3.243KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,075 Memtable.java:469 - Writing Memtable-keyspaces@908057134(0.673KiB serialized bytes, 7 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,076 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb-1-big-Data.db (0.556KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42108) > INFO [MemtableFlushWriter:1] 2022-03-30 16:08:36,089 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb_txn_flush_a7796150-b043-11ec-8e41-d522e6a825ce.log > INFO [MigrationStage:1] 2022-03-30 16:08:36,162 Keyspace.java:386 - Creating replication strategy system_auth params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}} > INFO [MigrationStage:1] 2022-03-30 16:08:36,167 ColumnFamilyStore.java:385 - Initializing system_auth.network_permissions > INFO [MigrationStage:1] 2022-03-30 16:08:36,176 ColumnFamilyStore.java:385 - Initializing system_auth.resource_role_permissons_index > INFO [MigrationStage:1] 2022-03-30 16:08:36,182 ColumnFamilyStore.java:385 - Initializing system_auth.role_members > INFO [MigrationStage:1] 2022-03-30 16:08:36,189 ColumnFamilyStore.java:385 - Initializing system_auth.role_permissions > INFO [MigrationStage:1] 2022-03-30 16:08:36,195 ColumnFamilyStore.java:385 - Initializing system_auth.roles > INFO [MigrationStage:1] 2022-03-30 16:08:36,206 Keyspace.java:386 - Creating replication strategy system_distributed params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}} > INFO [MigrationStage:1] 2022-03-30 16:08:36,209 ColumnFamilyStore.java:385 - Initializing system_distributed.parent_repair_history > INFO [MigrationStage:1] 2022-03-30 16:08:36,218 ColumnFamilyStore.java:385 - Initializing system_distributed.repair_history > INFO [MigrationStage:1] 2022-03-30 16:08:36,224 ColumnFamilyStore.java:385 - Initializing system_distributed.view_build_status > INFO [MigrationStage:1] 2022-03-30 16:08:36,231 Keyspace.java:386 - Creating replication strategy system_traces params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}} > INFO [MigrationStage:1] 2022-03-30 16:08:36,234 ColumnFamilyStore.java:385 - Initializing system_traces.events > INFO [MigrationStage:1] 2022-03-30 16:08:36,242 ColumnFamilyStore.java:385 - Initializing system_traces.sessions > INFO [main] 2022-03-30 16:08:36,289 StorageService.java:1633 - JOINING: Finish joining ring > INFO [main] 2022-03-30 16:08:36,295 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.604KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,302 Memtable.java:469 - Writing Memtable-local@473268192(0.084KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:1] 2022-03-30 16:08:36,312 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db (0.058KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42264) > INFO [MemtableFlushWriter:2] 2022-03-30 16:08:36,330 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_a79c0480-b043-11ec-8e41-d522e6a825ce.log > INFO [main] 2022-03-30 16:08:36,351 ColumnFamilyStore.java:878 - Enqueuing flush of local: 2.853KiB (0%) on-heap, 0.000KiB (0%) off-heap > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,363 Memtable.java:469 - Writing Memtable-local@252234776(0.592KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (null, null] > INFO [PerDiskMemtableFlushWriter_0:2] 2022-03-30 16:08:36,365 Memtable.java:498 - Completed flushing /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db (0.360KiB) for commitlog position CommitLogPosition(segmentId=1648656465078, position=42676) > INFO [MemtableFlushWriter:1] 2022-03-30 16:08:36,375 LogTransaction.java:240 - Unfinished transaction log, deleting /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_a7a50530-b043-11ec-8e41-d522e6a825ce.log > INFO [main] 2022-03-30 16:08:36,397 StorageService.java:2784 - Node /172.17.0.5:7000 state jump to NORMAL > INFO [main] 2022-03-30 16:08:36,403 Gossiper.java:2214 - Waiting for gossip to settle... > INFO [main] 2022-03-30 16:08:44,405 Gossiper.java:2245 - No gossip backlog; proceeding > INFO [main] 2022-03-30 16:08:44,491 NativeTransportService.java:130 - Using Netty Version: [netty-buffer=netty-buffer-4.1.58.Final.10b03e6, netty-codec=netty-codec-4.1.58.Final.10b03e6, netty-codec-dns=netty-codec-dns-4.1.58.Final.10b03e6, netty-codec-haproxy=netty-codec-haproxy-4.1.58.Final.10b03e6, netty-codec-http=netty-codec-http-4.1.58.Final.10b03e6, netty-codec-http2=netty-codec-http2-4.1.58.Final.10b03e6, netty-codec-memcache=netty-codec-memcache-4.1.58.Final.10b03e6, netty-codec-mqtt=netty-codec-mqtt-4.1.58.Final.10b03e6, netty-codec-redis=netty-codec-redis-4.1.58.Final.10b03e6, netty-codec-smtp=netty-codec-smtp-4.1.58.Final.10b03e6, netty-codec-socks=netty-codec-socks-4.1.58.Final.10b03e6, netty-codec-stomp=netty-codec-stomp-4.1.58.Final.10b03e6, netty-codec-xml=netty-codec-xml-4.1.58.Final.10b03e6, netty-common=netty-common-4.1.58.Final.10b03e6, netty-handler=netty-handler-4.1.58.Final.10b03e6, netty-handler-proxy=netty-handler-proxy-4.1.58.Final.10b03e6, netty-resolver=netty-resolver-4.1.58.Final.10b03e6, netty-resolver-dns=netty-resolver-dns-4.1.58.Final.10b03e6, netty-resolver-dns-native-macos=netty-resolver-dns-native-macos-4.1.58.Final.10b03e65f1, netty-transport=netty-transport-4.1.58.Final.10b03e6, netty-transport-native-epoll=netty-transport-native-epoll-4.1.58.Final.10b03e6, netty-transport-native-kqueue=netty-transport-native-kqueue-4.1.58.Final.10b03e65f1, netty-transport-native-unix-common=netty-transport-native-unix-common-4.1.58.Final.10b03e6, netty-transport-rxtx=netty-transport-rxtx-4.1.58.Final.10b03e6, netty-transport-sctp=netty-transport-sctp-4.1.58.Final.10b03e6, netty-transport-udt=netty-transport-udt-4.1.58.Final.10b03e6] > INFO [CompactionExecutor:2] 2022-03-30 16:08:44,502 CompactionTask.java:150 - Compacting (ac7ca4f0-b043-11ec-8e41-d522e6a825ce) [/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db:level=0, /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db:level=0, ] > INFO [main] 2022-03-30 16:08:44,519 NativeTransportService.java:68 - Netty using native Epoll event loop > INFO [main] 2022-03-30 16:08:44,599 PipelineConfigurator.java:124 - Using Netty Version: [netty-buffer=netty-buffer-4.1.58.Final.10b03e6, netty-codec=netty-codec-4.1.58.Final.10b03e6, netty-codec-dns=netty-codec-dns-4.1.58.Final.10b03e6, netty-codec-haproxy=netty-codec-haproxy-4.1.58.Final.10b03e6, netty-codec-http=netty-codec-http-4.1.58.Final.10b03e6, netty-codec-http2=netty-codec-http2-4.1.58.Final.10b03e6, netty-codec-memcache=netty-codec-memcache-4.1.58.Final.10b03e6, netty-codec-mqtt=netty-codec-mqtt-4.1.58.Final.10b03e6, netty-codec-redis=netty-codec-redis-4.1.58.Final.10b03e6, netty-codec-smtp=netty-codec-smtp-4.1.58.Final.10b03e6, netty-codec-socks=netty-codec-socks-4.1.58.Final.10b03e6, netty-codec-stomp=netty-codec-stomp-4.1.58.Final.10b03e6, netty-codec-xml=netty-codec-xml-4.1.58.Final.10b03e6, netty-common=netty-common-4.1.58.Final.10b03e6, netty-handler=netty-handler-4.1.58.Final.10b03e6, netty-handler-proxy=netty-handler-proxy-4.1.58.Final.10b03e6, netty-resolver=netty-resolver-4.1.58.Final.10b03e6, netty-resolver-dns=netty-resolver-dns-4.1.58.Final.10b03e6, netty-resolver-dns-native-macos=netty-resolver-dns-native-macos-4.1.58.Final.10b03e65f1, netty-transport=netty-transport-4.1.58.Final.10b03e6, netty-transport-native-epoll=netty-transport-native-epoll-4.1.58.Final.10b03e6, netty-transport-native-kqueue=netty-transport-native-kqueue-4.1.58.Final.10b03e65f1, netty-transport-native-unix-common=netty-transport-native-unix-common-4.1.58.Final.10b03e6, netty-transport-rxtx=netty-transport-rxtx-4.1.58.Final.10b03e6, netty-transport-sctp=netty-transport-sctp-4.1.58.Final.10b03e6, netty-transport-udt=netty-transport-udt-4.1.58.Final.10b03e6] > INFO [main] 2022-03-30 16:08:44,600 PipelineConfigurator.java:125 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)... > INFO [main] 2022-03-30 16:08:44,614 CassandraDaemon.java:782 - Startup complete > INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,638 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big > INFO [CompactionExecutor:2] 2022-03-30 16:08:44,639 CompactionTask.java:241 - Compacted (ac7ca4f0-b043-11ec-8e41-d522e6a825ce) 5 sstables to [/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big,] to level=0. 0.773KiB to 0.629KiB (~81% of original) in 127ms. Read Throughput = 6.060KiB/s, Write Throughput = 4.927KiB/s, Row Throughput = ~2/s. 5 total partitions merged to 1. Partition merge counts were {5:1, } > INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,642 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big > INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,645 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big > INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,648 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big > INFO [NonPeriodicTasks:1] 2022-03-30 16:08:44,650 SSTable.java:111 - Deleting sstable: /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big > INFO [OptionalTasks:1] 2022-03-30 16:08:54,599 CassandraRoleManager.java:339 - Created default superuser role 'cassandra' ```

i mounted /var/lib/cassandra as a bind-mount volume， and the commitlog folder permissions are drwxrwxr-x， This is an occasional problem. I suspect that the service is interrupted due to out of memory, and finally the commitlog is damaged，but i do not know how to fix it.

[wglambert]

Hmm, you might try running nodetool repair on it https://support.axway.com/kb/181539/language/en

But otherwise you could try asking over at the Docker Community Forums, Docker Community Slack, or Stack Overflow