search

bitnami / containers

Bitnami container images
https://bitnami.com
Other
3.36k stars 4.82k forks source link

Unable to Internode Gossip over Multiple Machines #1009

Closed snowyaaron1 closed 2 years ago

snowyaaron1 commented 2 years ago

Name and Version

docker.io/bitnami/cassandra:4.0.4

What steps will reproduce the bug?

Run using docker-compose on 2 separate instances. IP: 10.117.50.41

version: '2'
services:
  cassandra:
    image: "bitnami/cassandra:4.0.4"
    ports:
      - 7000:7000
      - 7001:7001
      - 9042:9042
    environment:
      - CASSANDRA_SEEDS=10.117.50.41
      - CASSANDRA_CLUSTER_NAME=cassandra-cluster
      - CASSANDRA_PASSWORD=cassandra
      - CASSANDRA_BROADCAST_ADDRESS=10.117.50.41
      - BITNAMI_DEBUG=true
      # By default, Cassandra autodetects the available host memory and takes as much as it can.
      # Therefore, memory options are mandatory if multiple Cassandras are launched in the same node.
      - MAX_HEAP_SIZE=512M
      - HEAP_NEWSIZE=256M

IP: 10.117.50.47

version: '2'
services:
  cassandra:
    image: "bitnami/cassandra:4.0.4"
    ports:
      - 7000:7000
      - 7001:7001
      - 9042:9042
    environment:
      - CASSANDRA_SEEDS=10.117.50.41
      - CASSANDRA_CLUSTER_NAME=cassandra-cluster
      - CASSANDRA_PASSWORD=cassandra
      - CASSANDRA_BROADCAST_ADDRESS=10.117.50.47
      - BITNAMI_DEBUG=true
      # By default, Cassandra autodetects the available host memory and takes as much as it can.
      # Therefore, memory options are mandatory if multiple Cassandras are launched in the same node.
      - MAX_HEAP_SIZE=512M
      - HEAP_NEWSIZE=256M

What is the expected behavior?

Connect between two nodes via handshake and gossip

If I switch the image to "cassandra:4.0.4" the default Cassandra image this example gossips accordingly

What do you see instead?

Bitnami crashes on this error:

cassandra 20:52:18.01 ERROR ==> Nodetool output cassandra 20:52:18.00 ERROR ==> Cassandra failed to start up

BUT ITS EMPTY UNDER HERE

Additional information

I am not creating any docker networks. Im allowing docker to assign the IP at docker-compose up

I have no name!@651bbdc85957:/$ cat /opt/bitnami/cassandra/logs/cassandra.log 
INFO  [main] 2022-07-06 20:50:41,140 YamlConfigurationLoader.java:97 - Configuration location: file:/opt/bitnami/cassandra/conf/cassandra.yaml
INFO  [main] 2022-07-06 20:50:41,635 Config.java:706 - Node configuration:[allocate_tokens_for_keyspace=null; allocate_tokens_for_local_replication_factor=3; allow_extra_insecure_udfs=false; allow_insecure_udfs=false; audit_logging_options=AuditLogOptions{enabled=false, logger='BinAuditLogger', included_keyspaces='', excluded_keyspaces='system,system_schema,system_virtual_schema', included_categories='', excluded_categories='', included_users='', excluded_users='', audit_logs_dir='/opt/bitnami/cassandra/logs/audit/', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; authenticator=PasswordAuthenticator; authorizer=CassandraAuthorizer; auto_bootstrap=true; auto_optimise_full_repair_streams=false; auto_optimise_inc_repair_streams=false; auto_optimise_preview_repair_streams=false; auto_snapshot=true; autocompaction_on_startup_enabled=true; automatic_sstable_upgrade=false; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; block_for_peers_in_remote_dcs=false; block_for_peers_timeout_in_secs=10; broadcast_address=10.117.50.41; broadcast_rpc_address=194c9047e439; buffer_pool_use_heap_if_exhausted=false; cache_load_timeout_seconds=30; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=/bitnami/cassandra/data/cdc_raw; cdc_total_space_in_mb=0; check_for_duplicate_rows_during_compaction=true; check_for_duplicate_rows_during_reads=true; client_encryption_options=<REDACTED>; cluster_name=cassandra-cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/bitnami/cassandra/data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_group_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=64; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_builders=1; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_validations=0; concurrent_writes=32; consecutive_message_errors_threshold=1; corrupted_tombstone_strategy=disabled; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=true; data_file_directories=[Ljava.lang.String;@3e27ba32; diagnostic_events_enabled=false; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=1.0; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_drop_compact_storage=false; enable_materialized_views=false; enable_sasi_indexes=false; enable_scripted_user_defined_functions=false; enable_transient_replication=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; endpoint_snitch=SimpleSnitch; file_cache_enabled=false; file_cache_round_up=null; file_cache_size_in_mb=null; flush_compression=fast; force_new_prepared_statement_behaviour=false; full_query_logging_options=FullQueryLoggerOptions{log_dir='', archive_command='', roll_cycle='HOURLY', block=true, max_queue_weight=268435456, max_log_size=17179869184}; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/bitnami/cassandra/data/hints; hints_flush_period_in_ms=10000; ideal_consistency_level=null; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_range_tombstone_list_allocation_size=1; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_application_receive_queue_capacity_in_bytes=4194304; internode_application_receive_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_receive_queue_reserve_global_capacity_in_bytes=536870912; internode_application_send_queue_capacity_in_bytes=4194304; internode_application_send_queue_reserve_endpoint_capacity_in_bytes=134217728; internode_application_send_queue_reserve_global_capacity_in_bytes=536870912; internode_authenticator=null; internode_compression=dc; internode_max_message_size_in_bytes=null; internode_socket_receive_buffer_size_in_bytes=0; internode_socket_send_buffer_size_in_bytes=0; internode_streaming_tcp_user_timeout_in_ms=300000; internode_tcp_connect_timeout_in_ms=2000; internode_tcp_user_timeout_in_ms=30000; key_cache_keys_to_save=2147483647; key_cache_migrate_during_compaction=true; key_cache_save_period=14400; key_cache_size_in_mb=null; keyspace_count_warn_threshold=40; listen_address=194c9047e439; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; local_system_data_file_directory=null; max_concurrent_automatic_sstable_upgrades=1; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_allow_older_protocols=true; native_transport_flush_in_batches_legacy=false; native_transport_idle_timeout_in_ms=0; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_concurrent_requests_in_bytes=-1; native_transport_max_concurrent_requests_in_bytes_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_negotiable_protocol_version=null; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; native_transport_receive_queue_capacity_in_bytes=1048576; network_authorizer=AllowAllNetworkAuthorizer; networking_cache_size_in_mb=null; num_tokens=256; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; periodic_commitlog_sync_lag_block_in_ms=null; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; range_tombstone_list_growth_factor=1.5; read_request_timeout_in_ms=5000; reject_repair_compaction_threshold=2147483647; repair_command_pool_full_strategy=queue; repair_command_pool_size=0; repair_session_max_tree_depth=null; repair_session_space_in_mb=null; repaired_data_tracking_for_partition_reads_enabled=false; repaired_data_tracking_for_range_reads_enabled=false; report_unconfirmed_repaired_data_mismatches=false; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; saved_caches_directory=/bitnami/cassandra/data/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.117.50.41}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; snapshot_links_per_second=0; snapshot_on_duplicate_row_detection=false; snapshot_on_repaired_data_mismatch=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; storage_port=7000; stream_entire_sstables=true; stream_throughput_outbound_megabits_per_sec=200; streaming_connections_per_host=1; streaming_keep_alive_period_in_secs=300; table_count_warn_threshold=150; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@7ef82753; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; use_offheap_merkle_trees=true; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; validation_preview_purge_head_start_in_sec=3600; windows_timer_interval=1; write_request_timeout_in_ms=2000]
INFO  [main] 2022-07-06 20:50:41,640 DatabaseDescriptor.java:416 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  [main] 2022-07-06 20:50:41,641 DatabaseDescriptor.java:477 - Global memtable on-heap threshold is enabled at 121MB
INFO  [main] 2022-07-06 20:50:41,641 DatabaseDescriptor.java:481 - Global memtable off-heap threshold is enabled at 121MB
WARN  [main] 2022-07-06 20:50:41,658 DatabaseDescriptor.java:628 - Only 41.837GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
INFO  [main] 2022-07-06 20:50:41,780 JMXServerUtils.java:271 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
INFO  [main] 2022-07-06 20:50:41,786 CassandraDaemon.java:625 - Hostname: 194c9047e439:7000:7001
INFO  [main] 2022-07-06 20:50:41,786 CassandraDaemon.java:632 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_333
INFO  [main] 2022-07-06 20:50:41,787 CassandraDaemon.java:633 - Heap size: 486.438MiB/486.438MiB
INFO  [main] 2022-07-06 20:50:41,787 CassandraDaemon.java:638 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4682304(4572K) committed = 4718592(4608K) max = 251658240(245760K)
INFO  [main] 2022-07-06 20:50:41,787 CassandraDaemon.java:638 - Metaspace Non-heap memory: init = 0(0K) used = 22396928(21872K) committed = 22827008(22292K) max = -1(-1K)
INFO  [main] 2022-07-06 20:50:41,788 CassandraDaemon.java:638 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2661776(2599K) committed = 2830336(2764K) max = 1073741824(1048576K)
INFO  [main] 2022-07-06 20:50:41,788 CassandraDaemon.java:638 - Par Eden Space Heap memory: init = 214827008(209792K) used = 111713072(109094K) committed = 214827008(209792K) max = 214827008(209792K)
INFO  [main] 2022-07-06 20:50:41,788 CassandraDaemon.java:638 - Par Survivor Space Heap memory: init = 26804224(26176K) used = 0(0K) committed = 26804224(26176K) max = 26804224(26176K)
INFO  [main] 2022-07-06 20:50:41,788 CassandraDaemon.java:638 - CMS Old Gen Heap memory: init = 268435456(262144K) used = 0(0K) committed = 268435456(262144K) max = 268435456(262144K)
INFO  [main] 2022-07-06 20:50:41,788 CassandraDaemon.java:640 - Classpath: /opt/bitnami/cassandra/bin/../conf:/opt/bitnami/cassandra/bin/../lib/HdrHistogram-2.1.9.jar:/opt/bitnami/cassandra/bin/../lib/ST4-4.0.8.jar:/opt/bitnami/cassandra/bin/../lib/airline-0.8.jar:/opt/bitnami/cassandra/bin/../lib/antlr-runtime-3.5.2.jar:/opt/bitnami/cassandra/bin/../lib/apache-cassandra-4.0.4.jar:/opt/bitnami/cassandra/bin/../lib/asm-7.1.jar:/opt/bitnami/cassandra/bin/../lib/caffeine-2.5.6.jar:/opt/bitnami/cassandra/bin/../lib/cassandra-driver-core-3.11.0-shaded.jar:/opt/bitnami/cassandra/bin/../lib/chronicle-bytes-2.20.111.jar:/opt/bitnami/cassandra/bin/../lib/chronicle-core-2.20.126.jar:/opt/bitnami/cassandra/bin/../lib/chronicle-queue-5.20.123.jar:/opt/bitnami/cassandra/bin/../lib/chronicle-threads-2.20.111.jar:/opt/bitnami/cassandra/bin/../lib/chronicle-wire-2.20.117.jar:/opt/bitnami/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/bitnami/cassandra/bin/../lib/commons-codec-1.9.jar:/opt/bitnami/cassandra/bin/../lib/commons-lang3-3.11.jar:/opt/bitnami/cassandra/bin/../lib/commons-math3-3.2.jar:/opt/bitnami/cassandra/bin/../lib/concurrent-trees-2.4.0.jar:/opt/bitnami/cassandra/bin/../lib/ecj-4.6.1.jar:/opt/bitnami/cassandra/bin/../lib/guava-27.0-jre.jar:/opt/bitnami/cassandra/bin/../lib/high-scale-lib-1.0.6.jar:/opt/bitnami/cassandra/bin/../lib/hppc-0.8.1.jar:/opt/bitnami/cassandra/bin/../lib/j2objc-annotations-1.3.jar:/opt/bitnami/cassandra/bin/../lib/jackson-annotations-2.13.2.jar:/opt/bitnami/cassandra/bin/../lib/jackson-core-2.13.2.jar:/opt/bitnami/cassandra/bin/../lib/jackson-databind-2.13.2.2.jar:/opt/bitnami/cassandra/bin/../lib/jamm-0.3.2.jar:/opt/bitnami/cassandra/bin/../lib/java-cup-runtime-11b-20160615.jar:/opt/bitnami/cassandra/bin/../lib/javax.inject-1.jar:/opt/bitnami/cassandra/bin/../lib/jbcrypt-0.4.jar:/opt/bitnami/cassandra/bin/../lib/jcl-over-slf4j-1.7.25.jar:/opt/bitnami/cassandra/bin/../lib/jcommander-1.30.jar:/opt/bitnami/cassandra/bin/../lib/jctools-core-3.1.0.jar:/opt/bitnami/cassandra/bin/../lib/jflex-1.8.2.jar:/opt/bitnami/cassandra/bin/../lib/jna-5.6.0.jar:/opt/bitnami/cassandra/bin/../lib/json-simple-1.1.jar:/opt/bitnami/cassandra/bin/../lib/jvm-attach-api-1.5.jar:/opt/bitnami/cassandra/bin/../lib/log4j-over-slf4j-1.7.25.jar:/opt/bitnami/cassandra/bin/../lib/logback-classic-1.2.9.jar:/opt/bitnami/cassandra/bin/../lib/logback-core-1.2.9.jar:/opt/bitnami/cassandra/bin/../lib/lz4-java-1.8.0.jar:/opt/bitnami/cassandra/bin/../lib/metrics-core-3.1.5.jar:/opt/bitnami/cassandra/bin/../lib/metrics-jvm-3.1.5.jar:/opt/bitnami/cassandra/bin/../lib/metrics-logback-3.1.5.jar:/opt/bitnami/cassandra/bin/../lib/mxdump-0.14.jar:/opt/bitnami/cassandra/bin/../lib/netty-all-4.1.58.Final.jar:/opt/bitnami/cassandra/bin/../lib/netty-tcnative-boringssl-static-2.0.36.Final.jar:/opt/bitnami/cassandra/bin/../lib/ohc-core-0.5.1.jar:/opt/bitnami/cassandra/bin/../lib/ohc-core-j8-0.5.1.jar:/opt/bitnami/cassandra/bin/../lib/psjava-0.1.19.jar:/opt/bitnami/cassandra/bin/../lib/reporter-config-base-3.0.3.jar:/opt/bitnami/cassandra/bin/../lib/reporter-config3-3.0.3.jar:/opt/bitnami/cassandra/bin/../lib/sigar-1.6.4.jar:/opt/bitnami/cassandra/bin/../lib/sjk-cli-0.14.jar:/opt/bitnami/cassandra/bin/../lib/sjk-core-0.14.jar:/opt/bitnami/cassandra/bin/../lib/sjk-json-0.14.jar:/opt/bitnami/cassandra/bin/../lib/sjk-stacktrace-0.14.jar:/opt/bitnami/cassandra/bin/../lib/slf4j-api-1.7.25.jar:/opt/bitnami/cassandra/bin/../lib/snakeyaml-1.26.jar:/opt/bitnami/cassandra/bin/../lib/snappy-java-1.1.2.6.jar:/opt/bitnami/cassandra/bin/../lib/snowball-stemmer-1.3.0.581.1.jar:/opt/bitnami/cassandra/bin/../lib/stream-2.5.2.jar:/opt/bitnami/cassandra/bin/../lib/zstd-jni-1.5.0-4.jar:/opt/bitnami/cassandra/bin/../lib/jsr223/*/*.jar::/opt/bitnami/cassandra/bin/../lib/jamm-0.3.2.jar
INFO  [main] 2022-07-06 20:50:41,790 CassandraDaemon.java:642 - JVM Arguments: [-ea, -da:net.openhft..., -XX:+UseThreadPriorities, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:ThreadPriorityPolicy=42, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xloggc:/opt/bitnami/cassandra/logs/gc.log, -Xms512M, -Xmx512M, -Xmn256M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/opt/bitnami/cassandra/bin/../conf/hotspot_compiler, -javaagent:/opt/bitnami/cassandra/bin/../lib/jamm-0.3.2.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/bitnami/cassandra/bin/../lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/bitnami/cassandra/logs, -Dcassandra.storagedir=/opt/bitnami/cassandra/bin/../data, -Dcassandra-pidfile=/opt/bitnami/cassandra/tmp/cassandra.pid, -Dcassandra-foreground=yes]
WARN  [main] 2022-07-06 20:50:41,898 NativeLibrary.java:201 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK.
INFO  [main] 2022-07-06 20:50:41,959 MonotonicClock.java:202 - Scheduling approximate time conversion task with an interval of 10000 milliseconds
INFO  [main] 2022-07-06 20:50:41,963 MonotonicClock.java:338 - Scheduling approximate time-check task with a precision of 2 milliseconds
INFO  [main] 2022-07-06 20:50:41,965 StartupChecks.java:147 - jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.2
WARN  [main] 2022-07-06 20:50:41,966 StartupChecks.java:187 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
INFO  [main] 2022-07-06 20:50:41,968 SigarLibrary.java:44 - Initializing SIGAR library
INFO  [main] 2022-07-06 20:50:41,979 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
WARN  [main] 2022-07-06 20:50:41,980 StartupChecks.java:329 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
INFO  [main] 2022-07-06 20:50:42,587 Keyspace.java:386 - Creating replication strategy system params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}}
INFO  [main] 2022-07-06 20:50:42,636 ColumnFamilyStore.java:385 - Initializing system.IndexInfo
INFO  [main] 2022-07-06 20:50:44,038 ColumnFamilyStore.java:385 - Initializing system.batches
INFO  [main] 2022-07-06 20:50:44,049 ColumnFamilyStore.java:385 - Initializing system.paxos
INFO  [main] 2022-07-06 20:50:44,063 ColumnFamilyStore.java:385 - Initializing system.local
INFO  [main] 2022-07-06 20:50:44,085 ColumnFamilyStore.java:385 - Initializing system.peers_v2
INFO  [main] 2022-07-06 20:50:44,099 ColumnFamilyStore.java:385 - Initializing system.peers
INFO  [main] 2022-07-06 20:50:44,118 ColumnFamilyStore.java:385 - Initializing system.peer_events_v2
INFO  [main] 2022-07-06 20:50:44,134 ColumnFamilyStore.java:385 - Initializing system.peer_events
INFO  [main] 2022-07-06 20:50:44,141 ColumnFamilyStore.java:385 - Initializing system.compaction_history
INFO  [main] 2022-07-06 20:50:44,155 ColumnFamilyStore.java:385 - Initializing system.sstable_activity
INFO  [main] 2022-07-06 20:50:44,164 ColumnFamilyStore.java:385 - Initializing system.size_estimates
INFO  [main] 2022-07-06 20:50:44,173 ColumnFamilyStore.java:385 - Initializing system.table_estimates
INFO  [main] 2022-07-06 20:50:44,181 ColumnFamilyStore.java:385 - Initializing system.available_ranges_v2
INFO  [main] 2022-07-06 20:50:44,189 ColumnFamilyStore.java:385 - Initializing system.available_ranges
INFO  [main] 2022-07-06 20:50:44,197 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges_v2
INFO  [main] 2022-07-06 20:50:44,203 ColumnFamilyStore.java:385 - Initializing system.transferred_ranges
INFO  [main] 2022-07-06 20:50:44,210 ColumnFamilyStore.java:385 - Initializing system.view_builds_in_progress
INFO  [main] 2022-07-06 20:50:44,218 ColumnFamilyStore.java:385 - Initializing system.built_views
INFO  [main] 2022-07-06 20:50:44,224 ColumnFamilyStore.java:385 - Initializing system.prepared_statements
INFO  [main] 2022-07-06 20:50:44,233 ColumnFamilyStore.java:385 - Initializing system.repairs
INFO  [main] 2022-07-06 20:50:44,461 QueryProcessor.java:115 - Initialized prepared statement caches with 10 MB
INFO  [main] 2022-07-06 20:50:44,619 Keyspace.java:386 - Creating replication strategy system_schema params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}}
INFO  [main] 2022-07-06 20:50:44,622 ColumnFamilyStore.java:385 - Initializing system_schema.keyspaces
INFO  [main] 2022-07-06 20:50:44,628 ColumnFamilyStore.java:385 - Initializing system_schema.tables
INFO  [main] 2022-07-06 20:50:44,634 ColumnFamilyStore.java:385 - Initializing system_schema.columns
INFO  [main] 2022-07-06 20:50:44,639 ColumnFamilyStore.java:385 - Initializing system_schema.triggers
INFO  [main] 2022-07-06 20:50:44,646 ColumnFamilyStore.java:385 - Initializing system_schema.dropped_columns
INFO  [main] 2022-07-06 20:50:44,652 ColumnFamilyStore.java:385 - Initializing system_schema.views
INFO  [main] 2022-07-06 20:50:44,658 ColumnFamilyStore.java:385 - Initializing system_schema.types
INFO  [main] 2022-07-06 20:50:44,663 ColumnFamilyStore.java:385 - Initializing system_schema.functions
INFO  [main] 2022-07-06 20:50:44,669 ColumnFamilyStore.java:385 - Initializing system_schema.aggregates
INFO  [main] 2022-07-06 20:50:44,675 ColumnFamilyStore.java:385 - Initializing system_schema.indexes
INFO  [main] 2022-07-06 20:50:44,765 SystemKeyspaceMigrator40.java:76 - system.peers_v2 table was empty, migrating legacy system.peers, if this fails you should fix the issue and then truncate system.peers_v2 to have it try again.
INFO  [main] 2022-07-06 20:50:44,775 SystemKeyspaceMigrator40.java:100 - Migrating rows from legacy system.peers to system.peers_v2
INFO  [main] 2022-07-06 20:50:44,781 SystemKeyspaceMigrator40.java:119 - Migrated 0 rows from legacy system.peers to system.peers_v2
INFO  [main] 2022-07-06 20:50:44,781 SystemKeyspaceMigrator40.java:129 - system.peer_events_v2 table was empty, migrating legacy system.peer_events to system.peer_events_v2
INFO  [main] 2022-07-06 20:50:44,782 SystemKeyspaceMigrator40.java:152 - Migrated 0 rows from legacy system.peer_events to system.peer_events_v2
INFO  [main] 2022-07-06 20:50:44,783 SystemKeyspaceMigrator40.java:162 - system.transferred_ranges_v2 table was empty, migrating legacy system.transferred_ranges to system.transferred_ranges_v2
INFO  [main] 2022-07-06 20:50:44,785 SystemKeyspaceMigrator40.java:190 - Migrated 0 rows from legacy system.transferred_ranges to system.transferred_ranges_v2
INFO  [main] 2022-07-06 20:50:44,786 SystemKeyspaceMigrator40.java:200 - system.available_ranges_v2 table was empty, migrating legacy system.available_ranges to system.available_ranges_v2
INFO  [main] 2022-07-06 20:50:44,786 SystemKeyspaceMigrator40.java:226 - Migrated 0 rows from legacy system.available_ranges to system.available_ranges_v2
INFO  [main] 2022-07-06 20:50:44,787 StorageService.java:838 - Populating token metadata from system tables
INFO  [main] 2022-07-06 20:50:44,821 StorageService.java:832 - Token metadata: 
INFO  [main] 2022-07-06 20:50:44,871 CacheService.java:103 - Initializing key cache with capacity of 24 MBs.
INFO  [main] 2022-07-06 20:50:44,886 CacheService.java:125 - Initializing row cache with capacity of 0 MBs
INFO  [main] 2022-07-06 20:50:44,888 CacheService.java:154 - Initializing counter cache with capacity of 12 MBs
INFO  [main] 2022-07-06 20:50:44,889 CacheService.java:165 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  [main] 2022-07-06 20:50:44,923 CommitLog.java:168 - No commitlog files found; skipping replay
INFO  [main] 2022-07-06 20:50:44,923 StorageService.java:838 - Populating token metadata from system tables
INFO  [main] 2022-07-06 20:50:44,954 StorageService.java:832 - Token metadata: 
INFO  [main] 2022-07-06 20:50:45,115 ColumnFamilyStore.java:2252 - Truncating system.size_estimates
INFO  [main] 2022-07-06 20:50:45,156 ColumnFamilyStore.java:2289 - Truncating system.size_estimates with truncatedAt=1657140645129
INFO  [main] 2022-07-06 20:50:45,203 ColumnFamilyStore.java:878 - Enqueuing flush of local: 2.213KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:50:45,295 Memtable.java:469 - Writing Memtable-local@1822595959(0.434KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:50:45,298 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big-Data.db (0.229KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=31212)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:50:45,372 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_4e82cba0-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [main] 2022-07-06 20:50:45,387 ColumnFamilyStore.java:2316 - Truncate of system.size_estimates is complete
INFO  [main] 2022-07-06 20:50:45,387 ColumnFamilyStore.java:2252 - Truncating system.table_estimates
INFO  [main] 2022-07-06 20:50:45,388 ColumnFamilyStore.java:2289 - Truncating system.table_estimates with truncatedAt=1657140645388
INFO  [main] 2022-07-06 20:50:45,390 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.575KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:50:45,401 Memtable.java:469 - Writing Memtable-local@991622006(0.091KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:50:45,402 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big-Data.db (0.063KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=31323)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:50:45,425 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_4e9edf20-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [main] 2022-07-06 20:50:45,427 ColumnFamilyStore.java:2316 - Truncate of system.table_estimates is complete
INFO  [main] 2022-07-06 20:50:45,433 QueryProcessor.java:167 - Preloaded 0 prepared statements
INFO  [main] 2022-07-06 20:50:45,434 StorageService.java:736 - Cassandra version: 4.0.4
INFO  [main] 2022-07-06 20:50:45,434 StorageService.java:737 - CQL version: 3.4.5
INFO  [main] 2022-07-06 20:50:45,434 StorageService.java:738 - Native protocol supported versions: 3/v3, 4/v4, 5/v5, 6/v6-beta (default: 5/v5)
INFO  [main] 2022-07-06 20:50:45,482 IndexSummaryManager.java:84 - Initializing index summary manager with a memory pool size of 24 MB and a resize interval of 60 minutes
INFO  [main] 2022-07-06 20:50:45,484 StorageService.java:755 - Loading persisted ring state
INFO  [main] 2022-07-06 20:50:45,484 StorageService.java:838 - Populating token metadata from system tables
INFO  [main] 2022-07-06 20:50:45,515 BufferPools.java:49 - Global buffer pool limit is 121.000MiB for chunk-cache and 30.000MiB for networking
INFO  [main] 2022-07-06 20:50:45,720 InboundConnectionInitiator.java:127 - Listening on address: (194c9047e439/192.168.0.2:7000), nic: eth0, encryption: unencrypted
WARN  [main] 2022-07-06 20:50:45,798 SystemKeyspace.java:1130 - No host ID found, created 45797ec1-493f-4836-9d6d-05f196a8b074 (Note: This should happen exactly once per node).
INFO  [main] 2022-07-06 20:50:45,828 StorageService.java:653 - Unable to gossip with any peers but continuing anyway since node is in its own seed list
INFO  [main] 2022-07-06 20:50:45,868 StorageService.java:960 - Starting up server gossip
INFO  [main] 2022-07-06 20:50:45,872 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.583KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:50:45,883 Memtable.java:469 - Writing Memtable-local@1212460448(0.079KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:50:45,884 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=31487)
INFO  [MemtableFlushWriter:1] 2022-07-06 20:50:45,901 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_4ee7f610-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [main] 2022-07-06 20:50:46,010 StorageService.java:1050 - This node will not auto bootstrap because it is configured to be a seed node.
INFO  [main] 2022-07-06 20:51:16,022 Gossiper.java:2214 - Waiting for gossip to settle...
INFO  [main] 2022-07-06 20:51:24,023 Gossiper.java:2245 - No gossip backlog; proceeding
INFO  [main] 2022-07-06 20:51:24,024 Gossiper.java:2214 - Waiting for gossip to settle...
INFO  [main] 2022-07-06 20:51:32,025 Gossiper.java:2245 - No gossip backlog; proceeding
INFO  [main] 2022-07-06 20:51:32,031 NetworkTopologyStrategy.java:88 - Configured datacenter replicas are datacenter1:rf(3)
INFO  [main] 2022-07-06 20:51:32,033 TokenAllocatorFactory.java:44 - Using ReplicationAwareTokenAllocator.
INFO  [main] 2022-07-06 20:51:32,130 TokenAllocation.java:106 - Selected tokens [-7647052918464736098, 3561354590263346818, -1248995368978552615, 7908646773860740679, -3846164117191709987, 1625680832079972802, 6073123436505962869, -5436168669705608625, -8726425725912337756, 529638931845034513, -2227140821494666690, 5001622820165930056, -6403532912784144124, 2716977942772185061, 7090974841248389884, 8984402655086320218, -137130302923822609, -2938502086619755859, -4482686852690421064, 4401889539786632810, -6887249975736480972, -551755222656404633, 1175568064220984641, 2290012869853841930, -8070412281946332441, 8569250909408979841, 5661349157825897225, 6675985486803066619, -1620459279827994168, -5859069465072800765, -4903156997842731225, -3277804778862317279, 3229483484977027839, 4071066524652027827, 7551755815630638866, -7200662428165701792, -9055289651956595395, -2541456665691111158, -845350552519933962, 248391753694119328, 2035731907305357913, 8321955355651926315, 5409147222232299886, -8333650804902988292, 895370791009399207, -4120439099805810361, -1852895394012533975, 6406814765530453069, 4764553280209784955, -3499104082560258413, -6073458502071939030, -5133082745033874949, 3030384481813794794, 3879254990345229399, -6619341282120896206, 7359311411625705557, 1453206942463795899, -7373978183703199819, 2557846694287825076, -7808344692655605969, -5619894423172608747, -4640387723777981149, 8821293076292049883, 6926468067410785846, -297542025163810485, 8150801549272546776, 5903243076969554301, 1878579203089753461, 5242235015175190033, -9207045825386550590, -996522944076056355, -2708505839781843195, -8507912725946795284, 78051985104655656, -2019446035720628531, -1401644128619677156, 758471015639977625, 4629268894318298324, -4261590906538637289, 7762461098808490784, -3647662851758459856, -3082375498185706916, 6278813255054359155, 3414233721519616554, 4254193460305402162, -6212312430284921574, -8871358773339837998, 3757359187230992164, -2367620462150266215, -7020174969005223218, 2902720536466211793, -5247740917254318023, -676950978068285668, 422302653550406321, 1063076569498879792, 1347963579292325967, -3967908418517000574, -7482855620845593241, 6565638057793318559, 7249505141817841268, -6728691412644586442, 2445427051615623868, -8178706195634841421, -4744028094986657358, -7919200858117347271, 9125749745320221959, 2189664530524218445, -397018733195485354, 1776998278954002311, -1104980670977855562, 5555919831711164786, 8725806027194991825, 6817593455742664423, 8458729560566219631, 8047317575713468991, 5811262352123075140, 5139999861827857856, -5719163010103531454, 4907826915110647166, -1717286670880990110, -2805273232046070576, -4994398823649768194, 668841537146473662, 4542834963437130380, -3367147017129843858, -4351998572110788918, -1494381461254446571, -8596781174241735641, -6484294991420027028, -8277163862558248, -5954510300969190825, 7679119303816381830, -2102511882922324836, 6199442535360749967, 3151510398212022441, -3727034841129812845, 3674332292775898336, -3158776078398706964, 7468437566064816111, 3987048086305068914, -6287599169847890138, -5318822528395814621, 2823987884636593975, 3338948251882297297, -8951146195095385252, -5516764094940806933, 4181690210993849806, -7095757673959583239, -8403840268442980488, 356796000008573354, -2438398338372175843, -7275387341708566719, 1552865137937687697, 1274105309638436499, 8254854836508415220, 173377934646558571, 6000521235472181524, -749878683165400164, 991821488140147327, -2606660682406397341, 5335085022288193998, -1926303997182559248, 7028413438575580657, -7549702916695981983, 8912505515086235689, -7712757083114006997, -207985653015943076, 2647921995922370139, -4804648690098932957, 6498784389359291632, -6787876671905933328, 7185455467395563048, -4546066049957912515, 1968670063449878651, 8661818102128967140, 2386935669184058467, -8245469087066444892, -455660298352129017, 2121047326304445412, -1314122640553858979, -4028471348491151908, -9119880544820196970, 1710053296815646327, -7984258656504243483, -911368831603783819, 5751006274668103057, -3557433084999792106, 4339203809420206075, 3502460434095531630, 5490675130189327231, 7848504331900473511, -8790234431717454991, -1161710452674607462, -2998159621200026784, 4852557314901324663, 6760684800593489820, 9069663894231908173, -4178799515781024829, 4483444573672849648, -2287126890559986774, -5773912261462374074, 608740631065784552, -6133998643716253897, -5046847096768831701, 7992454335321821303, 5087629563479052318, 834510887294549929, 8399564217622692181, -1772074371686282992, 4713843316247558003, -6541329387101109765, -2863474141009361183, -6945725805421827589, -3418047684042273953, -4403877830197135913, -8650589757810773864, -61263846508553744, 6357534940642175229, 2978457394223371562, 7627743002405883204, 6148610524036376398, -1547225070165933826, -607729061612776834, -2151802685928826954, 3831266740620341629, -3899977546001836009, 3101914004016147392, -3774133923650676716, -3204858365765318082, -6001267928758508391, -5368295626900534717, -6338891955869318687, -5181355810171745244, 9193190580349011720, 3631910244300663192, 1131107242827278850, 2508596531918700643, -7853719437028290794, 4139695928663905226, 8521210062588831631, 6629527976020273988, 7315929493734870754, 3290260410321240267, -6666111533960247484, 7423440014931961394, -7419350969666251771, 6879292340475019865, -1042639219655234391, 312997166711966244, -8117474548154321035, 3941846865366221478, 483525661819137277, 2780452867104287389, 5617006344861806782, 1408588101267723107, -7136083531527137696]
INFO  [MigrationStage:1] 2022-07-06 20:51:32,172 ColumnFamilyStore.java:878 - Enqueuing flush of columns: 200.121KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,181 Memtable.java:469 - Writing Memtable-columns@903678565(41.994KiB serialized bytes, 277 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,201 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb-1-big-Data.db (20.847KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42113)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:51:32,218 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_flush_6a80eee0-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,221 ColumnFamilyStore.java:878 - Enqueuing flush of dropped_columns: 1.079KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,231 Memtable.java:469 - Writing Memtable-dropped_columns@945723512(0.120KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,232 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb-1-big-Data.db (0.097KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:1] 2022-07-06 20:51:32,258 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/dropped_columns-5e7583b5f3f43af19a39b7e1d6f5f11f/nb_txn_flush_6a8868f0-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,260 ColumnFamilyStore.java:878 - Enqueuing flush of triggers: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,265 Memtable.java:469 - Writing Memtable-triggers@836933638(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,266 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:51:32,289 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/triggers-4df70b666b05325195a132b54005fd48/nb_txn_flush_6a8e3550-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,294 ColumnFamilyStore.java:878 - Enqueuing flush of types: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,302 Memtable.java:469 - Writing Memtable-types@1194930721(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,303 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:1] 2022-07-06 20:51:32,322 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/types-5a8b1ca866023f77a0459273d308917a/nb_txn_flush_6a936570-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,325 ColumnFamilyStore.java:878 - Enqueuing flush of functions: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,331 Memtable.java:469 - Writing Memtable-functions@1960837214(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,332 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:51:32,352 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/functions-96489b7980be3e14a70166a0b9159450/nb_txn_flush_6a982060-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,355 ColumnFamilyStore.java:878 - Enqueuing flush of aggregates: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,363 Memtable.java:469 - Writing Memtable-aggregates@1393683104(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,363 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:1] 2022-07-06 20:51:32,383 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/aggregates-924c55872e3a345bb10c12f37c1ba895/nb_txn_flush_6a9cdb50-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,386 ColumnFamilyStore.java:878 - Enqueuing flush of indexes: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,392 Memtable.java:469 - Writing Memtable-indexes@411950327(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,392 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:51:32,411 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/indexes-0feb57ac311f382fba6d9024d305702f/nb_txn_flush_6aa16f30-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,413 ColumnFamilyStore.java:878 - Enqueuing flush of tables: 92.656KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,419 Memtable.java:469 - Writing Memtable-tables@1054172367(28.858KiB serialized bytes, 42 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,422 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb-1-big-Data.db (17.879KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:1] 2022-07-06 20:51:32,445 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/tables-afddfb9dbc1e30688056eed6c302ba09/nb_txn_flush_6aa5b4f0-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,447 ColumnFamilyStore.java:878 - Enqueuing flush of views: 0.511KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,456 Memtable.java:469 - Writing Memtable-views@1163817889(0.016KiB serialized bytes, 2 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,456 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb-1-big-Data.db (0.048KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:51:32,476 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/views-9786ac1cdd583201a7cdad556410c985/nb_txn_flush_6aaa96f0-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,478 ColumnFamilyStore.java:878 - Enqueuing flush of keyspaces: 3.243KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,485 Memtable.java:469 - Writing Memtable-keyspaces@1921300021(0.673KiB serialized bytes, 7 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,486 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb-1-big-Data.db (0.556KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42121)
INFO  [MemtableFlushWriter:1] 2022-07-06 20:51:32,500 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system_schema/keyspaces-abac5682dea631c5b535b3d6cffd0fb6/nb_txn_flush_6aaf78f0-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [MigrationStage:1] 2022-07-06 20:51:32,578 Keyspace.java:386 - Creating replication strategy system_auth params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}
INFO  [MigrationStage:1] 2022-07-06 20:51:32,581 ColumnFamilyStore.java:385 - Initializing system_auth.network_permissions
INFO  [MigrationStage:1] 2022-07-06 20:51:32,587 ColumnFamilyStore.java:385 - Initializing system_auth.resource_role_permissons_index
INFO  [MigrationStage:1] 2022-07-06 20:51:32,592 ColumnFamilyStore.java:385 - Initializing system_auth.role_members
INFO  [MigrationStage:1] 2022-07-06 20:51:32,597 ColumnFamilyStore.java:385 - Initializing system_auth.role_permissions
INFO  [MigrationStage:1] 2022-07-06 20:51:32,602 ColumnFamilyStore.java:385 - Initializing system_auth.roles
INFO  [MigrationStage:1] 2022-07-06 20:51:32,629 Keyspace.java:386 - Creating replication strategy system_distributed params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}
INFO  [MigrationStage:1] 2022-07-06 20:51:32,632 ColumnFamilyStore.java:385 - Initializing system_distributed.parent_repair_history
INFO  [MigrationStage:1] 2022-07-06 20:51:32,639 ColumnFamilyStore.java:385 - Initializing system_distributed.repair_history
INFO  [MigrationStage:1] 2022-07-06 20:51:32,645 ColumnFamilyStore.java:385 - Initializing system_distributed.view_build_status
INFO  [MigrationStage:1] 2022-07-06 20:51:32,650 Keyspace.java:386 - Creating replication strategy system_traces params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}
INFO  [MigrationStage:1] 2022-07-06 20:51:32,654 ColumnFamilyStore.java:385 - Initializing system_traces.events
INFO  [MigrationStage:1] 2022-07-06 20:51:32,661 ColumnFamilyStore.java:385 - Initializing system_traces.sessions
INFO  [main] 2022-07-06 20:51:32,716 StorageService.java:1634 - JOINING: Finish joining ring
INFO  [main] 2022-07-06 20:51:32,720 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.604KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,729 Memtable.java:469 - Writing Memtable-local@1624895908(0.084KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:2] 2022-07-06 20:51:32,730 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db (0.058KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=42277)
INFO  [MemtableFlushWriter:2] 2022-07-06 20:51:32,745 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_6ad46610-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [main] 2022-07-06 20:51:32,764 ColumnFamilyStore.java:878 - Enqueuing flush of local: 47.635KiB (0%) on-heap, 0.000KiB (0%) off-heap
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,770 Memtable.java:469 - Writing Memtable-local@2091002494(8.883KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (null, null]
INFO  [PerDiskMemtableFlushWriter_0:1] 2022-07-06 20:51:32,771 Memtable.java:498 - Completed flushing /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db (5.371KiB) for commitlog position CommitLogPosition(segmentId=1657140641952, position=47812)
INFO  [MemtableFlushWriter:1] 2022-07-06 20:51:32,785 LogTransaction.java:240 - Unfinished transaction log, deleting /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb_txn_flush_6adb1cd0-fd6d-11ec-9792-3f8ce372cd42.log 
INFO  [main] 2022-07-06 20:51:32,807 StorageService.java:2785 - Node /10.117.50.41:7000 state jump to NORMAL
INFO  [main] 2022-07-06 20:51:32,812 Gossiper.java:2214 - Waiting for gossip to settle...
INFO  [main] 2022-07-06 20:51:40,812 Gossiper.java:2245 - No gossip backlog; proceeding
INFO  [main] 2022-07-06 20:51:40,821 AuthCache.java:215 - (Re)initializing CredentialsCache (validity period/update interval/max entries) (2000/2000/1000)
INFO  [main] 2022-07-06 20:51:40,923 NativeTransportService.java:130 - Using Netty Version: [netty-buffer=netty-buffer-4.1.58.Final.10b03e6, netty-codec=netty-codec-4.1.58.Final.10b03e6, netty-codec-dns=netty-codec-dns-4.1.58.Final.10b03e6, netty-codec-haproxy=netty-codec-haproxy-4.1.58.Final.10b03e6, netty-codec-http=netty-codec-http-4.1.58.Final.10b03e6, netty-codec-http2=netty-codec-http2-4.1.58.Final.10b03e6, netty-codec-memcache=netty-codec-memcache-4.1.58.Final.10b03e6, netty-codec-mqtt=netty-codec-mqtt-4.1.58.Final.10b03e6, netty-codec-redis=netty-codec-redis-4.1.58.Final.10b03e6, netty-codec-smtp=netty-codec-smtp-4.1.58.Final.10b03e6, netty-codec-socks=netty-codec-socks-4.1.58.Final.10b03e6, netty-codec-stomp=netty-codec-stomp-4.1.58.Final.10b03e6, netty-codec-xml=netty-codec-xml-4.1.58.Final.10b03e6, netty-common=netty-common-4.1.58.Final.10b03e6, netty-handler=netty-handler-4.1.58.Final.10b03e6, netty-handler-proxy=netty-handler-proxy-4.1.58.Final.10b03e6, netty-resolver=netty-resolver-4.1.58.Final.10b03e6, netty-resolver-dns=netty-resolver-dns-4.1.58.Final.10b03e6, netty-resolver-dns-native-macos=netty-resolver-dns-native-macos-4.1.58.Final.10b03e65f1, netty-transport=netty-transport-4.1.58.Final.10b03e6, netty-transport-native-epoll=netty-transport-native-epoll-4.1.58.Final.10b03e6, netty-transport-native-kqueue=netty-transport-native-kqueue-4.1.58.Final.10b03e65f1, netty-transport-native-unix-common=netty-transport-native-unix-common-4.1.58.Final.10b03e6, netty-transport-rxtx=netty-transport-rxtx-4.1.58.Final.10b03e6, netty-transport-sctp=netty-transport-sctp-4.1.58.Final.10b03e6, netty-transport-udt=netty-transport-udt-4.1.58.Final.10b03e6]
INFO  [main] 2022-07-06 20:51:40,972 NativeTransportService.java:68 - Netty using native Epoll event loop
INFO  [CompactionExecutor:2] 2022-07-06 20:51:40,978 CompactionTask.java:150 - Compacting (6fb5a2c0-fd6d-11ec-9792-3f8ce372cd42) [/bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big-Data.db:level=0, /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big-Data.db:level=0, /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big-Data.db:level=0, /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big-Data.db:level=0, /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db:level=0, ]
INFO  [main] 2022-07-06 20:51:41,074 PipelineConfigurator.java:124 - Using Netty Version: [netty-buffer=netty-buffer-4.1.58.Final.10b03e6, netty-codec=netty-codec-4.1.58.Final.10b03e6, netty-codec-dns=netty-codec-dns-4.1.58.Final.10b03e6, netty-codec-haproxy=netty-codec-haproxy-4.1.58.Final.10b03e6, netty-codec-http=netty-codec-http-4.1.58.Final.10b03e6, netty-codec-http2=netty-codec-http2-4.1.58.Final.10b03e6, netty-codec-memcache=netty-codec-memcache-4.1.58.Final.10b03e6, netty-codec-mqtt=netty-codec-mqtt-4.1.58.Final.10b03e6, netty-codec-redis=netty-codec-redis-4.1.58.Final.10b03e6, netty-codec-smtp=netty-codec-smtp-4.1.58.Final.10b03e6, netty-codec-socks=netty-codec-socks-4.1.58.Final.10b03e6, netty-codec-stomp=netty-codec-stomp-4.1.58.Final.10b03e6, netty-codec-xml=netty-codec-xml-4.1.58.Final.10b03e6, netty-common=netty-common-4.1.58.Final.10b03e6, netty-handler=netty-handler-4.1.58.Final.10b03e6, netty-handler-proxy=netty-handler-proxy-4.1.58.Final.10b03e6, netty-resolver=netty-resolver-4.1.58.Final.10b03e6, netty-resolver-dns=netty-resolver-dns-4.1.58.Final.10b03e6, netty-resolver-dns-native-macos=netty-resolver-dns-native-macos-4.1.58.Final.10b03e65f1, netty-transport=netty-transport-4.1.58.Final.10b03e6, netty-transport-native-epoll=netty-transport-native-epoll-4.1.58.Final.10b03e6, netty-transport-native-kqueue=netty-transport-native-kqueue-4.1.58.Final.10b03e65f1, netty-transport-native-unix-common=netty-transport-native-unix-common-4.1.58.Final.10b03e6, netty-transport-rxtx=netty-transport-rxtx-4.1.58.Final.10b03e6, netty-transport-sctp=netty-transport-sctp-4.1.58.Final.10b03e6, netty-transport-udt=netty-transport-udt-4.1.58.Final.10b03e6]
INFO  [main] 2022-07-06 20:51:41,074 PipelineConfigurator.java:125 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO  [main] 2022-07-06 20:51:41,084 CassandraDaemon.java:782 - Startup complete
INFO  [CompactionExecutor:2] 2022-07-06 20:51:41,193 CompactionTask.java:241 - Compacted (6fb5a2c0-fd6d-11ec-9792-3f8ce372cd42) 5 sstables to [/bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-6-big,] to level=0.  5.224KiB to 5.085KiB (~97% of original) in 211ms.  Read Throughput = 24.674KiB/s, Write Throughput = 24.019KiB/s, Row Throughput = ~2/s.  5 total partitions merged to 1.  Partition merge counts were {5:1, }
INFO  [NonPeriodicTasks:1] 2022-07-06 20:51:41,199 SSTable.java:111 - Deleting sstable: /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big
INFO  [NonPeriodicTasks:1] 2022-07-06 20:51:41,202 SSTable.java:111 - Deleting sstable: /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-2-big
INFO  [NonPeriodicTasks:1] 2022-07-06 20:51:41,205 SSTable.java:111 - Deleting sstable: /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-1-big
INFO  [NonPeriodicTasks:1] 2022-07-06 20:51:41,211 SSTable.java:111 - Deleting sstable: /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-3-big
INFO  [NonPeriodicTasks:1] 2022-07-06 20:51:41,213 SSTable.java:111 - Deleting sstable: /bitnami/cassandra/data/data/system/local-7ad54392bcdd35a684174e047860b377/nb-5-big
INFO  [OptionalTasks:1] 2022-07-06 20:51:51,027 CassandraRoleManager.java:339 - Created default superuser role 'cassandra'
snowyaaron1 commented 2 years ago

So today I decided to comment out wait_for_nodetool_up() and commented out Line 966, rebuilt the image as a local copy and it worked, I am able to cluster between the two using this new image

IP: 10.117.50.47

services:
  cassandra:
    image: test
    ports:
      - 7000:7000
      - 7001:7001
      - 9042:9042
    environment:
      #- CASSANDRA_HOST=cassandra
      - CASSANDRA_SEEDS=10.117.50.41,10.117.50.47
      - CASSANDRA_CLUSTER_NAME=cassandra-cluster
      - CASSANDRA_PASSWORD=cassandra
      - CASSANDRA_BROADCAST_ADDRESS=10.117.50.47
      - BITNAMI_DEBUG=true
      # By default, Cassandra autodetects the available host memory and takes as much as it can.
      # Therefore, memory options are mandatory if multiple Cassandras are launched in the same node.
      - MAX_HEAP_SIZE=512M
      - HEAP_NEWSIZE=256M
marcosbc commented 2 years ago

Hi @snowyaaron1, I'm not sure that disabling the wait_for_nodetool_up function is a good idea. Have you considered entering into one of those pods and executing the nodetool status command for yourself? That might reveal your issue.

You can also change the following line in libcassandra.sh to reveal what it is checking against:

@@ -840,7 +840,9 @@ wait_for_nodetool_up() {
         if [[ "$BITNAMI_DEBUG" = "true" ]]; then
             output="/dev/stdout"
         fi
+        set -x
         "${check_cmd[@]}" "${check_args[@]}" | grep -E "${check_regex}" >"${output}"
+        set +x
     }

     if retry_while check_function_nodetool "$retries" "$sleep_time"; then
snowyaaron1 commented 2 years ago

After running nodetool status it shows that neither are connecting to each other. The Bitnami images continues to fail (exits container) on wait_for_nodetool_up with no error message. I'm curious why if I switch the image to the Apache Cassandra one with the exact same docker-compose above it works perfectly (they connect to one another). Or, if I take out wait_for_nodetool_up it also works.

rafariossaa commented 2 years ago

Hi, Have you tried to enable BITNAMI_DEBUG ? or running the check command manually ? Maybe that gives a clue of what is happening.

snowyaaron1 commented 2 years ago

Bitnami crashes on this error:

cassandra 20:52:18.01 ERROR ==> Nodetool output cassandra 20:52:18.00 ERROR ==> Cassandra failed to start up

rafariossaa commented 2 years ago

Hi, Have you tried creating a network and using the same network in both docker-compose ? My guess is that this issue is more related to a misconfiguration of cassandra/docker than a issue in the image itself.

snowyaaron1 commented 2 years ago

Well i thought the same thing, These are two separate instances over AWS. If i change the image to Apache Cassandra it works perfectly. It's something bitnami is doing

carrodher commented 2 years ago

We are going to transfer this issue to bitnami/containers

In order to unify the approaches followed in Bitnami containers and Bitnami charts, we are moving some issues in bitnami/bitnami-docker-<container> repositories to bitnami/containers.

Please follow bitnami/containers to keep you updated about the latest bitnami images.

More information here: https://blog.bitnami.com/2022/07/new-source-of-truth-bitnami-containers.html

rafariossaa commented 2 years ago

Sorry for the delay. Could you check the listening IP in the image that works for you ? I guess both images generates a very similar configuration.

github-actions[bot] commented 2 years ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 2 years ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.