NoSpamLogger.java - Maximum memory usage reached cannot allocate chunk

[KEDLogic]

We have a k8ssandra cluster in production, 3 racks, each with 2 nodes. Using AWS EKS, m5.2xlarge instances.

k8ssandra given 30GB

Noted issue, spamming logs:

NoSpamLogger.java - Maximum memory usage reached cannot allocate chunk

Increased file_cache_size_in_mb from 1024 to 2048

Log quantity reduced, but still present.

Environment

• K8ssandra Operator version: v1.10.2
• Cassandra version 3.11.15

AWS EKS v 1.25

• Manifests: cassandra yaml from k8c manifest

  
    cassandraYaml:
      ssl_storage_port: 7001
      storage_port: 7000
      batchlog_replay_throttle_in_kb: 1024
      commit_failure_policy: stop
      unlogged_batch_across_partitions_warn_threshold: 10
      commitlog_segment_size_in_mb: 32
      start_rpc: true
      credentials_validity_in_ms: 2000
      client_encryption_options:
        enabled: false
      concurrent_materialized_view_writes: 32
      inter_dc_tcp_nodelay: false
      column_index_cache_size_in_kb: 2
      streaming_socket_timeout_in_ms: 72000000
      rpc_server_type: sync
      row_cache_save_period: 0
      disk_failure_policy: stop
      native_transport_port: 9042
      enable_user_defined_functions_threads: true
      server_encryption_options:
        internode_encryption: none
      dynamic_snitch_reset_interval_in_ms: 600000
      compaction_throughput_mb_per_sec: 64
      role_manager: org.apache.cassandra.auth.CassandraRoleManager
      column_index_size_in_kb: 64
      batch_size_warn_threshold_in_kb: 64
      windows_timer_interval: 1
      compaction_large_partition_warning_threshold_mb: 100
      rpc_keepalive: true
      commitlog_total_space_in_mb: 25600
      memtable_heap_space_in_mb: 2048
      batch_size_fail_threshold_in_kb: 640
      snapshot_before_compaction: false
      tracetype_query_ttl: 86400
      concurrent_reads: 32
      key_cache_save_period: 14400
      row_cache_size_in_mb: 0
      tracetype_repair_ttl: 604800
      enable_materialized_views: true
      tombstone_warn_threshold: 1000
      rpc_address: 0.0.0.0
      concurrent_writes: 32
      commitlog_sync: periodic
      counter_cache_save_period: 7200
      file_cache_size_in_mb: 2048
      back_pressure_enabled: false
      enable_sasi_indexes: true
      slow_query_log_timeout_in_ms: 500
      trickle_fsync: true
      write_request_timeout_in_ms: 2000
      incremental_backups: true
      truncate_request_timeout_in_ms: 60000
      enable_scripted_user_defined_functions: false
      read_request_timeout_in_ms: 5000
      request_timeout_in_ms: 10000
      start_native_transport: true
      memtable_allocation_type: offheap_objects
      transparent_data_encryption_options:
        enabled: false
        chunk_length_kb: 64
        cipher: AES/CBC/PKCS5Padding
        key_alias: testing:1
      memtable_offheap_space_in_mb: 2048
      internode_compression: dc
      max_hints_delivery_threads: 2
      cross_node_timeout: false
      partitioner: org.apache.cassandra.dht.Murmur3Partitioner
      tombstone_failure_threshold: 100000
      hinted_handoff_enabled: true
      hints_flush_period_in_ms: 10000
      enable_user_defined_functions: false
      hinted_handoff_throttle_in_kb: 1024
      max_hint_window_in_ms: 10800000
      auto_snapshot: true
      index_summary_resize_interval_in_minutes: 60
      range_request_timeout_in_ms: 10000
      stream_throughput_outbound_megabits_per_sec: 200
      sstable_preemptive_open_interval_in_mb: 50
      dynamic_snitch_update_interval_in_ms: 100
      trickle_fsync_interval_in_kb: 10240
      commitlog_sync_period_in_ms: 10000
      cdc_enabled: true
      max_hints_file_size_in_mb: 128
      counter_write_request_timeout_in_ms: 5000
      concurrent_counter_writes: 32
      dynamic_snitch_badness_threshold: 0.1
      permissions_validity_in_ms: 2000
      roles_validity_in_ms: 2000
      rpc_port: 9160
      cas_contention_timeout_in_ms: 1000
      thrift_framed_transport_size_in_mb: 15
      gc_warn_threshold_in_ms: 1000
      request_scheduler: org.apache.cassandra.scheduler.NoScheduler

[adejanovski]

Hi, this is a common warning message with Cassandra and is not a k8ssandra related issue. I'd recommend to look for previous answers on that same question in the Cassandra mailing list or Jira for leads on how to configure it.