search

k8ssandra / k8ssandra-operator

The Kubernetes operator for K8ssandra
https://k8ssandra.io/
Apache License 2.0
171 stars 78 forks source link

Document the minimum resources required to run each component #779

Open tlb1galaxy opened 1 year ago

tlb1galaxy commented 1 year ago

What happened? New deployment of K8ssandracluster via Operator to include Stargate fails to startup.

How to reproduce it (as minimally and precisely as possible): brand new deployment of k8ssandra-operator

  1. kubectl create ns k8ssandra-operator
  2. helm install k8ssandra-operator k8ssandra/k8ssandra-operator --version 0.38.2 -n k8ssandra-operator a. v0.38.5 has issues, so using 0.38.2
  3. kubectl apply -n k8ssandra-operator -f 30-k8ssandra-k8ssandracluster.yaml

Environment

``` yaml

apiVersion: k8ssandra.io/v1alpha1 kind: K8ssandraCluster metadata: name: demo spec:

auth: true

cassandra: serverVersion: "4.0.7" softPodAntiAffinity: true datacenters:

racks:

- name: rack1

    size: 3
    resources:
      limits:
        cpu: "500m"
        memory: 4Gi
      requests:
        cpu: "500m"
        memory: 4Gi
    config:
      jvmOptions:
        heap_initial_size: 1G
        heap_max_size: 2G
    storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: rook-ceph-block-hdd7k
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi

stargate: size: 1 resources: limits: cpu: "250m" memory: 512Mi requests: cpu: "250m" memory: 512Mi heapSize: 256Mi allowStargateOnDataNodes: true

affinity:

podAffinity:

preferredDuringSchedulingIgnoredDuringExecution:

...


* k8ssandra pods

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES demo-dc1-default-stargate-deployment-7d788488c4-w69dh 0/1 Running 10 25m 192.168.205.5 tlb1-node-w2 demo-dc1-default-sts-0 2/2 Running 0 33m 192.168.205.13 tlb1-node-w2 demo-dc1-default-sts-1 2/2 Running 0 33m 192.168.57.17 tlb1-node-w1 demo-dc1-default-sts-2 2/2 Running 0 33m 192.168.221.25 tlb1-node-w3 k8ssandra-operator-56fb7bf949-6nx4d 1/1 Running 0 3h32m 192.168.57.30 tlb1-node-w1 k8ssandra-operator-cass-operator-87d58f57d-g594s 1/1 Running 4 3h32m 192.168.57.62 tlb1-node-w1 


* K8ssandra Operator Logs:

1.669311384718185e+09 INFO controller.k8ssandracluster Preparing to update replication for system keyspaces {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": "", "replication": {"dc1":3}} 1.6693113847182887e+09 INFO controller.k8ssandracluster Ensuring that keyspace system_traces exists in cluster demo... {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311384718559e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847225592e+09 INFO controller.k8ssandracluster keyspace system_traces already exists in cluster demo {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847227492e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847271338e+09 INFO controller.k8ssandracluster Successfully got keyspace system_traces replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311384727168e+09 INFO controller.k8ssandracluster Keyspace system_traces has desired replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847271755e+09 INFO controller.k8ssandracluster Ensuring that keyspace system_distributed exists in cluster demo... {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847273576e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847314203e+09 INFO controller.k8ssandracluster keyspace system_distributed already exists in cluster demo {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.66931138473166e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847365508e+09 INFO controller.k8ssandracluster Successfully got keyspace system_distributed replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847366087e+09 INFO controller.k8ssandracluster Keyspace system_distributed has desired replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847366242e+09 INFO controller.k8ssandracluster Ensuring that keyspace system_auth exists in cluster demo... {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311384736776e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847400286e+09 INFO controller.k8ssandracluster keyspace system_auth already exists in cluster demo {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847402036e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311384744554e+09 INFO controller.k8ssandracluster Successfully got keyspace system_auth replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113847445831e+09 INFO controller.k8ssandracluster Keyspace system_auth has desired replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311384753976e+09 DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/validate-k8ssandra-io-v1alpha1-k8ssandracluster", "UID": "3c471ae8-70ea-4080-8ab4-23a4072e0316", "kind": "k8ssandra.io/v1alpha1, Kind=K8ssandraCluster", "resource": {"group":"k8ssandra.io","version":"v1alpha1","resource":"k8ssandraclusters"}} 1.669311384754744e+09 INFO k8ssandracluster-webhook validate K8ssandraCluster update {"K8ssandraCluster": "demo"} 1.6693113847548587e+09 DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/validate-k8ssandra-io-v1alpha1-k8ssandracluster", "code": 200, "reason": "", "UID": "3c471ae8-70ea-4080-8ab4-23a4072e0316", "allowed": true} 1.6693113852773392e+09 INFO controller.k8ssandracluster Reconciling Stargate auth keyspace data_endpoint_auth {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852773778e+09 INFO controller.k8ssandracluster Ensuring that keyspace data_endpoint_auth exists in cluster demo... {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852775557e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311385281156e+09 INFO controller.k8ssandracluster keyspace data_endpoint_auth already exists in cluster demo {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852814054e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852877092e+09 INFO controller.k8ssandracluster Successfully got keyspace data_endpoint_auth replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311385287738e+09 INFO controller.k8ssandracluster Keyspace data_endpoint_auth has desired replication {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.669311385287745e+09 INFO controller.k8ssandracluster Reconciling Stargate auth table data_endpoint_auth.token {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852879176e+09 INFO controller.k8ssandracluster client::callNodeMgmtEndpoint {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852932274e+09 INFO controller.k8ssandracluster Successfully got keyspace data_endpoint_auth tables {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852932549e+09 INFO controller.k8ssandracluster Table data_endpoint_auth.token already exists {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} 1.6693113852932658e+09 INFO controller.k8ssandracluster All DCs reconciled {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo"} 1.6693113852932825e+09 INFO controller.k8ssandracluster Reconciling Stargate and Reaper for dc dc1 {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1"} 1.669311385293289e+09 INFO controller.k8ssandracluster reconciling telemetry {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1"} 1.66931138529332e+09 INFO controller.k8ssandracluster merged TelemetrySpec constructed {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "mergedSpec": null, "cluster": "demo"} I1124 17:36:27.044352 1 request.go:665] Waited for 1.748218688s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s 1.6693113870963464e+09 INFO controller.k8ssandracluster Reconcile Stargate {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "Stargate": "k8ssandra-operator/demo-dc1-stargate"} 1.6693113870965798e+09 INFO controller.k8ssandracluster Waiting for Stargate to become ready {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo", "CassandraDatacenter": "k8ssandra-operator/dc1", "Stargate": "k8ssandra-operator/demo-dc1-stargate"} 1.6693113871966133e+09 INFO controller.k8ssandracluster updated k8ssandracluster status {"reconciler group": "k8ssandra.io", "reconciler kind": "K8ssandraCluster", "name": "demo", "namespace": "k8ssandra-operator", "K8ssandraCluster": "k8ssandra-operator/demo"} 1.6693113927903256e+09 INFO controller.stargate Fetching Stargate resource {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "demo-dc1-stargate", "namespace": "k8ssandra-operator", "Stargate": {"namespace": "k8ssandra-operator", "name": "demo-dc1-stargate"}} 1.6693113927904809e+09 INFO controller.stargate Fetching CassandraDatacenter resource {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "demo-dc1-stargate", "namespace": "k8ssandra-operator", "CassandraDatacenter": {"namespace": "k8ssandra-operator", "name": "dc1"}} 1.669311392790532e+09 INFO controller.stargate Reconciling Stargate configmap {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "demo-dc1-stargate", "namespace": "k8ssandra-operator"} 1.6693113927905777e+09 INFO controller.stargate Reconciling Stargate Cassandra yaml configMap on namespace k8ssandra-operator for cluster demo and dc dc1 {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "demo-dc1-stargate", "namespace": "k8ssandra-operator"} 1.6693113927906632e+09 INFO controller.stargate Stargate ConfigMap successfully reconciled {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "demo-dc1-stargate", "namespace": "k8ssandra-operator", "StargateConfigMap": "k8ssandra-operator/demo-dc1-cassandra-config"} 1.6693113927911608e+09 INFO controller.stargate reconciling telemetry {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "demo-dc1-stargate", "namespace": "k8ssandra-operator", "stargate": "demo-dc1-stargate"} 1.6693113945947533e+09 INFO controller.stargate Waiting for deployments to be rolled out {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "demo-dc1-stargate", "namespace": "k8ssandra-operator", "Stargate": {"namespace": "k8ssandra-operator", "name": "demo-dc1-stargate"}}


* server-system-logging -> sts-0

INFO [epollEventLoopGroup-7-3] 2022-11-24 17:37:28,147 Keyspace.java:386 - Creating replication strategy none params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc1=1}} INFO [epollEventLoopGroup-7-3] 2022-11-24 17:37:28,147 NetworkTopologyStrategy.java:89 - Configured datacenter replicas are dc1:rf(1) INFO [Messaging-EventLoop-3-1] 2022-11-24 17:39:26,839 InboundConnectionInitiator.java:464 - /192.168.205.5:7000(/192.168.205.5:38640)->/192.168.205.13:7000-URGENT_MESSAGES-4347a7c8 messaging connection established, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:39:27,243 OutboundConnection.java:1150 - /192.168.205.13:7000(/192.168.205.13:55068)->/192.168.205.5:7000-URGENT_MESSAGES-cbaee9ed successfully connected, version = 12, framing = LZ4, encryption = unencrypted INFO [GossipStage:1] 2022-11-24 17:39:37,155 Gossiper.java:1364 - Node /192.168.205.5:7000 is now part of the cluster INFO [GossipStage:1] 2022-11-24 17:39:37,157 Gossiper.java:1312 - InetAddress /192.168.205.5:7000 is now UP INFO [Messaging-EventLoop-3-1] 2022-11-24 17:39:44,442 InboundConnectionInitiator.java:464 - /192.168.205.5:7000(/192.168.205.5:38770)->/192.168.205.13:7000-SMALL_MESSAGES-f7f76a0e messaging connection established, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:39:44,538 OutboundConnection.java:1150 - /192.168.205.13:7000(/192.168.205.13:55194)->/192.168.205.5:7000-SMALL_MESSAGES-88064e56 successfully connected, version = 12, framing = LZ4, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:39:44,933 InboundConnectionInitiator.java:464 - /192.168.205.5:7000(/192.168.205.5:38780)->/192.168.205.13:7000-LARGE_MESSAGES-b43f054b messaging connection established, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:39:47,751 NoSpamLogger.java:92 - /192.168.205.13:7000->/192.168.205.5:7000-URGENT_MESSAGES-[no-channel] failed to connect io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /192.168.205.5:7000 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) INFO [GossipStage:1] 2022-11-24 17:40:11,846 Gossiper.java:1328 - InetAddress /192.168.205.5:7000 is now DOWN INFO [GossipTasks:1] 2022-11-24 17:40:14,849 Gossiper.java:1057 - FatClient /192.168.205.5:7000 has been silent for 30000ms, removing from gossip INFO [Messaging-EventLoop-3-1] 2022-11-24 17:40:17,820 NoSpamLogger.java:92 - /192.168.205.13:7000->/192.168.205.5:7000-URGENT_MESSAGES-[no-channel] failed to connect io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /192.168.205.5:7000 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) WARN [epollEventLoopGroup-7-3] 2022-11-24 17:40:46,726 K8SeedProvider4x.java:58 - Seed provider couldn't lookup host demo-dc1-additional-seed-service INFO [epollEventLoopGroup-7-3] 2022-11-24 17:40:46,826 Keyspace.java:386 - Creating replication strategy none params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc1=1}} INFO [epollEventLoopGroup-7-3] 2022-11-24 17:40:46,826 NetworkTopologyStrategy.java:89 - Configured datacenter replicas are dc1:rf(1) INFO [Messaging-EventLoop-3-1] 2022-11-24 17:40:47,841 NoSpamLogger.java:92 - /192.168.205.13:7000->/192.168.205.5:7000-URGENT_MESSAGES-[no-channel] failed to connect io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /192.168.205.5:7000 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) WARN [epollEventLoopGroup-7-3] 2022-11-24 17:40:52,735 K8SeedProvider4x.java:58 - Seed provider couldn't lookup host demo-dc1-additional-seed-service INFO [epollEventLoopGroup-7-3] 2022-11-24 17:40:52,764 Keyspace.java:386 - Creating replication strategy none params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc1=1}} INFO [epollEventLoopGroup-7-3] 2022-11-24 17:40:52,765 NetworkTopologyStrategy.java:89 - Configured datacenter replicas are dc1:rf(1) INFO [Messaging-EventLoop-3-1] 2022-11-24 17:41:14,241 InboundConnectionInitiator.java:464 - /192.168.205.5:7000(/192.168.205.5:39592)->/192.168.205.13:7000-URGENT_MESSAGES-d3ff6833 messaging connection established, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:41:14,439 OutboundConnection.java:1150 - /192.168.205.13:7000(/192.168.205.13:56006)->/192.168.205.5:7000-URGENT_MESSAGES-27f98657 successfully connected, version = 12, framing = LZ4, encryption = unencrypted INFO [GossipStage:1] 2022-11-24 17:41:16,232 Gossiper.java:1364 - Node /192.168.205.5:7000 is now part of the cluster INFO [GossipStage:1] 2022-11-24 17:41:16,234 Gossiper.java:1312 - InetAddress /192.168.205.5:7000 is now UP INFO [Messaging-EventLoop-3-1] 2022-11-24 17:41:21,938 InboundConnectionInitiator.java:464 - /192.168.205.5:7000(/192.168.205.5:39668)->/192.168.205.13:7000-SMALL_MESSAGES-0c600d1b messaging connection established, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:41:22,240 OutboundConnection.java:1150 - /192.168.205.13:7000(/192.168.205.13:56096)->/192.168.205.5:7000-SMALL_MESSAGES-96fd42cc successfully connected, version = 12, framing = LZ4, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:41:22,735 InboundConnectionInitiator.java:464 - /192.168.205.5:7000(/192.168.205.5:39684)->/192.168.205.13:7000-LARGE_MESSAGES-82bb03b4 messaging connection established, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:41:25,136 NoSpamLogger.java:92 - /192.168.205.13:7000->/192.168.205.5:7000-URGENT_MESSAGES-[no-channel] failed to connect io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /192.168.205.5:7000 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) at io.netty.channel.unix.Socket.finishConnect(Socket.java:251) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:673) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:650) at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:530) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source) INFO [GossipStage:1] 2022-11-24 17:41:37,143 Gossiper.java:1328 - InetAddress /192.168.205.5:7000 is now DOWN


* Stargate logs

INFO [MigrationStage:1] 2022-11-24 17:47:31,037 NetworkTopologyStrategy.java:88 - Configured datacenter replicas are dc1:rf(3) INFO [MigrationStage:1] 2022-11-24 17:47:31,038 ColumnFamilyStore.java:385 - Initializing system_traces.events INFO [MigrationStage:1] 2022-11-24 17:47:31,434 ColumnFamilyStore.java:385 - Initializing system_traces.sessions INFO [MigrationStage:1] 2022-11-24 17:47:31,741 Keyspace.java:386 - Creating replication strategy system_auth params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.NetworkTopologyStrategy, dc1=3}} INFO [MigrationStage:1] 2022-11-24 17:47:31,741 NetworkTopologyStrategy.java:88 - Configured datacenter replicas are dc1:rf(3) INFO [main] 2022-11-24 17:47:34,342 Gossiper.java:2206 - No gossip backlog; proceeding INFO [main] 2022-11-24 17:47:34,354 StartupClusterConnectivityChecker.java:111 - Blocking coordination until only a single peer is DOWN in the local datacenter, timeout=10s INFO [Messaging-EventLoop-3-1] 2022-11-24 17:47:34,536 OutboundConnection.java:1150 - /192.168.205.5:7000(/192.168.205.5:42720)->demo-seed-service.k8ssandra-operator.svc.cluster.local/192.168.205.13:7000-LARGE_MESSAGES-7b8c5bd9 successfully connected, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:47:34,539 OutboundConnection.java:1150 - /192.168.205.5:7000(/192.168.205.5:42276)->demo-seed-service.k8ssandra-operator.svc.cluster.local/192.168.57.17:7000-LARGE_MESSAGES-8b6c39f7 successfully connected, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:47:34,633 OutboundConnection.java:1150 - /192.168.205.5:7000(/192.168.205.5:37600)->demo-seed-service.k8ssandra-operator.svc.cluster.local/192.168.221.25:7000-LARGE_MESSAGES-f45d43ee successfully connected, version = 12, framing = CRC, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:47:34,734 InboundConnectionInitiator.java:464 - /192.168.221.25:7000(/192.168.221.25:45236)->/192.168.205.5:7000-LARGE_MESSAGES-10fb42be messaging connection established, version = 12, framing = LZ4, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:47:34,735 InboundConnectionInitiator.java:464 - /192.168.205.13:7000(/192.168.205.13:59148)->/192.168.205.5:7000-LARGE_MESSAGES-34d65030 messaging connection established, version = 12, framing = LZ4, encryption = unencrypted INFO [Messaging-EventLoop-3-1] 2022-11-24 17:47:34,735 InboundConnectionInitiator.java:464 - /192.168.57.17:7000(/192.168.57.17:38906)->/192.168.205.5:7000-LARGE_MESSAGES-e2883410 messaging connection established, version = 12, framing = LZ4, encryption = unencrypted INFO [main] 2022-11-24 17:47:34,833 StartupClusterConnectivityChecker.java:166 - Ensured sufficient healthy connections with [dc1] after 475 milliseconds INFO [main] 2022-11-24 17:47:34,833 CassandraDaemon.java:695 - Not starting native transport as requested. Use JMX (StorageService->startNativeTransport()) or nodetool (enablebinary) to start it INFO [main] 2022-11-24 17:47:35,038 Keyspace.java:386 - Creating replication strategy stargate_system params KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.LocalStrategy}} INFO [main] 2022-11-24 17:47:35,039 ColumnFamilyStore.java:385 - Initializing stargate_system.local INFO [main] 2022-11-24 17:47:35,538 ColumnFamilyStore.java:385 - Initializing stargate_system.peers INFO [main] 2022-11-24 17:47:36,136 ColumnFamilyStore.java:385 - Initializing stargate_system.peers_v2 INFO [main] 2022-11-24 17:47:37,634 BaseActivator.java:178 - Registering persistence-cassandra-4.0 as io.stargate.db.Persistence INFO [main] 2022-11-24 17:47:37,635 BaseActivator.java:185 - Started persistence-cassandra-4.0 INFO [main] 2022-11-24 17:47:37,636 BaseActivator.java:178 - Registering core services as io.stargate.core.metrics.api.MetricsScraper INFO [main] 2022-11-24 17:47:37,636 BaseActivator.java:178 - Registering core services as com.codahale.metrics.health.HealthCheckRegistry INFO [main] 2022-11-24 17:47:37,637 BaseActivator.java:178 - Registering core services as io.stargate.core.metrics.api.HttpMetricsTagProvider INFO [main] 2022-11-24 17:47:37,637 BaseActivator.java:185 - Started core services Starting bundle io.stargate.cql INFO [main] 2022-11-24 17:47:39,437 BaseActivator.java:97 - Starting CQL ... Starting bundle io.stargate.graphql INFO [main] 2022-11-24 17:47:40,833 BaseActivator.java:97 - Starting GraphQL ... Starting bundle io.stargate.grpc INFO [main] 2022-11-24 17:47:41,537 BaseActivator.java:97 - Starting gRPC ... Starting bundle io.stargate.health INFO [main] 2022-11-24 17:47:42,446 BaseActivator.java:97 - Starting healthchecker ... Starting bundle org.apache.felix.scr Starting bundle null Starting bundle null Starting bundle osgi.cmpn Starting bundle org.osgi.util.function Starting bundle org.osgi.util.promise Starting bundle io.stargate.db INFO [main] 2022-11-24 17:47:42,638 BaseActivator.java:97 - Starting DB services ... INFO [main] 2022-11-24 17:47:42,641 BaseActivator.java:178 - Registering DB services as io.stargate.db.Persistence INFO [main] 2022-11-24 17:47:42,642 BaseActivator.java:178 - Registering DB services as io.stargate.db.datastore.DataStoreFactory INFO [main] 2022-11-24 17:47:42,643 HealthCheckerActivator.java:50 - Starting healthchecker.... INFO [Service Thread] 2022-11-24 17:47:44,454 GCInspector.java:299 - MarkSweepCompact GC in 416ms. Eden Space: 1024208 -> 0; Survivor Space: 8912888 -> 0; Tenured Gen: 81992056 -> 39566024 INFO [main] 2022-11-24 17:47:44,843 Version.java:21 - HV000001: Hibernate Validator null INFO [main] 2022-11-24 17:47:46,034 FileUtil.java:249 - No oshi.properties file found from ClassLoader sun.misc.Launcher$AppClassLoader70dea4e INFO [main] 2022-11-24 17:47:46,034 FileUtil.java:249 - No oshi.properties file found from ClassLoader sun.misc.Launcher$AppClassLoader70dea4e INFO [main] 2022-11-24 17:47:51,542 Log.java:170 - Logging initialized 70908ms to org.eclipse.jetty.util.log.Slf4jLog INFO [main] 2022-11-24 17:47:52,136 SimpleServerFactory.java:142 - Registering jersey handler with root path prefix: / INFO [main] 2022-11-24 17:47:52,138 SimpleServerFactory.java:145 - Registering admin handler with root path prefix: /admin INFO [main] 2022-11-24 17:47:52,150 SimpleServerFactory.java:142 - Registering jersey handler with root path prefix: / INFO [main] 2022-11-24 17:47:52,150 SimpleServerFactory.java:145 - Registering admin handler with root path prefix: /admin INFO [main] 2022-11-24 17:47:52,150 AbstractServerFactory.java:717 - Starting Server INFO [main] 2022-11-24 17:47:52,939 SetUIDListener.java:217 - Opened Server@5ba603a3{HTTP/1.1, (http/1.1)}{0.0.0.0:8084} INFO [main] 2022-11-24 17:47:52,941 Server.java:375 - jetty-9.4.41.v20210516; built: 2021-05-16T23:56:28.993Z; git: 98607f93c7833e7dc59489b13f3cb0a114fb9f4c; jvm 1.8.0_292-b10 INFO [main] 2022-11-24 17:47:53,132 AdminEnvironment.java:74 - tasks =

POST    /tasks/log-level (io.dropwizard.servlets.tasks.LogConfigurationTask)
POST    /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)

INFO [main] 2022-11-24 17:47:53,138 ContextHandler.java:916 - Started i.d.j.MutableServletContextHandler@5dd44130{/admin,null,AVAILABLE} INFO [main] 2022-11-24 17:47:57,136 DropwizardResourceConfig.java:287 - The following paths were found for the configured resources:

GET     /checker/liveness (io.stargate.health.CheckerResource)
GET     /checker/readiness (io.stargate.health.CheckerResource)
GET     /metrics (io.stargate.health.PrometheusResource)

INFO [main] 2022-11-24 17:47:57,138 ContextHandler.java:916 - Started i.d.j.MutableServletContextHandler@7b555abb{/,null,AVAILABLE} INFO [main] 2022-11-24 17:47:57,238 AbstractConnector.java:331 - Started Server@5ba603a3{HTTP/1.1, (http/1.1)}{0.0.0.0:8084} INFO [main] 2022-11-24 17:47:57,239 Server.java:415 - Started 76606ms INFO [main] 2022-11-24 17:47:57,240 HealthCheckerActivator.java:69 - Started healthchecker.... INFO [main] 2022-11-24 17:47:57,240 BaseActivator.java:185 - Started healthchecker INFO [main] 2022-11-24 17:47:57,340 AuthnTableBasedService.java:78 - Initializing keyspace data_endpoint_auth and table token for table based auth INFO [main] 2022-11-24 17:47:58,233 BaseActivator.java:178 - Registering authnTableBasedService and authzTableBasedServie as io.stargate.auth.AuthenticationService WARN [dw-99 - GET /checker/liveness] 2022-11-24 17:47:59,435 BundleStateChecker.java:48 - Inactive bundles: [io.stargate.db, io.stargate.db.limiter.global, io.stargate.web] INFO [StorageServiceShutdownHook] 2022-11-24 17:48:00,132 HintsService.java:220 - Paused hints dispatch WARN [StorageServiceShutdownHook] 2022-11-24 17:48:00,137 Gossiper.java:1993 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown INFO [StorageServiceShutdownHook] 2022-11-24 17:48:00,138 MessagingService.java:441 - Waiting for messaging service to quiesce WARN [dw-100 - GET /checker/readiness] 2022-11-24 17:48:00,138 BundleStateChecker.java:48 - Inactive bundles: [io.stargate.db, io.stargate.db.limiter.global, io.stargate.web] WARN [dw-100 - GET /checker/readiness] 2022-11-24 17:48:00,233 CheckerResource.java:88 - Failed health check: bundles WARN [dw-100 - GET /checker/readiness] 2022-11-24 17:48:00,233 CheckerResource.java:88 - Failed health check: grpc WARN [dw-100 - GET /checker/readiness] 2022-11-24 17:48:00,233 CheckerResource.java:94 - Missing required health checks: [graphql, restapi, cql] INFO [JettyShutdownThread] 2022-11-24 17:48:00,538 AbstractConnector.java:381 - Stopped Server@5ba603a3{HTTP/1.1, (http/1.1)}{0.0.0.0:8084} INFO [dw-99] 2022-11-24 17:48:00,634 Slf4jRequestLogWriter.java:62 - 10.253.50.16 - - [24/Nov/2022:17:47:58 +0000] "GET /checker/liveness HTTP/1.1" 503 4 "-" "kube-probe/1.21" 2296 INFO [dw-100] 2022-11-24 17:48:00,634 Slf4jRequestLogWriter.java:62 - 10.253.50.16 - - [24/Nov/2022:17:47:58 +0000] "GET /checker/readiness HTTP/1.1" 503 9 "-" "kube-probe/1.21" 2296 INFO [JettyShutdownThread] 2022-11-24 17:48:00,840 ContextHandler.java:1154 - Stopped i.d.j.MutableServletContextHandler@7b555abb{/,null,STOPPED} INFO [JettyShutdownThread] 2022-11-24 17:48:00,933 ContextHandler.java:1154 - Stopped i.d.j.MutableServletContextHandler@5dd44130{/admin,null,STOPPED} INFO [main] 2022-11-24 17:48:01,456 BaseActivator.java:185 - Started gRPC INFO [main] 2022-11-24 17:48:01,532 BaseActivator.java:164 - The persistence-cassandra-4.0 is already started. Ignoring the start request. INFO [StorageServiceShutdownHook] 2022-11-24 17:48:02,546 ColumnFamilyStore.java:878 - Enqueuing flush of local: 40.037KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:1] 2022-11-24 17:48:02,635 Memtable.java:469 - Writing Memtable-local@12264324(9.268KiB serialized bytes, 1 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)] INFO [PerDiskMemtableFlushWriter_0:1] 2022-11-24 17:48:02,732 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.01716269871846557906/data/stargate_system/local-bf0dc51a28c73c8d9b9a5b70e206e04d/nb-1-big-Data.db (5.532KiB) for commitlog position CommitLogPosition(segmentId=1669312020041, position=61409) INFO [StorageServiceShutdownHook] 2022-11-24 17:48:02,934 ColumnFamilyStore.java:878 - Enqueuing flush of peers_v2: 12.523KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [StorageServiceShutdownHook] 2022-11-24 17:48:02,935 ColumnFamilyStore.java:878 - Enqueuing flush of peers: 11.824KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [StorageServiceShutdownHook] 2022-11-24 17:48:02,935 ColumnFamilyStore.java:878 - Enqueuing flush of local: 0.501KiB (0%) on-heap, 0.000KiB (0%) off-heap INFO [PerDiskMemtableFlushWriter_0:2] 2022-11-24 17:48:02,940 Memtable.java:469 - Writing Memtable-peers_v2@1568217655(2.280KiB serialized bytes, 40 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)] INFO [PerDiskMemtableFlushWriter_0:2] 2022-11-24 17:48:02,941 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.01716269871846557906/data/system/peers_v2-c4325fbb8e5e3bafbd070f9250ed818e/nb-1-big-Data.db (1.486KiB) for commitlog position CommitLogPosition(segmentId=1669312020041, position=61409) INFO [PerDiskMemtableFlushWriter_0:1] 2022-11-24 17:48:02,942 Memtable.java:469 - Writing Memtable-peers@819013565(2.210KiB serialized bytes, 40 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)] INFO [PerDiskMemtableFlushWriter_0:1] 2022-11-24 17:48:03,032 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.01716269871846557906/data/system/peers-37f71aca7dc2383ba70672528af04d4f/nb-1-big-Data.db (1.457KiB) for commitlog position CommitLogPosition(segmentId=1669312020041, position=61409) INFO [PerDiskMemtableFlushWriter_0:2] 2022-11-24 17:48:03,336 Memtable.java:469 - Writing Memtable-local@710275964(0.060KiB serialized bytes, 3 ops, 0%/0% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)] INFO [PerDiskMemtableFlushWriter_0:2] 2022-11-24 17:48:03,336 Memtable.java:498 - Completed flushing /tmp/stargate-persistence-cassandra-4.01716269871846557906/data/system/local-7ad54392bcdd35a684174e047860b377/nb-4-big-Data.db (0.043KiB) for commitlog position CommitLogPosition(segmentId=1669312020041, position=61409) INFO [StorageServiceShutdownHook] 2022-11-24 17:48:03,438 HintsService.java:220 - Paused hints dispatch INFO [main] 2022-11-24 17:48:03,537 Version.java:21 - HV000001: Hibernate Validator null



┆Issue is synchronized with this [Jira Story](https://datastax.jira.com/browse/K8OP-222) by [Unito](https://www.unito.io)
┆Issue Number: K8OP-222
tlb1galaxy commented 1 year ago

So this issue seems to stem from setting resource limits below what the pod/container actually works. Figured this out with trial and error. Once I set the limits higher, the pod was able to function. I have noticed the same issue with other components within the stack (Reaper, Cassandra, Medusa)

Suggestion: Document the minimum resources required to run each component (pod/container). I know every situation can be covered but at least a baseline for people starting with the stack. The JVM heap size has a documented default, which is helpful.

adejanovski commented 1 year ago

Thanks @tlb1galaxy, I've updated the issue title to match your findings.