Nutties93 commented 9 months ago

INFO [main] 2023-12-18 10:00:10,554 IPCController.java:111 - Starting Server INFO [main] 2023-12-18 10:00:10,560 IPCController.java:121 - Started Server Started service on file:///tmp/oss-mgmt.sock INFO [nioEventLoopGroup-3-1] 2023-12-18 10:00:19,268 LifecycleResources.java:208 - Started Cassandra INFO [nioEventLoopGroup-2-2] 2023-12-18 10:00:19,297 Cli.java:642 - address=/172.25.1.216:40284 url=/api/v0/lifecycle/start status=201 Created INFO [nioEventLoopGroup-2-1] 2023-12-18 10:00:28,816 Cli.java:642 - address=/172.25.1.139:41604 url=/api/v0/probes/liveness status=200 OK INFO [nioEventLoopGroup-2-2] 2023-12-18 10:00:28,829 Cli.java:642 - address=/172.25.1.139:41618 url=/api/v0/probes/readiness status=500 Internal Server Error INFO [nioEventLoopGroup-2-1] 2023-12-18 10:00:38,803 Cli.java:642 - address=/172.25.1.139:40786 url=/api/v0/probes/readiness status=500 Internal Server Error

I am getting the above error when i try to install single-dc-reaper on my EKS cluster 12.7. I've also tried other setup as well like medusa or stargate. Somehow, it is not runnning proper with EKS cluster.

What happened?

Did you expect to see something different?

How to reproduce it (as minimally and precisely as possible):

Environment

K8ssandra Operator version:

Insert image tag or Git SHA here
Kubernetes version information:

kubectl version
Kubernetes cluster kind:

insert how you created your cluster: kops, bootkube, etc.
Manifests:

insert manifests relevant to the issue

K8ssandra Operator Logs:

insert K8ssandra Operator logs relevant to the issue here

No issues w ith k8ssandra operator control plane.

Anything else we need to know?:

┆Issue is synchronized with this Jira Story by Unito

adejanovski commented 9 months ago

Hi,

we'd need more information to investigate this ticket. Most of them are part of the issue template which you haven't filled in: K8ssandraCluster manifest, logs from the operator container.

My understanding is that you're using the test fixture from the single-dc-reaper e2e test, right? These fixtures use host networking which should be disabled in a cloud managed environment, as mentioned in the note here. Did you disable host networking before applying the manifest?

Nutties93 commented 9 months ago

@adejanovski

Manifest file for single-dc-reaper is this

apiVersion: k8ssandra.io/v1alpha1 kind: K8ssandraCluster metadata: name: test spec: reaper: httpManagement: enabled: true heapSize: 256Mi autoScheduling: enabled: false telemetry: vector: enabled: true resources: requests: cpu: 100m memory: 128Mi limits: cpu: 100m memory: 512Mi cassandra: networking: hostNetwork: false serverVersion: "3.11.14" jmxInitContainerImage: repository: library name: busybox datacenters:

metadata: name: dc1

k8sContext: kind-k8ssandra-0

size: 2
storageConfig:
  cassandraDataVolumeClaimSpec:
    storageClassName: ebs-sc
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 5Gi
config:
  jvmOptions:
    heapSize: 384Mi

mgmtAPIHeap: 64Mi

These are the pod logs

Normal SuccessfulAttachVolume 6m10s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c7255072-84b6-4c44-9023-99643089359a" Normal Pulled 6m8s kubelet Container image "datastax/cass-config-builder:1.0-ubi7" already present on machine Normal Created 6m8s kubelet Created container server-config-init Normal Started 6m8s kubelet Started container server-config-init Normal Pulled 6m1s kubelet Container image "k8ssandra/cass-management-api:3.11.14" already present on machine Normal Created 6m1s kubelet Created container cassandra Normal Started 6m1s kubelet Started container cassandra Normal Pulled 6m1s kubelet Container image "k8ssandra/system-logger:v1.18.2" already present on machine Normal Created 6m kubelet Created container server-system-logger Normal Started 6m kubelet Started container server-system-logger Warning Unhealthy 67s (x32 over 5m39s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500

Operator logs are as follows

2023-12-18T15:12:51.110Z 2023-12-18T15:12:51.110Z 2023-12-18T15:12:51.110Z 2023-12-18T15:12:51.118Z 2023-12-18T15:12:51.118Z 2023-12-18T15:12:51.142Z 2023-12-18T15:12:51.330Z 2023-12-18T15:12:51.330Z 2023-12-18T15:12:51.330Z 2023-12-18T15:12:51.330Z 2023-12-18T15:12:51.330Z 2023-12-18T15:12:51.331Z 2023-12-18T15:12:51.331Z 2023-12-18T15:12:51.331Z 2023-12-18T15:12:51.331Z 2023-12-18T15:12:51.331Z 2023-12-18T15:12:51.334Z 2023-12-18T15:12:51.335Z 2023-12-18T15:12:51.391Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.939Z 2023-12-18T15:13:05.943Z 2023-12-18T15:13:05.944Z 2023-12-18T15:13:05.967Z 2023-12-18T15:15:15.737Z 2023-12-18T15:15:15.737Z 2023-12-18T15:15:15.737Z 2023-12-18T15:15:15.737Z 2023-12-18T15:15:15.737Z 2023-12-18T15:15:15.738Z 2023-12-18T15:15:15.738Z 2023-12-18T15:15:15.739Z 2023-12-18T15:15:15.739Z 2023-12-18T15:15:15.739Z 2023-12-18T15:15:15.744Z 2023-12-18T15:15:15.744Z 2023-12-18T15:15:15.757Z INFO Medusa reconcile for dc1 on namespace k8ssandra-operator {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "e7427e09-178a-4ff9-a51c-65dc479cbe12", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Medusa is not enabled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "e7427e09-178a-4ff9-a51c-65dc479cbe12", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Vector Agent ConfigMap successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "e7427e09-178a-4ff9-a51c-65dc479cbe12", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Reconciling seeds {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "e7427e09-178a-4ff9-a51c-65dc479cbe12", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Waiting for datacenter to satisfy Ready condition {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "e7427e09-178a-4ff9-a51c-65dc479cbe12", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO updated k8ssandracluster status {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "e7427e09-178a-4ff9-a51c-65dc479cbe12", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling Reaper user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reaper user secrets successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling Medusa user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Medusa user secrets successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling replicated secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test"} INFO new metrics endpoint is available, so MCAC can be disabled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test", "serverVersion": "3.11.14"} INFO Initial token computation could not be performed or is not required in this cluster {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test", "error": "cannot compute initial tokens: at least one DC has num_tokens >= 16"} INFO Medusa reconcile for dc1 on namespace k8ssandra-operator {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Medusa is not enabled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Vector Agent ConfigMap successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Reconciling seeds {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Waiting for datacenter to satisfy Ready condition {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO updated k8ssandracluster status {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "b820f54d-580d-45c7-a3e0-0c3b015088c1", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling Reaper user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reaper user secrets successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling Medusa user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Medusa user secrets successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling replicated secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test"} INFO new metrics endpoint is available, so MCAC can be disabled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test", "serverVersion": "3.11.14"} INFO Initial token computation could not be performed or is not required in this cluster {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test", "error": "cannot compute initial tokens: at least one DC has num_tokens >= 16"} INFO Medusa reconcile for dc1 on namespace k8ssandra-operator {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Medusa is not enabled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Vector Agent ConfigMap successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Reconciling seeds {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Waiting for datacenter to satisfy Ready condition {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO updated k8ssandracluster status {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "9f63b102-c04f-4268-995e-bcf55c2d7e11", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling Reaper user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reaper user secrets successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling Medusa user secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Medusa user secrets successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test"} INFO Reconciling replicated secrets {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test"} INFO new metrics endpoint is available, so MCAC can be disabled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test", "serverVersion": "3.11.14"} INFO Initial token computation could not be performed or is not required in this cluster {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test", "error": "cannot compute initial tokens: at least one DC has num_tokens >= 16"} INFO Medusa reconcile for dc1 on namespace k8ssandra-operator {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Medusa is not enabled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Vector Agent ConfigMap successfully reconciled {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Reconciling seeds {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO Waiting for datacenter to satisfy Ready condition {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "test", "reconcileID": "6f3ebae0-b6fb-4850-b8d3-22c5661e841e", "K8ssandraCluster": "k8ssandra-operator/test", "CassandraDatacenter": "k8ssandra-operator/dc1", "K8SContext": ""} INFO updated k8ssandracluster status {"controller": "k8ssandracluster", "controllerGroup": "k8ssandra.io", "controllerKind": "K8ssandraCluster", "K8ssandraCluster": {"name":"test","namespace":"k8ssandra-oper}

datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::CheckInternodeCredentialCreation {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO starting CheckRackForceUpgrade() {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::CheckRackScale {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::CheckPodsReady {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::findStartedNotReadyNodes {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::deleteStuckNodes {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::CheckSeedLabels {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::refreshSeeds {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO reconcile_racks::findStartingNodes {"controller": "cassandradatacenter_controller", "controllerGroup": "cassandra.datastax.com", "controllerKind": "CassandraDatacenter", "CassandraDatacenter": {"name":"dc1","namespace":"k8ssandra-operator"}, "namespace": "k8ssandra-operator", "name": "dc1", "reconcileID": "6aad1493-a9f3-4ada-953b-32dcf453f931", "namespace": "k8ssandra-operator", "datacenterName": "dc1", "clusterName": "test"} 2023-12-18T15:24:30.435Z INFO controllers.CassandraDatacenter Reconcile loop completed {"cassandradatacenter": "k8ssandra-operator/dc1", "requestNamespace": "k8ssandra-operator", "requestName": "dc1", "loopID": "85341ea0-e40d-41ee-8729-909fd210e409", "duration": 0.001077975}

Are there any recommended manifest for EKS setup?

k8ssandra / k8ssandra-operator

Trying to install k8ssandra single-dc-reaper on EKS cluster 1.27. #1140

k8sContext: kind-k8ssandra-0