search

k8ssandra / k8ssandra-operator

The Kubernetes operator for K8ssandra
https://k8ssandra.io/
Apache License 2.0
174 stars 79 forks source link

Reaper Error: Error getting pending compactions attribute from JMX #1412

Open asgeek opened 2 months ago

asgeek commented 2 months ago

What happened?

Reaper is continuously showing this error, but it doesn’t cause the pod to restart. Both the Cassandra Cluster and Reaper are functioning as expected.

Did you expect to see something different?

Get rid of the exception from the Reaper logs.

How to reproduce it (as minimally and precisely as possible):

Environment

apiVersion: k8ssandra.io/v1alpha1
kind: K8ssandraCluster
metadata:
  name: my-cluster-2
  namespace: k8ssandra-operator
spec:
  cassandra:
    serverVersion: "4.0.0"
    softPodAntiAffinity: true
    storageConfig:
      cassandraDataVolumeClaimSpec:
        storageClassName: ceph-rbd
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
    config:
      jvmOptions:
        heapSize: 512M
      cassandraYaml:
        read_request_timeout_in_ms: 10000
        range_request_timeout_in_ms: 15000
        write_request_timeout_in_ms: 10000
    networking:
      hostNetwork: false
    resources:
      requests:
        cpu: "2"
        memory: 8Gi
      limits:
        cpu: "2"
        memory: 32Gi
    datacenters:
    - metadata:
        name: e1
      k8sContext: e1
      size: 3
    - metadata:
        name: e2
      k8sContext: e2
      size: 3
  reaper:
    containerImage:
      tag: "3.6.1"
    # tried to fix the error disabling telemetry, but it didn't help
    telemetry:
      mcac:
        enabled: false
      prometheus: 
        enabled: false
      vector:
        enabled: false
  stargate:
    containerImage:
      registry: docker-proxy.domain.com
      tag: v1.0.77
    heapSize: 384Mi
    resources:
      requests:
        cpu: "1"
        memory: 1Gi
      limits:
        cpu: "2"
        memory: 4Gi

ERROR  [2024-09-18 07:46:35,566] [ForkJoinPool-1-worker-41] i.c.m.ICassandraManagementProxy - Error getting pending compactions attribute from JMX
javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=Compaction,name=PendingTasks
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:643)
    at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
    at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)
    at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
    at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
    at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
    at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)
    at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
    at sun.rmi.transport.Transport$1.run(Transport.java:200)
    at sun.rmi.transport.Transport$1.run(Transport.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
    at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
    at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
    at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164)
    at jdk.remoteref/jdk.jmx.remote.internal.rmi.PRef.invoke(Unknown Source)
    at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
    at java.management.rmi/javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:904)
    at io.cassandrareaper.management.jmx.JmxCassandraManagementProxy.getPendingCompactions(JmxCassandraManagementProxy.java:426)
    at io.cassandrareaper.management.CompactionProxy.getPendingCompactions(CompactionProxy.java:97)
    at io.cassandrareaper.management.ClusterFacade.listCompactionStatsDirect(ClusterFacade.java:575)
    at io.cassandrareaper.service.MetricsService.grabAndStoreCompactionStats(MetricsService.java:124)
    at io.cassandrareaper.service.Heart.updateMetricsForNode(Heart.java:275)
    at io.cassandrareaper.service.Heart.lambda$updateMetricsForClusters$6(Heart.java:222)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)

Anything else we need to know?:

┆Issue is synchronized with this Jira Story by Unito ┆Issue Number: K8OP-257