search

instaclustr / cassandra-exporter

Java agent for exporting Cassandra metrics to Prometheus
Apache License 2.0
73 stars 46 forks source link

Unable to compute when histogram overflowed #104

Open tomushu opened 2 years ago

tomushu commented 2 years ago

Hi, I'm using cassandra-exporter-standalone-0.9.10.jar with cassandra 3.11.10 and suddenly the exporter for some of the nodes in one DC and two different clusters, started crashing with the errors below. What is strange is, using the same jar from a different machine and connecting to one of the troubled cassandra nodes, the exporter doesn't crash. 16:52:47.688 [main] INFO com.datastax.driver.core.GuavaCompatibility - Detected Guava < 19 in the classpath, using legacy compatibility layer 16:52:47.827 [main] INFO com.datastax.driver.core.ClockFactory - Using native clock to generate timestamps. 16:52:47.910 [main] INFO com.datastax.driver.core.NettyUtil - Found Netty's native epoll transport in the classpath, using it 16:52:48.374 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /192.168.10.1:9042 added 16:52:48.375 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /192.168.10.2:9042 added 16:52:48.375 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /192.168.11.1:9042 added 16:52:48.375 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /192.168.11.2:9042 added 16:52:48.569 [main] INFO com.zegelin.cassandra.exporter.netty.Server - cassandra-exporter server has started. Listening on http://0:0:0:0:0:0:0:0:9500. 16:52:59.996 [prometheus-netty-pool-0] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. java.lang.IllegalStateException: Unable to compute when histogram overflowed at org.apache.cassandra.metrics.DecayingEstimatedHistogramReservoir$EstimatedHistogramReservoirSnapshot.getValue(DecayingEstimatedHistogramReservoir.java:376) at com.codahale.metrics.Snapshot.getMedian(Snapshot.java:38) at org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxTimer.get50thPercentile(CassandraMetricsRegistry.java:505) at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401) at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639) at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) at sun.rmi.transport.Transport$1.run(Transport.java:200) at sun.rmi.transport.Transport$1.run(Transport.java:197) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:196) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)