search

criteo / cassandra_exporter

Apache Cassandra® metrics exporter for Prometheus
Apache License 2.0
168 stars 96 forks source link

Connection to JMX with SSL enabled is interrupting with NoSuchObjectException: no such object in table #114

Open uladzimir-shelhunou opened 1 year ago

uladzimir-shelhunou commented 1 year ago

We are trying to use an exporter with JMX configured to accept only SSL connections.

Here is the part of Config.yml:

host: cass-cluster-0.cass-cluster.cassandra-cluster.svc.cluster.local:7199 ssl: true user: cassandra password: cassandra listenAddress: 0.0.0.0 listenPort: 8888 additionalLabelsFromEnvvars: "^cassandraNode$"

We also have trustStore specified with JVM args.

It is working fine for some time but after several minutes, we start to see the errors in the logs: [main] INFO com.criteo.nosql.cassandra.exporter.Config - Loading yaml config from ./config.yml [main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 11995ms for the whole run [main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 6781ms for the whole run [main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 5891ms for the whole run [main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 3686ms for the whole run [main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 3549ms for the whole run [main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 3204ms for the whole run [main] INFO com.criteo.nosql.cassandra.exporter.JmxScraper - Scrap took 3361ms for the whole run Mar 02, 2023 1:17:07 PM com.sun.jmx.remote.internal.ClientCommunicatorAdmin restart WARNING: Failed to restart: java.rmi.NoSuchObjectException: no such object in table [main] ERROR com.criteo.nosql.cassandra.exporter.JmxScraper - Cannot retrieve the cluster name information for the node java.rmi.NoSuchObjectException: no such object in table at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303) at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279) at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164) at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source) at java.management.rmi/javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:904) at com.criteo.nosql.cassandra.exporter.JmxScraper$NodeInfo.getNodeInfo(JmxScraper.java:433) at com.criteo.nosql.cassandra.exporter.JmxScraper.run(JmxScraper.java:195) at com.criteo.nosql.cassandra.exporter.Main.start(Main.java:44) at com.criteo.nosql.cassandra.exporter.Main.main(Main.java:30) [main] ERROR com.criteo.nosql.cassandra.exporter.Main - Scrapper stopped due to uncaught exception java.rmi.NoSuchObjectException: no such object in table at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303) at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279) at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164) at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105) at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321) at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at com.criteo.nosql.cassandra.exporter.JmxScraper.run(JmxScraper.java:186) at com.criteo.nosql.cassandra.exporter.Main.start(Main.java:44) at com.criteo.nosql.cassandra.exporter.Main.main(Main.java:30) [main] ERROR com.criteo.nosql.cassandra.exporter.Main - Scrapper stopped due to uncaught exception java.rmi.NoSuchObjectException: no such object in table at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303) at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279) at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164) at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105) at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321) at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at com.criteo.nosql.cassandra.exporter.JmxScraper.run(JmxScraper.java:186) at com.criteo.nosql.cassandra.exporter.Main.start(Main.java:44) at com.criteo.nosql.cassandra.exporter.Main.main(Main.java:30) [main] ERROR com.criteo.nosql.cassandra.exporter.Main - Scrapper stopped due to uncaught exception java.rmi.NoSuchObjectException: no such object in table at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303) at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279) at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164) at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105) at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321) at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at com.criteo.nosql.cassandra.exporter.JmxScraper.run(JmxScraper.java:186) at com.criteo.nosql.cassandra.exporter.Main.start(Main.java:44) at com.criteo.nosql.cassandra.exporter.Main.main(Main.java:30) [main] ERROR com.criteo.nosql.cassandra.exporter.Main - Scrapper stopped due to uncaught exception java.rmi.NoSuchObjectException: no such object in table at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303) at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279) at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164) at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105) at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321) at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at com.criteo.nosql.cassandra.exporter.JmxScraper.run(JmxScraper.java:186) at com.criteo.nosql.cassandra.exporter.Main.start(Main.java:44) at com.criteo.nosql.cassandra.exporter.Main.main(Main.java:30)

Could you please suggest why is it happening and how to fix it?

NitinBagadiya commented 1 year ago

Hi Team, can somebody please take a look at mentioned issue? is this an active issue channel to report an issue?