Closed AshutoshNirkhe closed 1 year ago
To narrow this down we need to figure out if the root cause is in jmx_exporter
, in the Artifactory, or in the network connection. Is there any chance that you can connect another JMX client, like Java's jconsole
and see if that works?
To narrow this down we need to figure out if the root cause is in
jmx_exporter
, in the Artifactory, or in the network connection. Is there any chance that you can connect another JMX client, like Java'sjconsole
and see if that works?
Hi @fstab Thanks for your reply and really sorry for getting back so late on this topic. The overall setup is bit weird I guess. I did try to do expose the port 9010 outside through Artifactory container via the k8s service then port-forwarded it and used jconsole on my machine to connect to localhost:9010 , but the connection doesn't work that way. But I feel this is different networking issue than what face with jmx-exporter. Any other way I can try this out or other suggestions ?
@fstab I can still reproduce this here. The jmx exporter is directly installed on the same machine where the remote jmx is located.
These are the JMX settings of the application:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=9911
This is the exporter config
---
hostPort: localhost:9911
ssl: false
startDelaySeconds: 0
rules:
- pattern: ".*"
Running it like that results in this Stacktrace:
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: Sep. 20, 2022 10:21:23 VORM. io.prometheus.jmx.JmxCollector collect
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: SCHWERWIEGEND: JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table]
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:95)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:599)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.client.Collector.collect(Collector.java:45)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.client.exporter.common.TextFormat.writeOpenMetrics100(TextFormat.java:202)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:57)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:100)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.base/java.lang.Thread.run(Thread.java:834)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: Caused by: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table]
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:207)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: ... 19 more
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: Caused by: java.rmi.NoSuchObjectException: no such object in table
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:380)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:123)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: ... 24 more
I was expecting to run it on hostPort: localhost:9010
but that doesn't work at all (connecting to the exporter times out in that case).
Okay, apparently it boils down that you should use the jmx port (in my example 9010) and NOT the rmi port (9911). I've turned up the debug logs and there is a connection but somehow it's stopping midway. I'm investigating this, is probably something in the app.
HA! Found it. My JNDI datasources somehow didn't respond correctly to JMX calls. I put it on the blocklist and now everything works.
So I guess the main part of this issue was, that the OP used the RMI port instead of the JMX port which lead to the error.
Closed as resolved.
We are running jmx-exporter as sidecar to JFROG Artifactory container in AKS cluster so as to get the visibility into the Java Beans metrics.
Extract of our current JMX config,
But JMX Exporter keeps failing intermittently with error "JMX scrape failed: java.rmi.NoSuchObjectException: no such object in table", resulting in 'TargetDown' Prometheus alert. Exception trace below,
I am not sure what's the reason behind this because Artifactory itself appears to be running fine. But the metrics can be seen coming in Grafana dashboards in Gaps as below,
I couldn't find any known bugs around this apart from this generic one https://stackoverflow.com/questions/645208/java-rmi-nosuchobjectexception-no-such-object-in-table