search

cryostatio / cryostat-core

Core library providing a convenience wrapper and headless stubs for managing JFR with JDK Mission Control API
https://cryostat.io/
Other
8 stars 21 forks source link

[Bug] java.rmi.NoSuchObjectException: no such object in table #216

Open andrewazores opened 1 year ago

andrewazores commented 1 year ago

https://stackoverflow.com/questions/645208/java-rmi-nosuchobjectexception-no-such-object-in-table

May 09, 2023 9:55:24 PM io.cryostat.core.log.Logger info
INFO: Connection for service:jmx:rmi:///jndi/rmi://10-131-1-37.redacted.pod:9091/jmxrmi closed
May 09, 2023 9:55:24 PM io.cryostat.core.log.Logger error
SEVERE: Exception thrown
org.openjdk.jmc.rjmx.ConnectionException caused by java.rmi.NoSuchObjectException: no such object in table
at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:345)
at io.cryostat.core.net.JFRJMXConnection.attemptConnect(JFRJMXConnection.java:443)
at io.cryostat.core.net.JFRJMXConnection.connect(JFRJMXConnection.java:399)
at io.cryostat.core.net.JFRJMXConnection.getJvmId(JFRJMXConnection.java:204)
at io.cryostat.net.TargetConnectionManager.lambda$executeConnectedTaskAsync$2(TargetConnectionManager.java:153)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: java.rmi.NoSuchObjectException: no such object in table
at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:304)
at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:280)
at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:165)
at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(RMIServerImpl_Stub.java:83)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2107)
at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
at org.openjdk.jmc.rjmx.internal.RJMXConnection.connectJmxConnector(RJMXConnection.java:575)
at org.openjdk.jmc.rjmx.internal.RJMXConnection.establishConnection(RJMXConnection.java:552)
andrewazores commented 1 year ago

I think this happened when I was helping someone get Cryostat to discover and connect to an application, and the JMX ports were set up like I had written in this blog post:

https://quarkus.io/blog/monitoring-quarkus-jvm-mode-with-cryostat/

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=${RJMX_PORT}
-Dcom.sun.management.jmxremote.rmi.port=${RJMX_PORT}
-Djava.rmi.server.hostname=127.0.0.1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false

Where both the remote JMX and RMI ports were set to ex. 9091.

Trimming these env vars down to simply:

-Dcom.sun.management.jmxremote.port=${RJMX_PORT}
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

resolved the issue.

I seem to remember using the original configuration successfully in the past, and I did document this in the linked blog post, so I don't know what the difference is compared to now. I need to verify that this is actually the root cause and a real fix for the problem, not just a heisenbug and a coincidence. If this is real then it would be good to understand why, and at least to document the updated requirements.