search

cryostatio / cryostat-agent

Java Agent implementing Cryostat Discovery and JFR data pushing
https://cryostat.io/
Other
3 stars 8 forks source link

[Bug] Agent startup failure with JBoss EAP #269

Open andrewazores opened 11 months ago

andrewazores commented 11 months ago 
2023-11-27 18:57:54:554 +0000 [cryostat-agent-main] ERROR io.cryostat.agent.Agent - Agent startup failure
java.lang.RuntimeException: org.openjdk.jmc.rjmx.ConnectionException caused by javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class org.jboss.as.jmx.PluggableMBeanServerBuilder: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
    at io.cryostat.agent.MainModule.provideJvmId(MainModule.java:307)
    at io.cryostat.agent.MainModule_ProvideJvmIdFactory.provideJvmId(MainModule_ProvideJvmIdFactory.java:42)
    at io.cryostat.agent.MainModule_ProvideJvmIdFactory.get(MainModule_ProvideJvmIdFactory.java:34)
    at io.cryostat.agent.MainModule_ProvideJvmIdFactory.get(MainModule_ProvideJvmIdFactory.java:12)
    at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.cryostat.agent.MainModule_ProvideCryostatClientFactory.get(MainModule_ProvideCryostatClientFactory.java:66)
    at io.cryostat.agent.MainModule_ProvideCryostatClientFactory.get(MainModule_ProvideCryostatClientFactory.java:15)
    at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.cryostat.agent.MainModule_ProvideRegistrationFactory.get(MainModule_ProvideRegistrationFactory.java:74)
    at io.cryostat.agent.MainModule_ProvideRegistrationFactory.get(MainModule_ProvideRegistrationFactory.java:13)
    at dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.cryostat.agent.DaggerAgent_Client$ClientImpl.registration(DaggerAgent_Client.java:257)
    at io.cryostat.agent.Agent.main(Agent.java:67)
    at io.cryostat.agent.Agent.lambda$agentmain$2(Agent.java:153)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.openjdk.jmc.rjmx.ConnectionException caused by javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class org.jboss.as.jmx.PluggableMBeanServerBuilder: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
    at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:345)
    at io.cryostat.agent.shaded.io.cryostat.core.net.JFRJMXConnection.attemptConnect(JFRJMXConnection.java:412)
    at io.cryostat.agent.shaded.io.cryostat.core.net.JFRJMXConnection.connect(JFRJMXConnection.java:368)
    at io.cryostat.agent.shaded.io.cryostat.core.net.JFRJMXConnection.getJvmId(JFRJMXConnection.java:171)
    at io.cryostat.agent.MainModule.provideJvmId(MainModule.java:302)
    ... 14 more
Caused by: javax.management.JMRuntimeException: Failed to load MBeanServerBuilder class org.jboss.as.jmx.PluggableMBeanServerBuilder: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
    at java.management/javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:502)
    at java.management/javax.management.MBeanServerFactory.getNewMBeanServerBuilder(MBeanServerFactory.java:535)
    at java.management/javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:316)
    at java.management/javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:273)
    at org.openjdk.jmc.rjmx.internal.SyntheticAttributeRepository.<init>(SyntheticAttributeRepository.java:78)
    at org.openjdk.jmc.rjmx.internal.MCMBeanServerConnection.<init>(MCMBeanServerConnection.java:80)
    at org.openjdk.jmc.rjmx.internal.RJMXConnection.connect(RJMXConnection.java:336)
    ... 18 more
Caused by: java.lang.ClassNotFoundException: org.jboss.as.jmx.PluggableMBeanServerBuilder
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
    at java.management/javax.management.MBeanServerFactory.loadBuilderClass(MBeanServerFactory.java:444)
    at java.management/javax.management.MBeanServerFactory.checkMBeanServerBuilder(MBeanServerFactory.java:487)
    ... 24 more
ebaron commented 11 months ago

Came across this VisualVM issue: https://stackoverflow.com/a/38455790. Adding -Djboss.platform.mbeanserver=true fixed it for us too.

andrewazores commented 11 months ago

Maybe we should set up a project wiki to track specific bugs and workarounds like that.

ebaron commented 11 months ago

Came across this VisualVM issue: https://stackoverflow.com/a/38455790. Adding -Djboss.platform.mbeanserver=true fixed it for us too.

I might have spoken too soon. I think I just missed the exception in the logs the first time.

andrewazores commented 11 months ago

I can do some deeper digging to see if it's feasible to replace all of the Agent's internal uses of various MBeans with direct accesses, but IIRC there was something that we're using from the FlightRecorderMXBean which the jdk.jfr package doesn't have in its API. At the very least this is likely to break the Smart Triggers functionality which explicitly depends on reading metrics from MBeans.

FranciscoMeloJr commented 7 months ago

Hello, this jira came to my attention today April 2nd by @Leticia Konno. Which EAP version is that specifically? I remember there was some discussion about agent usage and MODULES_OPTS, and for 7.4+ I'd try moving the agent to MODULE_OPTS so it doesn't start too early and see if that resolves it. I explain this here: https://access.redhat.com/solutions/7021605

andrewazores commented 7 months ago

I can do some deeper digging to see if it's feasible to replace all of the Agent's internal uses of various MBeans with direct accesses, but IIRC there was something that we're using from the FlightRecorderMXBean which the jdk.jfr package doesn't have in its API. At the very least this is likely to break the Smart Triggers functionality which explicitly depends on reading metrics from MBeans.

This was fixed in https://github.com/cryostatio/cryostat-agent/pull/280 .

@FranciscoMeloJr I can't recall which EAP version we tested at the time I filed this bug report. @ebaron @sefroberg do you know? We had a lot of discussions back then about MODULES_OPTS and things as well and IIRC we did come up with some form of workaround.

ebaron commented 7 months ago

I think the problem we wre facing with EAP 7.4+ with MODULE_OPTS was due to jboss-modules trying to load jars from the manifest's Class-Path entry. That's been removed with https://github.com/cryostatio/cryostat-agent/pull/311, so I would expect this to work now, but I haven't tested it myself yet.