JBoss plugin with EJB Trace error in agent logs

[amardeep-s]

What version of pinpoint are you using?

v1.8.4 and v2.0.0-RC1

Describe the bug

Installed Pinpoint 2.0.0-RC1 and setup Agents in JBoss. The initial startup of agent works correctly, though when make EJB Connection from application its throwing the below error. Refer to logs section.

Please note that the same behaviour works fine with Pinpoint 1.8.4 and Agent 1.8.4.

I have tried with agent v1.8.4 and v2.0.0-RC1 connected to collector v2.0.0-RC1

What did you do to trigger the bug?

Steps to reproduce, for example:

1. Started Jboss 7.0.2 with agent installed with following items in pinpoint.config

profiler.jboss.enable=true profiler.jboss.hidepinpointheader=true profiler.jboss.excludeurl= profiler.jboss.tracerequestparam=true profiler.jboss.traceEjb=true

2. Launched EJB application deployed in JBoss.
3. Agent throws the error in logs. The logs added below are from v.2.0.0-RC1 agent.

Expected behavior

Agent should start the remote ejb trace of methods.

Screenshots

If applicable, add screenshots to help explain your problem.

Logs

2019-12-06 10:24:00 [INFO ](c.n.p.p.t.TomcatPlugin ) Detected application type : JBOSS 2019-12-06 10:24:00 [INFO ](c.n.p.p.t.TomcatPlugin ) Not adding Tomcat transformers 2019-12-06 10:24:00 [INFO ](.DefaultProfilerPluginContextLoader) class com.navercorp.pinpoint.plugin.jboss.JbossPlugin Plugin Pinpoint-Plugin-Package:[com.navercorp.pinpoint.plugin] 2019-12-06 10:24:00 [INFO ](.DefaultProfilerPluginContextLoader) Loading plugin:com.navercorp.pinpoint.plugin.jboss.JbossPlugin pluginPackage:com.navercorp.pinpoint.plugin.jboss.JbossPlugin@74518890 2019-12-06 10:24:00 [INFO ](c.n.p.p.p.DefaultPluginSetup ) com.navercorp.pinpoint.plugin.jboss.JbossPlugin Plugin setup 2019-12-06 10:24:00 [INFO ](c.n.p.p.j.JbossPlugin ) JbossPlugin config:JbossConfig{hidePinpointHeader=true, excludeUrlFilter=com.navercorp.pinpoint.bootstrap.config.SkipFilter@4c5204af, traceEjb=true, enable=true, bootstrapMains=[org.jboss.modules.Main], realIpHeader='X-Real-IP', realIpEmptyValue='null', traceRequestParam=true, excludeProfileMethodFilter=com.navercorp.pinpoint.bootstrap.config.SkipFilter@3f3ddbd9} 2019-12-06 10:24:00 [INFO ](c.n.p.p.j.JbossPlugin ) Adding JBoss transformers 2019-12-06 10:24:01 [INFO ](c.n.p.p.AgentInfoSender ) Sending AgentInfo AgentInfo{agentInformation=DefaultAgentInformation{agentId='app1', applicationName='app-backend', isContainer=false, startTime=1575627833075, pid=3349, machineName='testserver', hostIp='192.168.x.x', serverType=JBOSS, jvmVersion='1.8.0_181', agentVersion='2.0.0-RC1'}, serverMetaData=DefaultServerMetaData{serverInfo='', vmArgs=[-D[Server:server-three], -Xms1024m, -Xmx2048m, -XX:MetaspaceSize=256m, -XX:MaxMetaspaceSize=1024m, -Xbootclasspath/p:/usr/share/jboss-eap-7.0/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.3.Final-redhat-1.jar, -javaagent:/opt/apps/pinpoint-agent/pinpoint-bootstrap-2.0.0-RC1.jar, -Dpinpoint.applicationName=PROPCO-QA, -Dpinpoint.agentId=QA2, -Dpinpoint.bootWait=500, -Djava.awt.headless=true, -Djava.net.preferIPv4Stack=true, -Djava.util.logging.manager=org.jboss.logmanager.LogManager, -Djboss.as.management.blocking.timeout=1500, -Djboss.home.dir=/usr/share/jboss-eap-7.0, -Djboss.modules.system.pkgs=org.jboss.logmanager,com.navercorp.pinpoint.bootstrap,com.navercorp.pinpoint.common,com.navercorp.pinpoint.exception, -Dnet.sf.jasperreports.compiler.classpath=/usr/share/jboss-eap-7.0/modules/system/layers/base/net/sf/jasperreports/main/*, -Dorg.jboss.resolver.warning=true, -Dsun.rmi.dgc.client.gcInterval=3600000, -Dsun.rmi.dgc.server.gcInterval=3600000, -Djboss.server.log.dir=/usr/share/jboss-eap-7.0/domain/servers/server-three/log, -Djboss.server.temp.dir=/usr/share/jboss-eap-7.0/domain/servers/server-three/tmp, -Djboss.server.data.dir=/usr/share/jboss-eap-7.0/domain/servers/server-three/data, -Dlogging.configuration=file:/usr/share/jboss-eap-7.0/domain/servers/server-three/data/logging.properties], connectors={}, serviceInfo=[]}, jvmInformation=com.navercorp.pinpoint.profiler.JvmInformation@719434fd} 2019-12-06 10:24:01 [INFO ](c.n.p.p.s.Java7ShutdownHookRegister) register() started. 2019-12-06 10:24:01 [INFO ](c.n.p.p.s.Java7ShutdownHookRegister) register() completed. 2019-12-06 10:24:01 [INFO ](c.n.p.p.r.g.GrpcCommandService ) Connect to CommandServiceStream completed. 2019-12-06 10:24:01 [INFO ](c.n.p.p.s.g.PingStreamContext ) PingStream-2 onReady 2019-12-06 10:24:01 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) onReadyHandler:StatStream-3 eventNumber:0 2019-12-06 10:24:01 [INFO ](c.n.p.p.AgentInfoSender ) AgentInfo sent. 2019-12-06 10:24:01 [INFO ](c.n.p.p.s.g.PingStreamContext ) PingStream-2 success: 2019-12-06 10:25:01 [INFO ](c.n.p.p.s.g.PingStreamContext ) PingStream-2 success: 2019-12-06 10:25:44 [WARN ](c.n.p.p.i.c.ReflectionDefineClass ) com.navercorp.pinpoint.plugin.jboss.JbossAsyncListener define fail cl:ModuleClassLoader for Module "org.jboss.as.ejb3:main" from local module loader @f1a45f8 (finder: local module finder @5edf2821 (roots: /usr/share/jboss-eap-7.0/modules,/usr/share/jboss-eap-7.0/modules/system/layers/base)) Caused by:null java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.navercorp.pinpoint.profiler.instrument.classloading.ReflectionDefineClass.defineClass(ReflectionDefineClass.java:48) at com.navercorp.pinpoint.profiler.instrument.classloading.PlainClassLoaderHandler.defineClass(PlainClassLoaderHandler.java:289) at com.navercorp.pinpoint.profiler.instrument.classloading.PlainClassLoaderHandler.define0(PlainClassLoaderHandler.java:276) at com.navercorp.pinpoint.profiler.instrument.classloading.PlainClassLoaderHandler.defineJarClass(PlainClassLoaderHandler.java:214) at com.navercorp.pinpoint.profiler.instrument.classloading.PlainClassLoaderHandler.getClassLoaderAttachment(PlainClassLoaderHandler.java:160) at com.navercorp.pinpoint.profiler.instrument.classloading.PlainClassLoaderHandler.injectClass0(PlainClassLoaderHandler.java:116) at com.navercorp.pinpoint.profiler.instrument.classloading.PlainClassLoaderHandler.injectClass(PlainClassLoaderHandler.java:74) at com.navercorp.pinpoint.profiler.instrument.classloading.JarProfilerPluginClassInjector.injectClass(JarProfilerPluginClassInjector.java:65) at com.navercorp.pinpoint.profiler.plugin.PluginInstrumentContext.injectClass(PluginInstrumentContext.java:163) at com.navercorp.pinpoint.profiler.plugin.DynamicTransformCallbackProvider.getTransformCallback(DynamicTransformCallbackProvider.java:49) at com.navercorp.pinpoint.profiler.plugin.MatchableClassFileTransformerDelegate.transform(MatchableClassFileTransformerDelegate.java:65) at com.navercorp.pinpoint.profiler.BaseClassFileTransformer.transform(BaseClassFileTransformer.java:56) at com.navercorp.pinpoint.profiler.DefaultClassFileTransformerDispatcher.transform(DefaultClassFileTransformerDispatcher.java:95) at sun.instrument.TransformerManager.transform(TransformerManager.java:188) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at org.jboss.modules.ModuleClassLoader.doDefineOrLoadClass(ModuleClassLoader.java:358) at org.jboss.modules.ModuleClassLoader.defineClass(ModuleClassLoader.java:437) at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:274) at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:78) at org.jboss.modules.Module.loadModuleClass(Module.java:605) at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190) at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363) at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351) at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93) at org.jboss.as.ejb3.component.stateless.StatelessComponentDescription$1.configure(StatelessComponentDescription.java:103) at org.jboss.as.ee.component.deployers.EEModuleConfigurationProcessor.deploy(EEModuleConfigurationProcessor.java:92) at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:147) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: javax/servlet/AsyncListener at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.lang.ClassLoader.defineClass(ClassLoader.java:642) ... 37 more Caused by: java.lang.ClassNotFoundException: javax.servlet.AsyncListener from [Module "org.jboss.as.ejb3:main" from local module loader @f1a45f8 (finder: local module finder @5edf2821 (roots: /usr/share/jboss-eap-7.0/modules,/usr/share/jboss-eap-7.0/modules/system/layers/base))] at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:198) at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:363) at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:351) at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:93) ... 40 more 2019-12-06 10:25:44 [WARN ](c.n.p.p.i.c.PlainClassLoaderHandler) Failed to load plugin class com.navercorp.pinpoint.plugin.jboss.JbossPlugin$EjbBMTInterceptorTransform with classLoader ModuleClassLoader for Module "org.jboss.as.ejb3:main" from local module loader @f1a45f8 (finder: local module finder @5edf2821 (roots: /usr/share/jboss-eap-7.0/modules,/usr/share/jboss-eap-7.0/modules/system/layers/base)) java.lang.RuntimeException: com.navercorp.pinpoint.plugin.jboss.JbossAsyncListener define fail Caused by:null

Additional context

Add any other context about the problem here, such as affected library for agents, how your collector/web/hbase is set up if applicable.

[jaehong-kim]

@amardeep-s Have you read the Jboss Plugin documentation? https://github.com/naver/pinpoint/tree/master/plugins/jboss

[amardeep-s]

@jaehong-kim - Yes indeed I read the documentation. We already have Pinpoint 1.8.4 server running with agent 1.7.1 deployed in JBoss Domain and Standalone.

I wanted to upgrade to 2.0 for testing new UI and new features.

[jaehong-kim]

@amardeep-s The error occurs because the javax.servlet.AsyncListener class file was not found.

java.lang.NoClassDefFoundError: javax/servlet/AsyncListener

Setting the profiler.jboss.traceEjb = true option in the log shows logic that should not be executed. Please show the full log.

[amardeep-s]

@jaehong-kim - Apologies of the delay. Please find the full logs attached as requested. QA1-pinpoint.log

[jaehong-kim]

@amardeep-s It may be a bug.

To solve without the patch, you need javax.servlet.* files. See the issue below. https://developer.jboss.org/thread/171327

[amardeep-s]

@jaehong-kim I already have this present in JBoss home dir /usr/share/jboss-eap-7.0/modules/system/layers/base/javax/servlet/api/main/jboss-servlet-api_3.1_spec-1.0.0.Final-redhat-1.jar

What exactly you want me to do?

[jaehong-kim]

@amardeep-s Try adding it to the -Xbootclasspath value. Please.

/usr/share/jboss-eap-7.0/modules/system/layers/base/javax/servlet/api/main/jboss-servlet-api_3.1_spec-1.0.0.Final-redhat-1.jar

[amardeep-s]

@jaehong-kim Tried the same, still no luck. In fact this time I setup fresh jboss with 2.0.0RC1 agent in standalone mode. Some lines from settings done.

-Xbootclasspath/p:/home/amardeep/jboss-eap-7.0/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.3.Final-redhat-1.jar:/home/amardeep/jboss-eap-7.0/modules/system/layers/base/javax/servlet/api/main/jboss-servlet-api_3.1_spec-1.0.0.Final-redhat-1.jar -javaagent:/home/amardeep/jboss-eap-7.0/pinpoint-agent/pinpoint-bootstrap-2.0.0-RC1.jar -Dpinpoint.applicationName=APP-TEST -Dpinpoint.agentId=APP-TEST

I have also attached log for review. APP-TEST-pinpoint.log

[amardeep-s]

Any updates on the issue? I can also confirm that using Agent Version 1.7.1 EJB trace is working with Collector 2.0.0-RC1.

Another thing I noticed is, MYSQL connections are not visible in server map on home page. I can see MySQL connections in transaction detail view if I click Server Map. Do need to raise another issue for that?

[stale[bot]]

This issue/proposal has been automatically marked as stale because it hasn't had any recent activity. It will automatically be closed if no further activity occurs for 20days. If you think this should still be open, or the problem still persists, just pop a reply in the comments and one of the maintainers will (try!) to follow up. Thank you for your interest and contribution to the Pinpoint Community.