search

pinpoint-apm / pinpoint

APM, (Application Performance Management) tool for large-scale distributed systems.
https://pinpoint-apm.gitbook.io/
Apache License 2.0
13.42k stars 3.76k forks source link

Running Pinpoint Agent with JBoss on Java 11 #7393

Open davide-parini opened 4 years ago

davide-parini commented 4 years ago

I'm trying to monitor a JBoss instance running on OpenJDK 11 with Pinpoint 2.0.1.

I've added the required java options in order to do this, as outlined here.

Since Java 9, however, -Xbootclasspath\p has been removed. Could you please update the doc with the proper setup to run on Java 9+ also?

emeroad commented 4 years ago

https://stackoverflow.com/questions/53529593/how-to-add-a-jar-to-the-boot-classpath-in-java-11

-Djboss.modules.system.pkgs=org.jboss.logmanager

Could you test if it works with -Djboss.modules.system.pkgs?

davide-parini commented 4 years ago

Hi @emeroad, yes. That was one of the required system properties also with previous versions of Java. Here is some more detail.

I have set the following parameters, replacing -Xbootclasspath/p with -Xbootclasspath/a:

set "JAVA_OPTS=%JAVA_OPTS%
-Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager,com.navercorp.pinpoint.bootstrap,com.navercorp.pinpoint.common,com.navercorp.pinpoint.exception"
set "JAVA_OPTS=%JAVA_OPTS% -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
set "JAVA_OPTS=%JAVA_OPTS% -Xbootclasspath/a:C:\jboss-eap-7.3\modules\system\layers\base\org\jboss\logmanager\main\jboss-logmanager-2.1.14.Final-redhat-00001.jar"

And got a NoClassDefFoundError on org.wildfly.common.net.HostName. So i added its jar to the boot classpath:

set "JAVA_OPTS=%JAVA_OPTS%
-Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager,com.navercorp.pinpoint.bootstrap,com.navercorp.pinpoint.common,com.navercorp.pinpoint.exception"
set "JAVA_OPTS=%JAVA_OPTS% -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
set "JAVA_OPTS=%JAVA_OPTS% -Xbootclasspath/a:C:\jboss-eap-7.3\modules\system\layers\base\org\jboss\logmanager\main\jboss-logmanager-2.1.14.Final-redhat-00001.jar;C:\jboss-eap-7.3\modules\system\layers\base\org\wildfly\common\main\wildfly-common-1.5.1.Final-redhat-00001.jar"

And now i get the following error on startup in server.log:

2020-11-10T09:38:18.903+0100 INFO  [org.jboss.modules] (main) JBoss Modules version 1.10.0.Final-redhat-00001
2020-11-10T09:38:19.619+0100 FATAL [org.jboss.as.server] (main) WFLYSRV0239: Aborting with exit code 1
2020-11-10T09:38:23.872+0100 SEVERE [io.grpc.internal.SerializingExecutor] (Pinpoint-AgentGrpcDataSender-Channel-Executor(11-0)) Exception while executing runnable io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed@4ba96d0e: java.lang.IllegalStateException: call already half-closed
    at pinpoint.agent/pinpoint.agent/com.google.common.base.Preconditions.checkState(Preconditions.java:444) [guava-20.0.jar:]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:406) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.PartialForwardingClientCall.halfClose(PartialForwardingClientCall.java:44) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:22) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.ForwardingClientCall$SimpleForwardingClientCall.halfClose(ForwardingClientCall.java:44) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.stub.ClientCalls$CallToStreamObserverAdapter.onCompleted(ClientCalls.java:331) [grpc-stub-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.profiler.sender.grpc.StreamUtils.close(StreamUtils.java:31) [pinpoint-profiler-2.0.1.1.0.jar:2.0.1.1.0]
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.profiler.receiver.grpc.GrpcCommandService$CommandServiceMainStreamObserver.onCompleted(GrpcCommandService.java:164) [pinpoint-profiler-2.0.1.1.0.jar:2.0.1.1.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:418) [grpc-stub-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.14.0.jar:1.14.0]
    at pinpoint.agent/pinpoint.agent/io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.14.0.jar:1.14.0]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
    at java.base/java.lang.Thread.run(Thread.java:834) [java.base:]

Here is the corresponding console output when launching the start script: startup.log

Any help would be appreciated. :-)

emeroad commented 4 years ago 
11-10 20:08:38.736 INFO  PinpointStarter                     : pinpoint agent [com.navercorp.pinpoint.profiler.DefaultAgent] starting...
11-10 20:08:38.742 INFO  Log4j2LoggingSystem                 : logPath:file:/D:/master/pinpoint/agent/target/pinpoint-agent-2.1.1-SNAPSHOT/profiles/release/log4j2.xml
11-10 20:08:38.747 INFO  Log4j2LoggingSystem                 : patch ReflectionUtil codeSource:(file:/D:/master/pinpoint/agent/target/pinpoint-agent-2.1.1-SNAPSHOT/lib/pinpoint-profiler-logging-2.1.1-SNAPSHOT.jar <no signer certificates>)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jboss.logmanager.LogLevelInitTask to constructor java.util.logging.Level$KnownLevel(java.util.logging.Level)
WARNING: Please consider reporting this to the maintainers of org.jboss.logmanager.LogLevelInitTask
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
11-10 20:09:54.054 [           main] WARN  p.p.c.p.s.f.FileDescriptorMetricProvider -- Unsupported operating system WINDOW/JAVA_9/ORACLE
11-10 20:09:54.424 INFO  PinpointStarter                     : pinpoint agent started normally.
20:09:54,740 INFO  [org.jboss.modules] (main) JBoss Modules version 1.8.6.Final
java.lang.IllegalStateException: The LogManager was not properly installed (you must set the "java.util.logging.manager" system property to "org.jboss.logmanager.LogManager")
    at org.jboss.logmanager.Logger.getLogger(Logger.java:57)
    at org.jboss.as.server@6.0.2.Final//org.jboss.as.server.Main.main(Main.java:89)
    at org.jboss.modules.Module.run(Module.java:352)
    at org.jboss.modules.Module.run(Module.java:320)
    at org.jboss.modules.Main.main(Main.java:593)
20:09:54,997 FATAL [org.jboss.as.server] (main) WFLYSRV0239: Aborting with exit code 1

vm option 

-Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager,com.navercorp.pinpoint.bootstrap,com.navercorp.pinpoint.common,com.navercorp.pinpoint.exception
-Xbootclasspath/a:D:\project\was\wildfly-servlet-14.0.1.Final/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.4.Final.jar;D:\project\was\wildfly-servlet-14.0.1.Final\modules\system\layers\base\org\wildfly\common\main\wildfly-common-1.4.0.Final.jar
-Djava.util.logging.manager=org.jboss.logmanager.LogManager
-javaagent:D:\master\pinpoint\agent\target\pinpoint-agent-2.1.1-SNAPSHOT\pinpoint-bootstrap.jar
-Dpinpoint.agentId=emeroad-agent-wild
-Dpinpoint.applicationName=emeroad-app-wild

image

I'm not sure why. java.util.logging.LogManager does not find jboss.LogManager in Java9,

davide-parini commented 4 years ago

Hello @emeroad, i think i found a workaround. Adding -Dsun.util.logging.disableCallerCheck=true did the trick.

I also found an issue in RedHat Jira about this precise issue: https://issues.redhat.com/browse/WFCORE-1494

Do you think this could be considered a solution?

zakariamestaoui commented 1 year ago

Hello @davide-parini, I'm trying to monitor a JBoss instance running on java 11 with Pinpoint 2.5.0, I have the same issue, I tried all the above and I still get (java.lang.IllegalStateException: call already half-closed), do you please have any suggestions?

JsckSons commented 1 year ago

@emeroad hi, i have the same problem. how to resolved it.

JsckSons commented 1 year ago

Hello @emeroad, i think i found a workaround. Adding -Dsun.util.logging.disableCallerCheck=true did the trick.

I also found an issue in RedHat Jira about this precise issue: https://issues.redhat.com/browse/WFCORE-1494

Do you think this could be considered a solution?

@davide-parini hi, i think you add the options of -Dsun.util.logging.disableCallerCheck=true and -Xbootclasspath/a ,right?