search

pinpoint-apm / pinpoint

APM, (Application Performance Management) tool for large-scale distributed systems.
https://pinpoint-apm.gitbook.io/
Apache License 2.0
13.43k stars 3.76k forks source link

Issue with Concurrent Execution of Multiple APMs Due to Module Permissions(타 APM 동시 실행 시 Module 권한 관련 이슈.) #11169

Open leon4652 opened 5 months ago

leon4652 commented 5 months ago

Prerequisites

Please check the FAQ, and search existing issues for similar questions before creating a new issue.YOU MAY DELETE THIS PREREQUISITES SECTION.

What version of pinpoint are you using?

master/v1.7.3/v1.7.2 etc.

Pinpoint Collector, Agent, Web Version: 2.3.3 and compatible versions Instrumentation Server: Java 9 and above (specifically tested on 11, 17, and 21, with logs from version 21)

핀포인트 Collector, Agent, Web 버전 : 2.3.3 및 상응 호환 버전
계측 서버 : Java 9 이상 (11, 17, 21), 로그는 21버전 실행 시 발생

Describe your problem**

Dear Support Team,

I am writing to inquire about an issue we are encountering with the concurrent use of multiple APM tools. Currently, we are using Pinpoint alongside another asm, bytecode manipulation monitoring agent, Opentelemetry, which is based on ByteBuddy.

Issue Summary:

Java 8:

When building with the same -javaagent options, there are no issues. Both agents run successfully and perform their respective instrumentation tasks without any conflicts.

Java 9 and above (Java 11, 17, 21):

Concurrent builds fail for Pinpoint, although Opentelemetry continues to run successfully. From the logs, it appears that the failure is due to the inability to obtain necessary module permissions (unnamed module). .. particularly seems to be an issue related to Netty and Grpc.
We suspect this issue arises from module permissions required by Pinpoint in Java versions 9 and above. We would like to know if there are any configuration settings or build options within Pinpoint that can resolve this issue.

Attached are the build scripts and error logs for your reference. Thank you for your assistance.

안녕하세요. APM 사용 이슈 관련 문의드립니다.
현재 Pinpoint와 유사한 ASM, ByteBuddy 기반의 바이트코드 조작 모니터링 Agent를 병행하여 사용중입니다. (Opentelemetry)

Java 8의 경우.
빌드 시 동일한 -javaagent 옵션을 통해 빌드해도 문제가 생기지 않습니다.
정상적으로 잘 실행되며 각각 계측이 가능합니다.

Java 9 이상 버전의 경우 (Java 11, 17, 21)
동시 빌드의 경우 Pinpoint의 빌드가 실패합니다. (타 Apm- opentelemetry의 경우는 실행됩니다.)
로그를 확인해 본 결과, Module 권한을 얻지 못해 발생한 일로 추측됩니다. (unnamed module) .. 특히 Netty와 Grpc 관련 이슈로 보입니다.

핀포인트에서 Configue 설정이나 빌드 시 설정을 통해 해당 이슈를 해결할 수 있는지 문의드립니다.
빌드 스크립트와 에러 로그를 첨부합니다. 읽어 주셔서 감사합니다.

Logs

에러 로그(Error log)

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-06-20 11:23:15:294 +0900] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.4.0
06-20 11:23:17.749 INFO  PinpointBootStrap                   : pinpoint agentArgs:null
06-20 11:23:17.750 INFO  PinpointBootStrap                   : PinpointBootStrap.ClassLoader:null
06-20 11:23:17.750 INFO  PinpointBootStrap                   : ContextClassLoader:jdk.internal.loader.ClassLoaders$AppClassLoader@4e0e2f2a
06-20 11:23:17.755 INFO  ClassAgentPathFinder                : agentPath:/opt/agent.pinpoint/pinpoint-bootstrap-2.3.3.jar
06-20 11:23:17.755 INFO  PinpointBootStrap                   : JavaAgentPath:/opt/agent.pinpoint/pinpoint-bootstrap-2.3.3.jar
06-20 11:23:17.756 INFO  AgentDirBaseClassPathResolver       : Agent original-path:/opt/agent.pinpoint
06-20 11:23:17.757 INFO  AgentDirBaseClassPathResolver       : Agent canonical-path:/opt/agent.pinpoint
06-20 11:23:17.758 INFO  BootDir                             : found pinpoint-commons.jar path:/opt/agent.pinpoint/boot/pinpoint-commons-2.3.3.jar
06-20 11:23:17.758 INFO  BootDir                             : found pinpoint-bootstrap-core.jar path:/opt/agent.pinpoint/boot/pinpoint-bootstrap-core-2.3.3.jar
06-20 11:23:17.770 INFO  BootDir                             : found pinpoint-annotations.jar path:/opt/agent.pinpoint/boot/pinpoint-annotations-2.3.3.jar
06-20 11:23:17.770 INFO  BootDir                             : found pinpoint-bootstrap-java8.jar path:/opt/agent.pinpoint/boot/pinpoint-bootstrap-java8-2.3.3.jar
06-20 11:23:17.771 INFO  BootDir                             : found pinpoint-bootstrap-java9.jar path:/opt/agent.pinpoint/boot/pinpoint-bootstrap-java9-2.3.3.jar
06-20 11:23:17.775 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-agentsdk-async-plugin-2.3.3.jar
06-20 11:23:17.775 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-common-servlet-2.3.3.jar
06-20 11:23:17.775 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-httpclient3-plugin-2.3.3.jar
06-20 11:23:17.775 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-httpclient4-plugin-2.3.3.jar
06-20 11:23:17.775 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-ning-asynchttpclient-plugin-2.3.3.jar
06-20 11:23:17.776 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-resttemplate-plugin-2.3.3.jar
06-20 11:23:17.776 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-netty-plugin-2.3.3.jar
06-20 11:23:17.776 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-gson-plugin-2.3.3.jar
06-20 11:23:17.776 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-cubrid-jdbc-driver-plugin-2.3.3.jar
06-20 11:23:17.776 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-informix-jdbc-driver-plugin-2.3.3.jar
06-20 11:23:17.777 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-mysql-jdbc-driver-plugin-2.3.3.jar
06-20 11:23:17.777 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-mariadb-jdbc-driver-plugin-2.3.3.jar
06-20 11:23:17.777 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-mssql-jdbc-driver-plugin-2.3.3.jar
06-20 11:23:17.777 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-postgresql-jdbc-driver-plugin-2.3.3.jar
06-20 11:23:17.777 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-jtds-plugin-2.3.3.jar
06-20 11:23:17.778 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-oracle-jdbc-driver-plugin-2.3.3.jar
06-20 11:23:17.778 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-commons-dbcp-plugin-2.3.3.jar
06-20 11:23:17.778 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-commons-dbcp2-plugin-2.3.3.jar
06-20 11:23:17.778 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-hikaricp-plugin-2.3.3.jar
06-20 11:23:17.778 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-cassandra-driver-plugin-2.3.3.jar
06-20 11:23:17.779 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-jdk-http-plugin-2.3.3.jar
06-20 11:23:17.779 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-redis-plugin-2.3.3.jar
06-20 11:23:17.779 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-redis-lettuce-plugin-2.3.3.jar
06-20 11:23:17.779 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-redis-redisson-plugin-2.3.3.jar
06-20 11:23:17.779 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-tomcat-plugin-2.3.3.jar
06-20 11:23:17.780 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-json-lib-plugin-2.3.3.jar
06-20 11:23:17.780 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-jackson-plugin-2.3.3.jar
06-20 11:23:17.780 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-thrift-plugin-2.3.3.jar
06-20 11:23:17.780 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-user-plugin-2.3.3.jar
06-20 11:23:17.780 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-arcus-plugin-2.3.3.jar
06-20 11:23:17.789 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-google-httpclient-plugin-2.3.3.jar
06-20 11:23:17.789 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-grpc-plugin-2.3.3.jar
06-20 11:23:17.790 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-jetty-plugin-2.3.3.jar
06-20 11:23:17.790 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-websphere-plugin-2.3.3.jar
06-20 11:23:17.790 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-spring-plugin-2.3.3.jar
06-20 11:23:17.790 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-spring-boot-plugin-2.3.3.jar
06-20 11:23:17.790 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-spring-webflux-plugin-2.3.3.jar
06-20 11:23:17.791 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-ibatis-plugin-2.3.3.jar
06-20 11:23:17.791 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-mybatis-plugin-2.3.3.jar
06-20 11:23:17.791 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-okhttp-plugin-2.3.3.jar
06-20 11:23:17.791 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-log4j-plugin-2.3.3.jar
06-20 11:23:17.791 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-logback-plugin-2.3.3.jar
06-20 11:23:17.792 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-log4j2-plugin-2.3.3.jar
06-20 11:23:17.792 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-dubbo-plugin-2.3.3.jar
06-20 11:23:17.792 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-activemq-client-plugin-2.3.3.jar
06-20 11:23:17.792 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-cxf-plugin-2.3.3.jar
06-20 11:23:17.792 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-hystrix-plugin-2.3.3.jar
06-20 11:23:17.793 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-jboss-plugin-2.3.3.jar
06-20 11:23:17.793 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-vertx-plugin-2.3.3.jar
06-20 11:23:17.793 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-resin-plugin-2.3.3.jar
06-20 11:23:17.793 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-jsp-plugin-2.3.3.jar
06-20 11:23:17.793 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-rxjava-plugin-2.3.3.jar
06-20 11:23:17.794 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-rabbitmq-plugin-2.3.3.jar
06-20 11:23:17.794 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-weblogic-plugin-2.3.3.jar
06-20 11:23:17.795 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-akka-http-plugin-2.3.3.jar
06-20 11:23:17.797 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-undertow-plugin-2.3.3.jar
06-20 11:23:17.798 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-undertow-servlet-plugin-2.3.3.jar
06-20 11:23:17.798 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-kafka-plugin-2.3.3.jar
06-20 11:23:17.798 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-mongodb-driver-plugin-2.3.3.jar
06-20 11:23:17.798 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-fastjson-plugin-2.3.3.jar
06-20 11:23:17.798 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-druid-plugin-2.3.3.jar
06-20 11:23:17.798 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-hbase-plugin-2.3.3.jar
06-20 11:23:17.799 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-openwhisk-plugin-2.3.3.jar
06-20 11:23:17.799 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-elasticsearch-bboss-plugin-2.3.3.jar
06-20 11:23:17.799 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-apache-dubbo-plugin-2.3.3.jar
06-20 11:23:17.799 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-elasticsearch-plugin-2.3.3.jar
06-20 11:23:17.799 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-reactor-plugin-2.3.3.jar
06-20 11:23:17.800 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-reactor-netty-plugin-2.3.3.jar
06-20 11:23:17.800 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-jdk-completable-future-plugin-2.3.3.jar
06-20 11:23:17.800 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-thread-plugin-2.3.3.jar
06-20 11:23:17.800 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-process-plugin-2.3.3.jar
06-20 11:23:17.800 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-paho-mqtt-plugin-2.3.3.jar
06-20 11:23:17.800 INFO  AgentDirBaseClassPathResolver       : Found plugins:/opt/agent.pinpoint/plugin/pinpoint-rocketmq-plugin-2.3.3.jar
06-20 11:23:17.802 INFO  PinpointBootStrap                   : appendToBootstrapClassLoader:/opt/agent.pinpoint/boot/pinpoint-commons-2.3.3.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
06-20 11:23:17.803 INFO  PinpointBootStrap                   : appendToBootstrapClassLoader:/opt/agent.pinpoint/boot/pinpoint-bootstrap-core-2.3.3.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
06-20 11:23:17.803 INFO  PinpointBootStrap                   : appendToBootstrapClassLoader:/opt/agent.pinpoint/boot/pinpoint-annotations-2.3.3.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
06-20 11:23:17.803 INFO  PinpointBootStrap                   : appendToBootstrapClassLoader:/opt/agent.pinpoint/boot/pinpoint-bootstrap-java8-2.3.3.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
06-20 11:23:17.804 INFO  PinpointBootStrap                   : appendToBootstrapClassLoader:/opt/agent.pinpoint/boot/pinpoint-bootstrap-java9-2.3.3.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
06-20 11:23:17.804 INFO  PinpointBootStrap                   : parentClassLoader:BootStrapClassLoader:null
06-20 11:23:17.805 INFO  PinpointBootStrap                   : java9 module detected
06-20 11:23:17.805 INFO  PinpointBootStrap                   : ModuleBootLoader start
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] pinpoint Module id:unnamed module @45f421c
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] pinpoint Module.isNamed:false
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] pinpoint Module.name:null
06-20 11:23:17.835 INFO  IdValidator                         : check SystemProperties(-D) agentId:pinpoint_agent_consumer_1
06-20 11:23:17.837 INFO  IdValidator                         : invalid Id. SystemProperties(-D) agentId can only contain [a-zA-Z0-9], '.', '-', '_'. maxLength:24 value:pinpoint_agent_consumer_1
06-20 11:23:17.838 INFO  AgentIdResolver                     : Failed to resolve AgentId(-Dpinpoint.agentId)
06-20 11:23:17.839 INFO  AgentIdResolver                     : Auto generate AgentId='hmly2lA2TxO9PLGqdrJmlQ'
06-20 11:23:17.839 INFO  IdValidator                         : check SystemProperties(-D) applicationName:otel_consumer_java_1
06-20 11:23:17.839 INFO  AgentIdResolver                     : SystemProperties(-D) pinpoint.applicationName=otel_consumer_java_1
06-20 11:23:17.839 INFO  AgentIdResolver                     : No AgentName(-Dpinpoint.agentName) provided, it's optional!
06-20 11:23:17.849 INFO  ProfilePropertyLoader               : load default config:/opt/agent.pinpoint/pinpoint-root.config
06-20 11:23:17.858 INFO  ProfilePropertyLoader               : active profile:release
06-20 11:23:17.858 INFO  ProfilePropertyLoader               : load profile:/opt/agent.pinpoint/profiles/release/pinpoint.config
06-20 11:23:17.861 INFO  ProfilePropertyLoader               : load external config:/opt/agent.pinpoint/pinpoint-root.config
06-20 11:23:17.868 INFO  ProfilePropertyLoader               : agent root path:/opt/agent.pinpoint
06-20 11:23:17.869 INFO  ProfilePropertyLoader               : logConfig path:/opt/agent.pinpoint/profiles/release/
06-20 11:23:17.899 INFO  PinpointStarter                     : logPath:/opt/agent.pinpoint/logs
06-20 11:23:17.899 INFO  PinpointStarter                     : pinpoint version:2.3.3
06-20 11:23:17.899 INFO  PinpointStarter                     : Log directory maxbackupsize=5
06-20 11:23:17.901 INFO  PinpointStarter                     : agent JarPath:/opt/agent.pinpoint/pinpoint-bootstrap-2.3.3.jar
06-20 11:23:17.901 INFO  PinpointStarter                     : agent LibDir:/opt/agent.pinpoint/lib
06-20 11:23:17.901 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-profiler-2.3.3.jar
06-20 11:23:17.901 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-thrift-2.3.3.jar
06-20 11:23:17.901 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-rpc-2.3.3.jar
06-20 11:23:17.902 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-plugins-loader-2.3.3.jar
06-20 11:23:17.902 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-profiler-logging-2.3.3.jar
06-20 11:23:17.902 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-profiler-optional-jdk7-2.3.3.jar
06-20 11:23:17.902 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-profiler-optional-jdk8-2.3.3.jar
06-20 11:23:17.902 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-profiler-optional-jdk9-2.3.3.jar
06-20 11:23:17.902 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-grpc-2.3.3.jar
06-20 11:23:17.902 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-agent-proxy-common-2.3.3.jar
06-20 11:23:17.903 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-agent-proxy-apache-plugin-2.3.3.jar
06-20 11:23:17.903 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-agent-proxy-app-plugin-2.3.3.jar
06-20 11:23:17.903 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-agent-proxy-nginx-plugin-2.3.3.jar
06-20 11:23:17.903 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-agent-proxy-user-plugin-2.3.3.jar
06-20 11:23:17.903 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-commons-buffer-2.3.3.jar
06-20 11:23:17.903 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/pinpoint-commons-profiler-2.3.3.jar
06-20 11:23:17.903 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/log4j2.component.properties
06-20 11:23:17.909 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-3.10.6.Final.jar
06-20 11:23:17.909 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/snakeyaml-1.27.jar
06-20 11:23:17.909 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/guava-30.1-android.jar
06-20 11:23:17.909 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/failureaccess-1.0.1.jar
06-20 11:23:17.909 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/jsr305-3.0.2.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/checker-compat-qual-2.5.5.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/error_prone_annotations-2.3.4.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/j2objc-annotations-1.3.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/guice-4.2.2.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/javax.inject-1.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/aopalliance-1.0.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/asm-9.1.jar
06-20 11:23:17.910 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/asm-commons-9.1.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/asm-util-9.1.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/asm-tree-9.1.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/asm-analysis-9.1.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/libthrift-0.12.0.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/grpc-core-1.36.2.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/grpc-api-1.36.2.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/grpc-context-1.36.2.jar
06-20 11:23:17.911 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/gson-2.8.6.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/annotations-4.1.1.4.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/animal-sniffer-annotations-1.19.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/perfmark-api-0.23.0.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/grpc-netty-1.36.2.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-handler-4.1.58.Final.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-common-4.1.58.Final.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-resolver-4.1.58.Final.jar
06-20 11:23:17.912 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-buffer-4.1.58.Final.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-transport-4.1.58.Final.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-codec-4.1.58.Final.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-transport-native-epoll-4.1.58.Final-linux-x86_64.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-transport-native-unix-common-4.1.58.Final.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-codec-http2-4.1.58.Final.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-codec-http-4.1.58.Final.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/grpc-stub-1.36.2.jar
06-20 11:23:17.913 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/grpc-protobuf-1.36.2.jar
06-20 11:23:17.914 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/protobuf-java-3.12.0.jar
06-20 11:23:17.914 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/proto-google-common-protos-2.0.1.jar
06-20 11:23:17.914 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/grpc-protobuf-lite-1.36.2.jar
06-20 11:23:17.914 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/netty-tcnative-boringssl-static-2.0.36.Final.jar
06-20 11:23:17.914 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/slf4j-api-1.7.30.jar
06-20 11:23:17.914 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/log4j-core-2.12.3.jar
06-20 11:23:17.914 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/log4j-api-2.12.3.jar
06-20 11:23:17.915 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/log4j-slf4j-impl-2.12.3.jar
06-20 11:23:17.915 INFO  PinpointStarter                     : agent Lib:file:/opt/agent.pinpoint/lib/
06-20 11:23:17.915 INFO  PinpointStarter                     : agent config:/opt/agent.pinpoint/pinpoint-root.config
06-20 11:23:17.957 INFO  PinpointStarter                     : defineAgentModule
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] bootstrap unnamedModule:unnamed module @45f421c
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] platform unnamedModule:unnamed module @58af5076
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] system unnamedModule:unnamed module @1816e24a
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] defineModule classLoader: Java9ClassLoader{name=pinpoint.agent} com.navercorp.pinpoint.bootstrap.java9.classloader.Java9ClassLoader@7fd987ef
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] defineModule classLoader-unnamedModule: unnamed module @49469ffa
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] packages:[io.netty.handler.codec.rtsp, com.navercorp.pinpoint.profiler.context.compress, com.google.protobuf, com.navercorp.pinpoint.profiler.monitor.collector.activethread, com.navercorp.pinpoint.profiler.monitor.metric.cpu.ibm, org.apache.logging.log4j.core.config.plugins.validation.validators, io.grpc.protobuf.lite, io.netty.handler.codec.serialization, org.jboss.netty.buffer, io.netty.channel.socket.nio, com.navercorp.pinpoint.profiler.instrument.classreading, com.google.inject.internal.cglib.core, com.google.inject, com.navercorp.pinpoint.profiler.monitor.collector.cpu, io.netty.handler.codec.protobuf, com.navercorp.pinpoint.profiler.monitor.metric, com.navercorp.pinpoint.profiler.plugin.filter, org.jboss.netty.channel.socket.oio, io.netty.handler.flush, com.navercorp.pinpoint.profiler.context.provider.stat.cpu, org.yaml.snakeyaml.emitter, org.jboss.netty.channel.group, io.netty.handler.ssl, com.navercorp.pinpoint.profiler.monitor.metric.buffer, com.navercorp.pinpoint.thrift.dto, com.navercorp.pinpoint.profiler.instrument, org.apache.logging.log4j.core.layout, org.jboss.netty.handler.timeout, org.apache.logging.log4j.core.message, com.google.gson.internal, com.navercorp.pinpoint.profiler.shutdown, io.netty.util.internal, org.jboss.netty.handler.execution, org.apache.logging.log4j, org.jboss.netty.handler.ipfilter, org.apache.logging.log4j.core.appender.rolling.action, org.slf4j.event, org.apache.logging.log4j.core.tools.picocli, com.navercorp.pinpoint.profiler.context.provider.stat.filedescriptor, org.jboss.netty.util.internal.jzlib, org.apache.logging.log4j.core.appender.rewrite, org.jboss.netty.channel.socket.nio, com.google.common.base, io.netty.channel.internal, io.netty.handler.codec.spdy, org.jboss.netty.handler.codec.spdy, com.navercorp.pinpoint.profiler.jdbc, com.google.common.cache, com.google.errorprone.annotations, com.google.inject.internal.cglib.core.internal, com.navercorp.pinpoint.profiler.monitor.metric.cpu.oracle, org.jboss.netty.handler.codec.compression, org.jboss.netty.util, com.google.longrunning, org.apache.logging.log4j.core.appender.mom.kafka, com.navercorp.pinpoint.profiler.context.storage, org.apache.logging.log4j.core.config.yaml, com.google.rpc, org.slf4j, com.navercorp.pinpoint.profiler.monitor.metric.activethread, org.apache.logging.log4j.core.selector, com.navercorp.pinpoint.profiler.monitor.metric.uri, com.navercorp.pinpoint.grpc.client.interceptor, com.navercorp.pinpoint.profiler, com.navercorp.pinpoint.io.request, org.apache.logging.log4j.core.config.builder.impl, com.navercorp.pinpoint.profiler.sampler, org.apache.logging.slf4j, org.jboss.netty.handler.ssl, com.navercorp.pinpoint.profiler.plugin.config, io.grpc.internal, com.google.cloud.audit, org.apache.logging.log4j.core.config.plugins.validation.constraints, org.apache.logging.log4j.core.osgi, org.objectweb.asm, com.navercorp.pinpoint.rpc.server, com.google.common.math, io.netty.handler.codec.http, com.navercorp.pinpoint.profiler.context.annotation, com.navercorp.pinpoint.thrift.io, com.navercorp.pinpoint.thrift.dto.command, io.netty.handler.traffic, org.apache.logging.log4j.core.appender.rolling, io.grpc.stub.annotations, com.navercorp.pinpoint.rpc.codec, io.netty.handler.codec.http.cookie, com.google.rpc.context, com.google.api, com.google.common.base.internal, org.apache.thrift.async, io.netty.channel, org.slf4j.helpers, com.navercorp.pinpoint.profiler.context.provider.metric, com.navercorp.pinpoint.profiler.monitor.metric.filedescriptor.ibm, com.navercorp.pinpoint.grpc.client.config, org.apache.logging.log4j.core.config.plugins.convert, org.apache.logging.log4j.core.config.composite, com.navercorp.pinpoint.profiler.monitor.collector.deadlock, com.google.common.primitives, com.navercorp.pinpoint.profiler.plugin, org.codehaus.mojo.animal_sniffer, org.jboss.netty.channel.socket.http, com.google.type, com.navercorp.pinpoint.thrift.util, com.google.inject.name, com.navercorp.pinpoint.profiler.context.provider.thrift, com.navercorp.pinpoint.profiler.context.provider.sampler, com.navercorp.pinpoint.profiler.context.active, com.google.inject.matcher, org.apache.logging.log4j.core.appender, com.google.inject.internal, com.navercorp.pinpoint.profiler.instrument.classloading, com.navercorp.pinpoint.agent.plugin.proxy.apache, com.google.common.html, org.apache.logging.log4j.core.tools, org.apache.logging.log4j.core.net, org.apache.logging.log4j.core.appender.mom, com.navercorp.pinpoint.profiler.util, org.jboss.netty.handler.ssl.util, org.yaml.snakeyaml.composer, io.netty.handler.codec, com.google.gson.stream, org.jboss.netty.channel.socket, io.netty.handler.stream, io.netty.handler.codec.http.multipart, io.netty.channel.socket, org.apache.logging.log4j.core.config.status, com.navercorp.pinpoint.profiler.context.provider.stat.totalthread, com.google.common.escape, com.navercorp.pinpoint.common.buffer, com.google.thirdparty.publicsuffix, javax.annotation.meta, com.navercorp.pinpoint.profiler.context.module.config, com.navercorp.pinpoint.profiler.interceptor.scope, com.navercorp.pinpoint.rpc.common, com.navercorp.pinpoint.profiler.instrument.lambda, org.apache.thrift.scheme, org.yaml.snakeyaml.error, org.jboss.netty.container.osgi, com.navercorp.pinpoint.profiler.context.id, com.google.gson.annotations, com.navercorp.pinpoint.profiler.context.config, com.navercorp.pinpoint.rpc.packet.stream, org.yaml.snakeyaml, com.google.inject.multibindings, org.objectweb.asm.tree.analysis, io.netty.util.concurrent, io.netty.handler.codec.json, com.navercorp.pinpoint.profiler.monitor.metric.deadlock, com.navercorp.pinpoint.agent.plugin.proxy.user, com.navercorp.pinpoint.profiler.monitor.collector.loadedclass, com.navercorp.pinpoint.profiler.context.recorder, org.objectweb.asm.signature, org.apache.logging.log4j.core.util, org.apache.logging.log4j.core.script, com.navercorp.pinpoint.grpc.trace, com.navercorp.pinpoint.common.profiler.concurrent.jsr166, com.navercorp.pinpoint.rpc.server.handler, org.yaml.snakeyaml.tokens, com.navercorp.pinpoint.profiler.monitor.collector.jvmgc, com.navercorp.pinpoint.profiler.context.javamodule, com.navercorp.pinpoint.rpc, com.google.inject.internal.util, org.aopalliance.aop, io.netty.channel.unix, io.grpc, com.google.inject.util, com.navercorp.pinpoint.profiler.monitor.metric.filedescriptor.oracle, com.google.gson, com.navercorp.pinpoint.profiler.context.grpc, io.netty.handler.codec.http.websocketx.extensions.compression, org.yaml.snakeyaml.introspector, com.google.common.util.concurrent, org.jboss.netty.handler.codec.serialization, com.navercorp.pinpoint.rpc.control, org.jboss.netty.handler.codec.replay, com.google.common.eventbus, com.google.gson.internal.bind, com.navercorp.pinpoint.rpc.util, com.navercorp.pinpoint.profiler.context.provider.stat.jvmgc, com.google.common.collect, io.netty.buffer, io.netty.channel.embedded, org.apache.logging.log4j.core.async, com.navercorp.pinpoint.profiler.context.provider.stat.buffer, com.navercorp.pinpoint.profiler.metadata, android.annotation, com.navercorp.pinpoint.grpc.logging, com.navercorp.pinpoint.profiler.context, io.netty.handler.ipfilter, org.apache.logging.log4j.core.appender.routing, io.netty.handler.codec.string, io.netty.buffer.search, javax.inject, com.navercorp.pinpoint.rpc.cluster, com.navercorp.pinpoint.grpc, org.jboss.netty.channel, com.navercorp.pinpoint.profiler.context.module, com.navercorp.pinpoint.io, com.google.common.net, com.navercorp.pinpoint.profiler.monitor.metric.gc, com.navercorp.pinpoint.loader.plugins, io.netty.channel.socket.oio, com.google.common.util.concurrent.internal, com.navercorp.pinpoint.grpc.util, com.navercorp.pinpoint.io.util, com.navercorp.pinpoint.profiler.context.provider.stat.loadedclass, com.google.gson.internal.reflect, org.apache.logging.log4j.core.config.xml, com.navercorp.pinpoint.profiler.context.provider.plugin, com.navercorp.pinpoint.grpc.server.flowcontrol, org.apache.thrift.protocol, com.navercorp.pinpoint.profiler.context.recorder.proxy, org.jboss.netty.handler.codec, org.yaml.snakeyaml.extensions.compactnotation, io.netty.handler.codec.marshalling, org.yaml.snakeyaml.external.com.google.gdata.util.common.base, com.navercorp.pinpoint.profiler.monitor.metric.transaction, com.google.errorprone.annotations.concurrent, org.apache.logging.log4j.core.config.plugins, io.netty.handler.ssl.util, com.navercorp.pinpoint.profiler.context.grpc.config, com.navercorp.pinpoint.profiler.interceptor.factory, io.grpc.inprocess, com.navercorp.pinpoint.profiler.monitor.collector.totalthread, com.google.logging.type, org.apache.logging.log4j.core.config.properties, org.slf4j.spi, org.jboss.netty.container.spring, io.netty.util.internal.shaded.org.jctools.queues.atomic, com.navercorp.pinpoint.grpc.security, com.navercorp.pinpoint.grpc.channelz, org.jboss.netty.handler.codec.http.cookie, com.navercorp.pinpoint.common.profiler.trace, org.jboss.netty.handler.traffic, org.apache.thrift.transport, com.navercorp.pinpoint.common.profiler.util, org.apache.logging.log4j.core.filter, com.navercorp.pinpoint.profiler.receiver.grpc, com.google.common.xml, com.navercorp.pinpoint.loader.plugins.trace.yaml, com.navercorp.pinpoint.profiler.sender, com.navercorp.pinpoint.profiler.monitor.metric.response, io.netty.util.internal.shaded.org.jctools.queues, com.navercorp.pinpoint.profiler.context.monitor.config, com.navercorp.pinpoint.profiler.context.errorhandler, com.navercorp.pinpoint.agent.plugin.proxy.common, io.netty.handler.codec.base64, com.navercorp.pinpoint.profiler.context.provider, org.jboss.netty.handler.codec.http.websocketx, com.navercorp.pinpoint.profiler.monitor.metric.totalthread, com.navercorp.pinpoint.grpc.server, com.google.inject.internal.cglib.transform, com.google.protobuf.compiler, org.apache.logging.log4j.message, com.navercorp.pinpoint.loader.service, com.google.j2objc.annotations, org.jboss.netty.logging, io.netty.util.collection, com.google.common.graph, org.jboss.netty.handler.codec.frame, io.netty.channel.epoll, org.objectweb.asm.commons, io.netty.handler.codec.http.websocketx.extensions, io.netty.resolver, org.apache.logging.log4j.core.parser, org.apache.logging.log4j.status, com.navercorp.pinpoint.profiler.receiver, org.apache.logging.log4j.core.config, io.netty.handler.codec.http2, org.apache.thrift.annotation, com.navercorp.pinpoint.profiler.context.provider.metadata, com.navercorp.pinpoint.profiler.interceptor.registry, com.navercorp.pinpoint.profiler.monitor.metric.loadedclass, io.netty.handler.timeout, io.netty.handler.codec.bytes, org.yaml.snakeyaml.util, org.yaml.snakeyaml.resolver, io.netty.channel.oio, org.apache.logging.log4j.core.config.plugins.util, org.jboss.netty.handler.codec.marshalling, com.navercorp.pinpoint.profiler.context.provider.stat.datasource, com.google.common.hash, com.navercorp.pinpoint.profiler.context.provider.stat.transaction, org.apache.logging.log4j.core.lookup, io.netty.handler.ssl.ocsp, com.navercorp.pinpoint.common.profiler.concurrent, org.apache.logging.log4j.core.util.datetime, com.navercorp.pinpoint.loader.plugins.trace, org.yaml.snakeyaml.representer, com.navercorp.pinpoint.profiler.monitor.collector.buffer, com.navercorp.pinpoint.profiler.util.jdk, com.navercorp.pinpoint.rpc.packet, org.yaml.snakeyaml.reader, com.navercorp.pinpoint.profiler.monitor.metric.custom, com.navercorp.pinpoint.profiler.monitor.collector.transaction, com.navercorp.pinpoint.profiler.context.provider.grpc, com.navercorp.pinpoint.profiler.sender.grpc.stream, io.netty.util.internal.logging, org.apache.logging.log4j.core.time.internal, com.navercorp.pinpoint.profiler.instrument.config, io.netty.channel.nio, org.checkerframework.checker.nullness.compatqual, org.slf4j.impl, org.jboss.netty.handler.codec.http.multipart, com.navercorp.pinpoint.grpc.client, io.grpc.protobuf, com.navercorp.pinpoint.profiler.instrument.transformer, io.netty.handler.codec.compression, com.navercorp.pinpoint.profiler.monitor, org.aopalliance.intercept, com.navercorp.pinpoint.profiler.context.provider.stat.response, io.netty.channel.pool, com.google.inject.spi, org.apache.logging.log4j.core.config.plugins.visitors, com.navercorp.pinpoint.profiler.monitor.metric.memory, org.apache.logging.log4j.core.appender.nosql, com.google.gson.reflect, org.apache.logging.log4j.core.appender.db.jdbc, org.apache.logging.log4j.core, com.navercorp.pinpoint.io.header.v2, com.navercorp.pinpoint.profiler.monitor.metric.filedescriptor, com.navercorp.pinpoint.io.header.v1, com.google.inject.internal.asm, io.netty.util.internal.shaded.org.jctools.util, com.navercorp.pinpoint.profiler.context.thrift, com.google.common.reflect, com.navercorp.pinpoint.common.profiler.sql, org.yaml.snakeyaml.external.biz.base64Coder, com.navercorp.pinpoint.grpc.server.lifecycle, com.navercorp.pinpoint.profiler.logging.jul, com.navercorp.pinpoint.profiler.context.provider.stat.deadlock, org.objectweb.asm.util, io.netty.handler.address, org.yaml.snakeyaml.nodes, com.google.inject.binder, org.apache.thrift.meta_data, org.apache.logging.log4j.core.appender.mom.jeromq, com.navercorp.pinpoint.rpc.stream, com.navercorp.pinpoint.profiler.monitor.collector, org.jboss.netty.handler.stream, org.yaml.snakeyaml.constructor, org.apache.logging.log4j.core.net.ssl, com.navercorp.pinpoint.profiler.receiver.service, org.yaml.snakeyaml.serializer, org.jboss.netty.handler.queue, org.jboss.netty.bootstrap, com.navercorp.pinpoint.profiler.context.scope, com.navercorp.pinpoint.profiler.monitor.metric.datasource, org.jboss.netty.handler.codec.protobuf, com.navercorp.pinpoint.profiler.transformer, com.navercorp.pinpoint.profiler.monitor.metric.cpu, org.yaml.snakeyaml.env, org.jboss.netty.channel.local, org.apache.logging.log4j.core.impl, com.navercorp.pinpoint.common.profiler.encoding, org.yaml.snakeyaml.parser, org.apache.logging.log4j.core.config.json, com.navercorp.pinpoint.profiler.instrument.interceptor, io.netty.bootstrap, com.navercorp.pinpoint.rpc.buffer, org.jboss.netty.handler.codec.http, org.apache.logging.log4j.util, com.navercorp.pinpoint.thrift.dto.flink, io.netty.channel.local, io.perfmark, com.navercorp.pinpoint.profiler.monitor.collector.filedescriptor, com.navercorp.pinpoint.io.header, com.navercorp.pinpoint.profiler.context.monitor, com.navercorp.pinpoint.profiler.monitor.collector.response, com.navercorp.pinpoint.rpc.client, com.google.common.annotations, org.jboss.netty.container.microcontainer, org.jboss.netty.handler.codec.string, io.netty.channel.group, org.apache.logging.log4j.core.jmx, com.navercorp.pinpoint.loader.plugins.profiler, io.grpc.util, org.apache.logging.log4j.core.jackson, org.objectweb.asm.tree, com.navercorp.pinpoint.profiler.instrument.scanner, io.netty.util.internal.svm, org.jboss.netty.handler.logging, com.navercorp.pinpoint.agent.plugin.proxy.nginx, org.jboss.netty.handler.codec.oneone, com.navercorp.pinpoint.profiler.context.provider.stat.activethread, io.netty.internal.tcnative, io.netty.handler.pcap, org.apache.logging.log4j.core.config.plugins.validation, javax.annotation, io.netty.handler.codec.xml, com.google.inject.internal.cglib.proxy, org.apache.logging.log4j.core.time, javax.annotation.concurrent, com.google.geo.type, com.navercorp.pinpoint.profiler.context.monitor.metric, org.apache.logging.log4j.core.appender.db, io.netty.util, com.navercorp.pinpoint.profiler.logging, org.yaml.snakeyaml.scanner, com.navercorp.pinpoint.profiler.sender.grpc.metric, com.navercorp.pinpoint.profiler.monitor.collector.datasource, org.apache.logging.log4j.spi, io.netty.handler.codec.http.websocketx, com.navercorp.pinpoint.profiler.context.thrift.config, org.jboss.netty.handler.codec.base64, org.apache.thrift, org.apache.logging.log4j.core.config.builder.api, com.navercorp.pinpoint.profiler.context.method, org.jboss.netty.util.internal, com.google.inject.internal.cglib.reflect, io.netty.handler.logging, org.apache.logging.log4j.core.config.plugins.processor, org.jboss.netty.handler.codec.embedder, io.netty.handler.codec.http.cors, io.grpc.stub, org.apache.logging.log4j.core.pattern, org.yaml.snakeyaml.events, org.jboss.netty.handler.codec.rtsp, com.navercorp.pinpoint.agent.plugin.proxy.app, org.jboss.netty.handler.codec.socks, io.netty.handler.flow, com.navercorp.pinpoint.profiler.objectfactory, io.grpc.netty, com.navercorp.pinpoint.profiler.sender.grpc, com.google.common.io, com.google.gson.internal.bind.util, org.apache.logging.log4j.simple, org.apache.thrift.server]
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] providers:{io.grpc.ServerProvider=[io.grpc.netty.NettyServerProvider], io.grpc.NameResolverProvider=[io.grpc.internal.DnsNameResolverProvider], io.grpc.LoadBalancerProvider=[io.grpc.internal.PickFirstLoadBalancerProvider, io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider], io.grpc.ManagedChannelProvider=[io.grpc.netty.NettyChannelProvider], com.navercorp.pinpoint.agent.plugin.proxy.common.ProxyRequestMetadataProvider=[com.navercorp.pinpoint.agent.plugin.proxy.apache.ApacheRequestMetadataProvider, com.navercorp.pinpoint.agent.plugin.proxy.user.UserRequestMetadataProvider, com.navercorp.pinpoint.agent.plugin.proxy.nginx.NginxRequestMetadataProvider, com.navercorp.pinpoint.agent.plugin.proxy.app.AppRequestMetadataProvider], org.apache.logging.log4j.spi.Provider=[org.apache.logging.log4j.core.impl.Log4jProvider], org.apache.logging.log4j.util.PropertySource=[org.apache.logging.log4j.util.SystemPropertiesPropertySource, org.apache.logging.log4j.util.EnvironmentPropertySource], reactor.blockhound.integration.BlockHoundIntegration=[io.netty.util.internal.Hidden$NettyBlockHoundIntegration], com.navercorp.pinpoint.profiler.context.recorder.proxy.ProxyRequestParserProvider=[com.navercorp.pinpoint.agent.plugin.proxy.app.AppRequestParserProvider, com.navercorp.pinpoint.agent.plugin.proxy.user.UserRequestParserProvider, com.navercorp.pinpoint.agent.plugin.proxy.apache.ApacheRequestParserProvider, com.navercorp.pinpoint.agent.plugin.proxy.nginx.NginxRequestParserProvider], javax.annotation.processing.Processor=[org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor], org.apache.logging.log4j.message.ThreadDumpMessage$ThreadInfoFactory=[org.apache.logging.log4j.core.message.ExtendedThreadInfoFactory]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleBuilder] defineModule module:module pinpoint.agent
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] com.navercorp.pinpoint.test package not found
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] loadModule:java.instrument
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] loadModule:java.management
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] loadModule:jdk.management
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] loadModule:jdk.unsupported
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] load provider:Providers{services='io.grpc.NameResolverProvider', providers=[io.grpc.internal.DnsNameResolverProvider]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] discard provider:Providers{services='javax.annotation.processing.Processor', providers=[org.apache.logging.log4j.core.config.plugins.processor.PluginProcessor]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] load provider:Providers{services='com.navercorp.pinpoint.profiler.context.recorder.proxy.ProxyRequestParserProvider', providers=[com.navercorp.pinpoint.agent.plugin.proxy.app.AppRequestParserProvider, com.navercorp.pinpoint.agent.plugin.proxy.user.UserRequestParserProvider, com.navercorp.pinpoint.agent.plugin.proxy.apache.ApacheRequestParserProvider, com.navercorp.pinpoint.agent.plugin.proxy.nginx.NginxRequestParserProvider]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] discard provider:Providers{services='io.grpc.ServerProvider', providers=[io.grpc.netty.NettyServerProvider]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] discard provider:Providers{services='org.apache.logging.log4j.message.ThreadDumpMessage$ThreadInfoFactory', providers=[org.apache.logging.log4j.core.message.ExtendedThreadInfoFactory]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] load provider:Providers{services='io.grpc.ManagedChannelProvider', providers=[io.grpc.netty.NettyChannelProvider]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] discard provider:Providers{services='org.apache.logging.log4j.util.PropertySource', providers=[org.apache.logging.log4j.util.SystemPropertiesPropertySource, org.apache.logging.log4j.util.EnvironmentPropertySource]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] discard provider:Providers{services='org.apache.logging.log4j.spi.Provider', providers=[org.apache.logging.log4j.core.impl.Log4jProvider]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] load provider:Providers{services='com.navercorp.pinpoint.agent.plugin.proxy.common.ProxyRequestMetadataProvider', providers=[com.navercorp.pinpoint.agent.plugin.proxy.apache.ApacheRequestMetadataProvider, com.navercorp.pinpoint.agent.plugin.proxy.user.UserRequestMetadataProvider, com.navercorp.pinpoint.agent.plugin.proxy.nginx.NginxRequestMetadataProvider, com.navercorp.pinpoint.agent.plugin.proxy.app.AppRequestMetadataProvider]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] load provider:Providers{services='io.grpc.LoadBalancerProvider', providers=[io.grpc.internal.PickFirstLoadBalancerProvider, io.grpc.util.SecretRoundRobinLoadBalancerProvider$Provider]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] discard provider:Providers{services='reactor.blockhound.integration.BlockHoundIntegration', providers=[io.netty.util.internal.Hidden$NettyBlockHoundIntegration]}
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] loadModule:java.xml
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] loadModule:java.desktop
[com.navercorp.pinpoint.bootstrap.java9.module.ModuleSupport] loadModule:java.logging
06-20 11:23:18.044 INFO  PinpointStarter                     : pinpoint agent [com.navercorp.pinpoint.profiler.DefaultAgent] starting...
06-20 11:23:18.082 INFO  Log4j2LoggingSystem                 : logPath:file:/opt/agent.pinpoint/profiles/release/log4j2.xml
06-20 11:23:22.049 WARN  PinpointStarter                     : pinpoint start failed.
com.navercorp.pinpoint.bootstrap.BootStrapException: execute fail. Error:null
    at com.navercorp.pinpoint.bootstrap.ContextClassLoaderExecuteTemplate.execute(ContextClassLoaderExecuteTemplate.java:51)
    at com.navercorp.pinpoint.bootstrap.AgentBootLoader.boot(AgentBootLoader.java:47)
    at com.navercorp.pinpoint.bootstrap.PinpointStarter.start(PinpointStarter.java:140)
    at com.navercorp.pinpoint.bootstrap.PinpointBootStrap.premain(PinpointBootStrap.java:89)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:560)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:572)
Caused by: java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
    at com.navercorp.pinpoint.bootstrap.AgentBootLoader$1.call(AgentBootLoader.java:52)
    at com.navercorp.pinpoint.bootstrap.ContextClassLoaderExecuteTemplate.execute(ContextClassLoaderExecuteTemplate.java:41)
    ... 7 more
Caused by: java.lang.IllegalAccessError: class io.opentelemetry.javaagent.instrumentation.netty.v4.common.FutureListenerWrappers (in unnamed module @0x49469ffa) cannot access class io.netty.util.concurrent.GenericFutureListener (in module pinpoint.agent) because module pinpoint.agent does not export io.netty.util.concurrent to unnamed module @0x49469ffa
    at pinpoint.agent//io.opentelemetry.javaagent.instrumentation.netty.v4.common.FutureListenerWrappers.<clinit>(FutureListenerWrappers.java:32)
    at pinpoint.agent/pinpoint.agent/io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:177)
    at pinpoint.agent/pinpoint.agent/io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35)
    at pinpoint.agent/pinpoint.agent/io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:123)
    at pinpoint.agent/pinpoint.agent/io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
    at pinpoint.agent/pinpoint.agent/io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52)
    at pinpoint.agent/pinpoint.agent/io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:116)
    at pinpoint.agent/pinpoint.agent/io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:80)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.grpc.client.ChannelTypeFactory$EpollChannelType.newEventLoopGroup(ChannelTypeFactory.java:46)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.grpc.client.DefaultChannelFactory.<init>(DefaultChannelFactory.java:101)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.grpc.client.DefaultChannelFactoryBuilder.build(DefaultChannelFactoryBuilder.java:107)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.profiler.context.provider.grpc.SpanGrpcDataSenderProvider.get(SpanGrpcDataSenderProvider.java:99)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.profiler.context.provider.grpc.SpanGrpcDataSenderProvider.get(SpanGrpcDataSenderProvider.java:55)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:59)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ExposedKeyFactory.get(ExposedKeyFactory.java:57)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:60)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182)
    at pinpoint.agent/pinpoint.agent/com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
    at pinpoint.agent/pinpoint.agent/com.google.inject.Guice.createInjector(Guice.java:87)
    at pinpoint.agent/pinpoint.agent/com.google.inject.Guice.createInjector(Guice.java:78)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.profiler.context.module.DefaultApplicationContext.<init>(DefaultApplicationContext.java:96)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.profiler.DefaultAgent.newApplicationContext(DefaultAgent.java:111)
    at pinpoint.agent/pinpoint.agent/com.navercorp.pinpoint.profiler.DefaultAgent.<init>(DefaultAgent.java:87)
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
    ... 11 more
*****************************************************************************
* Pinpoint Agent load failure
*****************************************************************************
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2024-06-20 11:23:22 [main] INFO  init.Banner -

Additional context

script.sh (Build Script)

#!/bin/bash

# JDK 21 system property setting
export JAVA_HOME=/path/to/java-21/jdk-21.0.2
export PATH=$JAVA_HOME/bin:$PATH

### Log
CURRENT_TIME="$(date "+%Y-%m-%d_%H-%M-%S")"
LOG_FILE="/path/to/logs/server2-${CURRENT_TIME}.log"

### Otel
OTEL_AGENT_PATH="/opt/Path..
OTEL_CONFIG_PATH="/opt/Path.."

### Pinpoint
PINPOINT_AGENT_PATH="/path/to/pinpoint-agent/pinpoint-bootstrap-2.3.3.jar"
PINPOINT_AGENT_ID="pinpoint_agent"
PINPOINT_APPLICATION_NAME="collector_java"
PINPOINT_CONFIG="/path/to/pinpoint-config/pinpoint-root.config"

# nohup java -Dconfig=root-config-2.txt -jar collector_java.jar > ${LOG_FILE} 2>&1 &
nohup java \
-javaagent:$OTEL_AGENT_PATH \
-Dotel.javaagent.configuration-file=$OTEL_CONFIG_PATH \
-Dconfig=root-config.txt \
-javaagent:$PINPOINT_AGENT_PATH \
-Dpinpoint.agentId=$PINPOINT_AGENT_ID \
-Dpinpoint.applicationName=$PINPOINT_APPLICATION_NAME \
-Dpinpoint.config=$PINPOINT_CONFIG \
-jar collector_java.jar > ${LOG_FILE} 2>&1 &
emeroad commented 5 months ago

Thank you for your reporting. We will look into the above issue.