search

sofastack / sofa-ark

SOFAArk is a light-weight，java based classloader isolation framework.
https://www.sofastack.tech/projects/sofa-boot/sofa-ark-readme/
Apache License 2.0
1.57k stars 500 forks source link

Skywalking启动失败,ClassNotFoundException #571

Closed l10178 closed 11 months ago

l10178 commented 2 years ago

Describe the bug

一个Spring Boot + sofa-ark项目，集成skywalking失败，报错日志如下。

ERROR 2022-09-27 17:47:34:258 main SkyWalkingAgent : Enhance class ch.qos.logback.classic.spi.LoggingEvent error. 
org.apache.skywalking.apm.agent.core.plugin.PluginException: Can't create InstanceConstructorInterceptorV2.
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ConstructorInter.<init>(ConstructorInter.java:50)
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhanceInstance(ClassEnhancePluginDefine.java:123)
    at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.enhance(AbstractClassEnhancePluginDefine.java:116)
    at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.define(AbstractClassEnhancePluginDefine.java:96)
    at org.apache.skywalking.apm.agent.SkyWalkingAgent$Transformer.transform(SkyWalkingAgent.java:163)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11360)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11298)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11016)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:11763)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:11702)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11240)
    at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:555)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at com.alipay.sofa.ark.bootstrap.ContainerClassLoader.loadClass(ContainerClassLoader.java:57)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
    at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
    at ch.qos.logback.classic.Logger.info(Logger.java:579)
    at com.alipay.sofa.ark.common.log.ArkLogger.info(ArkLogger.java:168)
    at com.alipay.sofa.ark.container.service.ArkServiceContainer.start(ArkServiceContainer.java:75)
    at com.alipay.sofa.ark.container.ArkContainer.start(ArkContainer.java:147)
    at com.alipay.sofa.ark.container.ArkContainer.main(ArkContainer.java:110)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.alipay.sofa.ark.bootstrap.MainMethodRunner.run(MainMethodRunner.java:48)
    at com.alipay.sofa.ark.bootstrap.AbstractLauncher.launch(AbstractLauncher.java:104)
    at com.alipay.sofa.ark.bootstrap.AbstractLauncher.launch(AbstractLauncher.java:73)
    at com.alipay.sofa.ark.support.startup.EmbedSofaArkBootstrap.launch(EmbedSofaArkBootstrap.java:58)
    at com.alipay.sofa.ark.springboot.listener.ArkApplicationStartListener.startUpArkEmbed(ArkApplicationStartListener.java:92)
    at com.alipay.sofa.ark.springboot.listener.ArkApplicationStartListener.onApplicationEvent(ArkApplicationStartListener.java:53)
    at com.alipay.sofa.ark.springboot.listener.ArkApplicationStartListener.onApplicationEvent(ArkApplicationStartListener.java:39)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131)
    at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85)
    at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
    at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114)
    at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65)
    at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:344)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:302)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
    at com.test.MyApplication.main(MyApplication.java:10)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.lang.NoClassDefFoundError: org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/InstanceConstructorInterceptor
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:878)
    at org.apache.skywalking.apm.agent.core.plugin.loader.AgentClassLoader.findClass(AgentClassLoader.java:112)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:398)
    at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:71)
    at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ConstructorInter.<init>(ConstructorInter.java:48)
    ... 67 more
Caused by: java.lang.ClassNotFoundException: Can't find org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstanceConstructorInterceptor
    at org.apache.skywalking.apm.agent.core.plugin.loader.AgentClassLoader.findClass(AgentClassLoader.java:117)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
    ... 77 more

Spring Boot项目启动方式如下：

java  -javaagent:/opt/spring-boot/skywalking-agent/skywalking-agent.jar -Dsofa.ark.embed.enable=true -Dcom.alipay.sofa.ark.master.biz=xxxxx  -jar my.jar

Spring Boot pom.xml关于sofa-ark的引入如下：


   <dependency>
      <groupId>com.alipay.sofa</groupId>
      <artifactId>runtime-sofa-boot-starter</artifactId>
      <exclusions>
        <exclusion>
          <groupId>com.alipay.sofa</groupId>
          <artifactId>sofa-ark-spi</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>com.alipay.sofa</groupId>
      <artifactId>healthcheck-sofa-boot-starter</artifactId>
      <version>${sofa.boot.version}</version>
    </dependency>

    <dependency>
      <groupId>com.alipay.sofa</groupId>
      <artifactId>sofa-ark-springboot-starter</artifactId>
    </dependency>
    <dependency>
      <groupId>com.alipay.sofa</groupId>
      <artifactId>web-ark-plugin</artifactId>
      <exclusions>
        <exclusion>
          <groupId>com.alipay.sofa</groupId>
          <artifactId>log-sofa-boot-starter</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

    <dependency>
      <groupId>com.alipay.sofa</groupId>
      <artifactId>sofa-ark-all</artifactId>
    </dependency>
    <dependency>
      <groupId>com.alipay.sofa</groupId>
      <artifactId>sofa-ark-api</artifactId>
      <version>${sofa.ark.version}</version>
    </dependency>

  <build>
    <plugins>
      <!--  package the sofa ark master biz    -->
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <version>${spring-boot.version}</version>
        <executions>
          <execution>
            <id>package</id>
            <goals>
              <goal>repackage</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Skywalking集成插件列表：

/skywalking-agent/plugins/apm-dubbo-2.7.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-dubbo-3.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-dubbo-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-feign-default-http-9.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-grpc-1.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-httpClient-4.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-httpclient-3.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-httpclient-5.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-httpclient-commons-8.10.0.jar 
/skywalking-agent/plugins/apm-jdbc-commons-8.10.0.jar 
/skywalking-agent/plugins/apm-mongodb-2.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-mongodb-3.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-mongodb-4.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-mysql-5.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-mysql-6.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-mysql-8.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-mysql-commons-8.10.0.jar 
/skywalking-agent/plugins/apm-okhttp-3.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-okhttp-4.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-okhttp-common-8.10.0.jar 
/skywalking-agent/plugins/apm-postgresql-8.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-rocketmq-3.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/apm-rocketmq-4.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/dubbo-2.7.x-conflict-patch-8.10.0.jar 
/skywalking-agent/plugins/dubbo-3.x-conflict-patch-8.10.0.jar 
/skywalking-agent/plugins/dubbo-conflict-patch-8.10.0.jar 
/skywalking-agent/plugins/okhttp-2.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/resteasy-server-3.x-plugin-8.10.0.jar 
/skywalking-agent/plugins/tomcat-7.x-8.x-plugin-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-kafka-activation-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-log4j-1.x-activation-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-log4j-2.x-activation-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-logback-1.x-activation-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-logging-common-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-meter-activation-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-opentracing-activation-8.10.0.jar 
/skywalking-agent/activations/apm-toolkit-trace-activation-8.10.0.jar

Environment

从源代码里看sofa自定义了AgentClassloader，但是实际上还没有走到，猜测是embed模式获取URLClassPath无法获取到java agent的包。

gaosaroma commented 2 years ago

embed 模式下，如果biz需要使用到某个依赖： 1.要么由biz自身的bizclassloader进行加载这个依赖的类。使用方式是：biz的pom中直接引入该依赖，并且该scope是compile。检查biz包中是否打包了这个依赖 2.要么由master-biz的appclassloader加载这个依赖的类。使用方式是：biz的pom中直接引入该依赖，并且该scope是provided，声明declaredMode。master-biz中的pom引入该依赖。当bizClassLoader没有找到这个类时，会去查找 master-biz的appclassloader加载这个类。

l10178 commented 2 years ago

这个类在skywalking的agent jar里，正常来讲不管是biz还是master-biz都不应该引入agent jar。 目前看这个失败的原因是Ark Container的com.alipay.sofa.ark.bootstrap.ContainerClassLoader加载失败，master-biz的classloader和普通biz的BizClassloader都没有问题。

gaosaroma commented 1 year ago

理解了。欢迎提 PR 适配 skywalking，共同建设 sofaserverless 社区～

Zwl0113 commented 1 year ago

@l10178 这个有解决方案吗 大佬

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue or help wanted) or other activity occurs. Thank you for your contributions.

lvjing2 commented 11 months ago

已修复 #748