search

apache / skywalking

APM, Application Performance Monitoring System
https://skywalking.apache.org/
Apache License 2.0
23.82k stars 6.52k forks source link

java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" #4949

Closed qingpu closed 4 years ago

qingpu commented 4 years ago

skywalking 8.0.0, jdk1.8 java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/apache/catalina/core/StandardHostValve at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.catalina.core.StandardHost.(StandardHost.java:65) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at org.apache.tomcat.util.digester.ObjectCreateRule.begin(ObjectCreateRule.java:145) at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1303) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1359) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2784) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643) at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1576) at org.apache.catalina.startup.Catalina.load(Catalina.java:634) at org.apache.catalina.startup.Catalina.load(Catalina.java:685) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:253) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:427)

wu-sheng commented 4 years ago

Read document. We don't support jdk 7 since SkyWalking 7.

qingpu commented 4 years ago

jdk18

------------------ 原始邮件 ------------------ 发件人: "吴晟 Wu Sheng"<notifications@github.com>; 发送时间: 2020年6月19日(星期五) 晚上7:17 收件人: "apache/skywalking"<skywalking@noreply.github.com>; 抄送: "普清"<707378620@qq.com>;"Author"<author@noreply.github.com>; 主题: Re: [apache/skywalking] java.lang.ClassFormatError: Duplicate interface name (#4949)

Read document. We don't support jdk 7 since SkyWalking 7.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

wu-sheng commented 4 years ago

What JDK are you using?

wu-sheng commented 4 years ago

Do you somehow manipulate the classloader? Or using some other agent tech?

liangjun-jiang commented 4 years ago

I am facing the same issue. A sample application to replicate this issue can be found: https://github.com/linkedin/datahub Of course, it is a big project so it would take some effort to understand it. But if we just use one of its module: https://github.com/linkedin/datahub/tree/master/metadata-jobs/mce-consumer-job and run this module's jar file. I am seeing the same error reported. environment: jdk 1.8.0 skywalking version: 8.1.0

o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.context.annotation.internalConfigurationAnnotationProcessor': Instantiation of bean failed; nested exception is java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/springframework/beans/BeanWrapperImpl
11:01:55.512 [main] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.context.annotation.internalConfigurationAnnotationProcessor': Instantiation of bean failed; nested exception is java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/springframework/beans/BeanWrapperImpl
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1318)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1213)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:207)
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:90)
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533)
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758)
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750)
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315)
    at com.linkedin.metadata.kafka.MceConsumerApplication.main(MceConsumerApplication.java:23)
Caused by: java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/springframework/beans/BeanWrapperImpl
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
Caused by: java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/springframework/beans/BeanWrapperImpl
wu-sheng commented 4 years ago

Could you open a new issue? And answer the question, do you have a custom class loader or code generation tech to manipulate somethings related to the class codes?

wu-sheng commented 4 years ago

I am assuming this is related to the datahub project only.

liangjun-jiang commented 4 years ago

As you can see, other people is reporting the same issue so it is probably not just about datahub project.

I am assuming this is related to the datahub project only.

liangjun-jiang commented 4 years ago

I am not the original creator of this datahub project so it will take some time to answer your questions regarding custom class loader. Yes, this datahub project does use code generation

Could you open a new issue? And answer the question, do you have a custom class loader or code generation tech to manipulate somethings related to the class codes?

wu-sheng commented 4 years ago

Could you try this

agent.is_cache_enhanced_class If true, SkyWalking agent will cache all instrumented classes files to memory or disk files (decided by class cache mode), allow another java agent to enhance those classes that enhanced by SkyWalking agent. To use so

at this doc, https://github.com/apache/skywalking/blob/master/docs/en/setup/service-agent/java-agent/README.md

This could help on keeping the agent only manipulate codes once. But if they did more tricky things in classloader level, we need to know what they are.

liangjun-jiang commented 4 years ago

It actually solved the problem. Besides this is_cache_enhanced_class, I also need to turn on the other two as follows. and seems it doesn't matter whether I set either true or false for is_cache_enhanced_class. either one works.

# If true, SkyWalking agent will save all instrumented classes files in `/debugging` folder.
# SkyWalking team may ask for these files in order to resolve compatible problem.
agent.is_open_debugging_class = ${SW_AGENT_OPEN_DEBUG:true}

# If true, SkyWalking agent will cache all instrumented classes files to memory or disk files (decided by class cache mode),
# allow other javaagent to enhance those classes that enhanced by SkyWalking agent.
agent.is_cache_enhanced_class = ${SW_AGENT_CACHE_CLASS:true}

# The instrumented classes cache mode: MEMORY or FILE
# MEMORY: cache class bytes to memory, if instrumented classes is too many or too large, it may take up more memory
# FILE: cache class bytes in `/class-cache` folder, automatically clean up cached class files when the application exits
agent.class_cache_mode = ${SW_AGENT_CLASS_CACHE_MODE:MEMORY}
wu-sheng commented 4 years ago

I am a little confused about seems it doesn't matter whether I set either true or false. is_cache_enhanced_class default is false, which is as same as nothing changed. Only true activated the cache mechanism to avoid duplicated enhancement.

This is the only place we use this config https://github.com/apache/skywalking/blob/master/apm-sniffer/apm-agent/src/main/java/org/apache/skywalking/apm/agent/SkyWalkingAgent.java#L114-L121

wu-sheng commented 4 years ago

agent.is_open_debugging_class = ${SW_AGENT_OPEN_DEBUG:true}

This is for debug only, there is nothing related to the logic.

agent.class_cache_mode = ${SW_AGENT_CLASS_CACHE_MODE:MEMORY}

The default value is this, configing it or not should be no difference.

liangjun-jiang commented 4 years ago

I totally believe you. but, if I don't comment out these two. I still saw the error. I leave those to you to figure out why those matter.

# If true, SkyWalking agent will save all instrumented classes files in `/debugging` folder.
# SkyWalking team may ask for these files in order to resolve compatible problem.
agent.is_open_debugging_class = ${SW_AGENT_OPEN_DEBUG:true}

# The instrumented classes cache mode: MEMORY or FILE
# MEMORY: cache class bytes to memory, if instrumented classes is too many or too large, it may take up more memory
# FILE: cache class bytes in `/class-cache` folder, automatically clean up cached class files when the application exits
agent.class_cache_mode = ${SW_AGENT_CLASS_CACHE_MODE:MEMORY}

Likewise, I set the following as true, or I set it as false. Either way, I won't see the error again. 

agent.is_cache_enhanced_class = ${SW_AGENT_CACHE_CLASS:true}

I wish I could only need to care this, but it alone doesn't solve the problem

agent.is_cache_enhanced_class = ${SW_AGENT_CACHE_CLASS:true}
wu-sheng commented 4 years ago

Could you try only opening this only?

# If true, SkyWalking agent will save all instrumented classes files in `/debugging` folder.
# SkyWalking team may ask for these files in order to resolve compatible problem.
agent.is_open_debugging_class = ${SW_AGENT_OPEN_DEBUG:true}
liangjun-jiang commented 4 years ago

I did. I still saw error. by the way, I watched your talk on bibilili. great knowledge sharing.

wu-sheng commented 4 years ago

I did. I still saw error.

OK, this is really interesting. SW_AGENT_OPEN_DEBUG would open ByteBuddy's TypeValidation, and SW_AGENT_CACHE_CLASS avoids the duplicated enhancement, but still, I am not sure why their combination could make your case back to work :)

I will close this issue, for now, if you have more time no the DataHub source codes, we may be able to dig deeper. OK?

by the way, I watched your talk on bibilili. great knowledge sharing.

Thanks. Glad it helps. You are still at Austin, right?(from your GitHub profile). Are the video and network speed OK of Bilibili? I was not expected people watching this from US. :)

liangjun-jiang commented 4 years ago

not sure about

if you have more time no the DataHub source codes, we may be able to dig deeper. OK? The datahub code is fairly easier to build, actually. I do plan to write a blog about integrating Skywalking with datahub.

bibilili is pretty smooth when I watch it. I used some distributed tracer tools while working for Alipay two years ago I like a lot. I always want to find something like that. Skywalking is very close.

I did. I still saw error.

OK, this is really interesting. SW_AGENT_OPEN_DEBUG would open ByteBuddy's TypeValidation, and SW_AGENT_CACHE_CLASS avoids the duplicated enhancement, but still, I am not sure why their combination could make your case back to work :)

I will close this issue, for now, if you have more time no the DataHub source codes, we may be able to dig deeper. OK?

by the way, I watched your talk on bibilili. great knowledge sharing.

Thanks. Glad it helps. You are still at Austin, right?(from your GitHub profile). Are the video and network speed OK of Bilibili? I was not expected people watching this from US. :)

enderwsp commented 3 years ago

@wu-sheng hi,i also got stuck on this error, i used skywalking v8.2,jdk8,springboot2.3.4 on windows system

i try to set agent.is_cache_enhanced_class = ${SW_AGENT_CACHE_CLASS:true} but this error also occurs

stack log below:

19:35:56: Executing task 'TestcrossthreadApplication.main()'...

Task :compileJava UP-TO-DATE Task :processResources UP-TO-DATE Task :classes UP-TO-DATE

Task :TestcrossthreadApplication.main() DEBUG 2020-12-02 19:35:57:929 main AgentPackagePath : The beacon class location is jar:file:/D:/afiles/skywalking/apache-skywalking-apm-bin/agent/skywalking-agent.jar!/org/apache/skywalking/apm/agent/core/boot/AgentPackagePath.class. INFO 2020-12-02 19:35:57:929 main SnifferConfigInitializer : Config file found in D:\afiles\skywalking\apache-skywalking-apm-bin\agent\config\agent.config.

. _ _ /\ / '_ () \ \ \ \ ( ( )\ | ' | '| | ' \/ ` | \ \ \ \ \/ _)| |)| | | | | || (| | ) ) ) ) ' |__| .|| ||| |\, | / / / / =========|_|==============|__/=//// :: Spring Boot :: (v2.3.4.RELEASE)

2020-12-02 19:36:11.548 INFO 10016 --- [ main] c.e.demo.TestcrossthreadApplication : Starting TestcrossthreadApplication on WIN-20200808WOT with PID 10016 (D:\afiles\git\maven\testcrossthread\build\classes\java\main started by Administrator in D:\afiles\git\maven\testcrossthread) 2020-12-02 19:36:11.564 INFO 10016 --- [ main] c.e.demo.TestcrossthreadApplication : No active profile set, falling back to default profiles: default 2020-12-02 19:36:12.060 WARN 10016 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.context.annotation.internalConfigurationAnnotationProcessor': Instantiation of bean failed; nested exception is java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/springframework/beans/BeanWrapperImpl 2020-12-02 19:36:12.622 ERROR 10016 --- [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.context.annotation.internalConfigurationAnnotationProcessor': Instantiation of bean failed; nested exception is java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/springframework/beans/BeanWrapperImpl at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1318) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1213) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:207) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:90) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:707) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:533) ~[spring-context-5.2.9.RELEASE.jar:5.2.9.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) [spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) [spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.4.RELEASE.jar:2.3.4.RELEASE] at com.example.demo.TestcrossthreadApplication.main(TestcrossthreadApplication.java:22) [main/:na] Caused by: java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file org/springframework/beans/BeanWrapperImpl at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_202] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[na:1.8.0_202] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.8.0_202] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[na:1.8.0_202] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[na:1.8.0_202] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[na:1.8.0_202] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[na:1.8.0_202] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_202] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[na:1.8.0_202] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_202] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[na:1.8.0_202] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_202] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1312) ~[spring-beans-5.2.9.RELEASE.jar:5.2.9.RELEASE] ... 18 common frames omitted

Task :TestcrossthreadApplication.main() FAILED 3 actionable tasks: 1 executed, 2 up-to-date

FAILURE: Build failed with an exception.

BUILD FAILED in 16s 19:36:13: Task execution finished 'TestcrossthreadApplication.main()'.

waiting 4 reply , no hurry

ZHANG-GXJ commented 3 years ago

Could you try only opening this only?

# If true, SkyWalking agent will save all instrumented classes files in `/debugging` folder.
# SkyWalking team may ask for these files in order to resolve compatible problem.
agent.is_open_debugging_class = ${SW_AGENT_OPEN_DEBUG:true}

I met the same error. And it works by only opening this switch. Besides, my skywalking version is 8.1 and there can not find other two configs mentioned by liangjun-jiang. But what confused me is the same project can run successfully in my colleague's computer.