search

alibaba / transmittable-thread-local

📌 a missing Java std lib(simple & 0-dependency) for framework/middleware, provide an enhanced InheritableThreadLocal that transmits values between threads even using thread pooling components.
https://github.com/alibaba/transmittable-thread-local
Apache License 2.0
7.66k stars 1.69k forks source link

agent方式使用TTL，jdk从8升到11之后项目启动报错 类重复加载： java.lang.LinkageError: loader java.net.URLClassLoader attempted duplicate class #518

Closed JiaYao9701 closed 1 year ago

JiaYao9701 commented 1 year ago

完整报错如下：

java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.taobao.pandora.boot.loader.SarLoaderUtils.invokeStart(SarLoaderUtils.java:258)
    at com.taobao.pandora.boot.loader.SarLoaderUtils.getClassCache(SarLoaderUtils.java:216)
    at com.taobao.pandora.boot.loader.SarLauncher.loadSar(SarLauncher.java:92)
    at com.taobao.pandora.boot.loader.SarLauncher.createClassLoader(SarLauncher.java:64)
    at com.taobao.pandora.boot.loader.Launcher.createClassLoader(Launcher.java:64)
    at com.taobao.pandora.boot.loader.Launcher.launch(Launcher.java:49)
    at com.taobao.pandora.boot.loader.SarLauncher.main(SarLauncher.java:171)
Caused by: java.lang.RuntimeException: com.taobao.pandora.common.exception.PandoraException: java.lang.LinkageError: loader java.net.URLClassLoader @3098cf3b attempted duplicate class definition for com.taobao.pandora.loader.jar.JarURLConnection$1. (com.taobao.pandora.loader.jar.JarURLConnection$1 is in unnamed module of loader java.net.URLClassLoader @3098cf3b, parent loader 'platform')
    at com.taobao.pandora.service.pipeline.Pipeline.execute(Pipeline.java:83)
    at com.taobao.pandora.PandoraContainer.start(PandoraContainer.java:142)
    ... 11 more
Caused by: com.taobao.pandora.common.exception.PandoraException: java.lang.LinkageError: loader java.net.URLClassLoader @3098cf3b attempted duplicate class definition for com.taobao.pandora.loader.jar.JarURLConnection$1. (com.taobao.pandora.loader.jar.JarURLConnection$1 is in unnamed module of loader java.net.URLClassLoader @3098cf3b, parent loader 'platform')
    at com.taobao.pandora.service.initializer.InitializerChecker.checkModule(InitializerChecker.java:80)
    at com.taobao.pandora.stage.InitializerCheck.stepIn(InitializerCheck.java:35)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:86)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.Pipeline.execute(Pipeline.java:78)
    ... 12 more
Caused by: java.lang.LinkageError: loader java.net.URLClassLoader @3098cf3b attempted duplicate class definition for com.taobao.pandora.loader.jar.JarURLConnection$1. (com.taobao.pandora.loader.jar.JarURLConnection$1 is in unnamed module of loader java.net.URLClassLoader @3098cf3b, parent loader 'platform')
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1133)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/java.net.URLClassLoader.defineClassInternal(URLClassLoader.java:648)
    at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:573)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:477)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:464)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.net.URLClassLoader.findClassInternal(URLClassLoader.java:463)
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:455)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:626)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:558)
    at com.taobao.pandora.loader.jar.Handler.setUseFastConnectionExceptions(Handler.java:308)
    at com.taobao.pandora.service.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:73)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:558)
    at com.taobao.pandora.service.initializer.InitializerChecker.checkModule(InitializerChecker.java:75)

通过打印每个类被define时的线程调用栈，发现是TTL的问题：

想问下大佬有没有什么好的解决方案

oldratlee commented 1 year ago

@JiaYao9701 照着你的说明，我也理了一遍前后的流程：

  1. pandora.ModuleClassLoader load classes
    • pandora.ModuleClassLoader need class pandora.JarURLConnection$1
  2. load class pandora.JarURLConnection$1(first) 🏋️‍♀️
  3. tigger TTL Transformer when loading class pandora.JarURLConnection$1
    • TTL Transformer need load class files by class loader(pandora.ModuleClassLoader)
    • pandora.ModuleClassLoader need class pandora.JarURLConnection$1
    • but class pandora.JarURLConnection$1 is loading/not completed
    • so load it (again「叕」) 🏋️‍♀️
    • ==> cause LinkageError: attempted duplicate class definition 💥

因为类加载会触发(TTL) Transformer，这个应该不能控制以跳过transform（由Java Agent机制决定）。

所以对于TTL库，应该需要支持配置TTL Transformer可以跳过哪些类的transform。 @JiaYao9701

TTL Transformer已经有一些跳过transform的判断：

https://github.com/alibaba/transmittable-thread-local/blob/4655ffc1168e249c534097e0d251a1f4faf487c1/src/main/java/com/alibaba/ttl/threadpool/agent/TtlTransformer.java#L49-L57

功能 🍀：TTL Transformer支持配置以跳过指定类或包的transform。 这个功能我先思考然后实现一下。

想问有没有什么好的解决方案

@JiaYao9701 对于这个你具体的问题，快速安全的解决方案 是 新加跳过判断，比如：

     final String className = toClassName(classFile);
     if (isClassUnderPackage(className, "com.alibaba.ttl")) return NO_TRANSFORM;
     if (isClassUnderPackage(className, "java.lang")) return NO_TRANSFORM;
+    if (isClassUnderPackage(className, "com.taobao.pandora.loader")) return NO_TRANSFORM;
+    if (isClassUnderPackage(className, "com.taobao.pandora.service.loader")) return NO_TRANSFORM;

修改后你可以自己打个TTL包，先快速安全地解决问题 💕 @JiaYao9701 注意：推荐使用最新稳定v2.x版本的Tag做 hot patch。

JiaYao9701 commented 1 year ago

按照这个方式：

final String className = toClassName(classFile);
     if (isClassUnderPackage(className, "com.alibaba.ttl")) return NO_TRANSFORM;
     if (isClassUnderPackage(className, "java.lang")) return NO_TRANSFORM;
+    if (isClassUnderPackage(className, "com.taobao.pandora.loader")) return NO_TRANSFORM;
+    if (isClassUnderPackage(className, "com.taobao.pandora.service.loader")) return NO_TRANSFORM;

我将TTL 2.14.2重新打包并且验证了一下，出现了新的问题：

2023-06-15 15:46:29.713 SEVERE [main] TtlTransformer: Fail to transform class com/taobao/spas/sdk/common/cache/ConcurrentLRUCache$1, cause: com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:474)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:440)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.Utils.doTryFinallyForMethod(Utils.java:89)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.Utils.doTryFinallyForMethod(Utils.java:61)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.TtlTimerTaskTransformlet.updateTimerTaskClass(TtlTimerTaskTransformlet.java:75)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.TtlTimerTaskTransformlet.doTransform(TtlTimerTaskTransformlet.java:54)
    at com.alibaba.ttl.threadpool.agent.TtlTransformer.transform(TtlTransformer.java:64)
    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1133)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/java.net.URLClassLoader.defineClassInternal(URLClassLoader.java:665)
    at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:593)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:477)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:464)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.net.URLClassLoader.findClassInternal(URLClassLoader.java:463)
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:455)
    at com.taobao.pandora.service.loader.ModuleClassLoader.resolveClassPath(ModuleClassLoader.java:289)
    at com.taobao.pandora.service.loader.ModuleClassLoader.loadClassInternal(ModuleClassLoader.java:126)
    at com.taobao.pandora.service.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:75)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:558)
    at com.taobao.pandora.service.sharedclass.ClassExporter.scanFromExportIndex(ClassExporter.java:151)
    at com.taobao.pandora.service.sharedclass.ClassExporter.exportClasses(ClassExporter.java:76)
    at com.taobao.pandora.stage.ExportClass2Cache.stepIn(ExportClass2Cache.java:30)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:86)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.Pipeline.execute(Pipeline.java:78)
    at com.taobao.pandora.PandoraContainer.start(PandoraContainer.java:142)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.taobao.pandora.boot.loader.SarLoaderUtils.invokeStart(SarLoaderUtils.java:258)
    at com.taobao.pandora.boot.loader.SarLoaderUtils.getClassCache(SarLoaderUtils.java:216)
    at com.taobao.pandora.boot.loader.SarLauncher.loadSar(SarLauncher.java:92)
    at com.taobao.pandora.boot.loader.SarLauncher.createClassLoader(SarLauncher.java:64)
    at com.taobao.pandora.boot.loader.Launcher.createClassLoader(Launcher.java:64)
    at com.taobao.pandora.boot.loader.Launcher.launch(Launcher.java:49)
    at com.taobao.pandora.boot.loader.SarLauncher.main(SarLauncher.java:171)
Caused by: compile error: no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.MemberResolver.searchImports(MemberResolver.java:479)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:422)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:329)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.TypeChecker.atCallExpr(TypeChecker.java:711)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.JvstTypeChecker.atCallExpr(JvstTypeChecker.java:170)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.CallExpr.accept(CallExpr.java:49)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.doTypeCheck(CodeGen.java:266)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atDeclarator(CodeGen.java:819)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.Declarator.accept(Declarator.java:103)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atStmnt(CodeGen.java:381)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.Stmnt.accept(Stmnt.java:53)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atStmnt(CodeGen.java:381)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.Stmnt.accept(Stmnt.java:53)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atMethodBody(CodeGen.java:321)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.Javac.compileBody(Javac.java:228)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:466)
    ... 44 more
2023-06-15 15:46:29.812 SEVERE [main] TtlTransformer: Fail to transform class com/taobao/vipserver/client/backups/FailoverReactor$DiskFileWriter, cause: com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:474)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:440)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.Utils.doTryFinallyForMethod(Utils.java:89)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.Utils.doTryFinallyForMethod(Utils.java:61)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.TtlTimerTaskTransformlet.updateTimerTaskClass(TtlTimerTaskTransformlet.java:75)
    at com.alibaba.ttl.threadpool.agent.internal.transformlet.impl.TtlTimerTaskTransformlet.doTransform(TtlTimerTaskTransformlet.java:54)
    at com.alibaba.ttl.threadpool.agent.TtlTransformer.transform(TtlTransformer.java:64)
    at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
    at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
    at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
    at java.base/java.lang.ClassLoader.defineClass1(Native Method)
    at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1133)
    at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
    at java.base/java.net.URLClassLoader.defineClassInternal(URLClassLoader.java:665)
    at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:593)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:477)
    at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:464)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/java.net.URLClassLoader.findClassInternal(URLClassLoader.java:463)
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:455)
    at com.taobao.pandora.service.loader.ModuleClassLoader.resolveClassPath(ModuleClassLoader.java:289)
    at com.taobao.pandora.service.loader.ModuleClassLoader.loadClassInternal(ModuleClassLoader.java:126)
    at com.taobao.pandora.service.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:75)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:558)
    at com.taobao.pandora.service.sharedclass.ClassExporter.scanFromExportIndex(ClassExporter.java:151)
    at com.taobao.pandora.service.sharedclass.ClassExporter.exportClasses(ClassExporter.java:76)
    at com.taobao.pandora.stage.ExportClass2Cache.stepIn(ExportClass2Cache.java:30)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:86)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.StageNode.stepIn(StageNode.java:90)
    at com.taobao.pandora.service.pipeline.Pipeline.execute(Pipeline.java:78)
    at com.taobao.pandora.PandoraContainer.start(PandoraContainer.java:142)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at com.taobao.pandora.boot.loader.SarLoaderUtils.invokeStart(SarLoaderUtils.java:258)
    at com.taobao.pandora.boot.loader.SarLoaderUtils.getClassCache(SarLoaderUtils.java:216)
    at com.taobao.pandora.boot.loader.SarLauncher.loadSar(SarLauncher.java:92)
    at com.taobao.pandora.boot.loader.SarLauncher.createClassLoader(SarLauncher.java:64)
    at com.taobao.pandora.boot.loader.Launcher.createClassLoader(Launcher.java:64)
    at com.taobao.pandora.boot.loader.Launcher.launch(Launcher.java:49)
    at com.taobao.pandora.boot.loader.SarLauncher.main(SarLauncher.java:171)
Caused by: compile error: no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.MemberResolver.searchImports(MemberResolver.java:479)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:422)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:329)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.TypeChecker.atCallExpr(TypeChecker.java:711)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.JvstTypeChecker.atCallExpr(JvstTypeChecker.java:170)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.CallExpr.accept(CallExpr.java:49)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.doTypeCheck(CodeGen.java:266)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atDeclarator(CodeGen.java:819)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.Declarator.accept(Declarator.java:103)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atStmnt(CodeGen.java:381)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.Stmnt.accept(Stmnt.java:53)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atStmnt(CodeGen.java:381)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.ast.Stmnt.accept(Stmnt.java:53)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.CodeGen.atMethodBody(CodeGen.java:321)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.compiler.Javac.compileBody(Javac.java:228)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:466)
    ... 44 more
2023-06-15 15:46:29.879 SEVERE [main] TtlTransformer: Fail to transform class com/taobao/diamond/identify/CredentialWatcher$1, cause: com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:474)
    at com.alibaba.ttl.threadpool.agent.internal.javassist.CtBehavior.setBody(CtBehavior.java:440)

目前还不知道原因，正在排查-_-

oldratlee commented 1 year ago

no such class，看起来是Classloader及其代理关系的设置问题，应该与TTL没有关系。 这个 issue 先 close 了。 @JiaYao9701 

2023-06-15 15:46:29.879 SEVERE [main] TtlTransformer: Fail to transform class com/taobao/diamond/identify/CredentialWatcher$1, cause: com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter
com.alibaba.ttl.threadpool.agent.internal.javassist.CannotCompileException: [source error] no such class: com.alibaba.ttl.TransmittableThreadLocal.Transmitter

如果确定问题原因涉及TTL，可以继续讨论。 💕