search

alibaba / transmittable-thread-local

📌 a missing Java std lib(simple & 0-dependency) for framework/middleware, provide an enhanced InheritableThreadLocal that transmits values between threads even using thread pooling components.
https://github.com/alibaba/transmittable-thread-local
Apache License 2.0
7.35k stars 1.67k forks source link

TTL Agent instrument the class exception #634

Open steverao opened 2 months ago

steverao commented 2 months ago

We encountered a problem by using TTL Agent recently. The phenomenon is as follows: When my application use TTL Agent and other Java Agent together, it will be blocked. The starting log of my application is as follows:

 TtlTransformer: Fail to transform class jdk/jfr/internal/instrument/ThrowableTracer, cause: java.lang.StackOverflowError
java.lang.StackOverflowError
        at java.base/java.lang.LinkageError.<init>(LinkageError.java:55)
        at java.base/java.lang.ClassCircularityError.<init>(ClassCircularityError.java:53)
        at java.base/java.lang.Throwable.<init>(Throwable.java:272)
        at java.base/java.lang.Error.<init>(Error.java:70)
        at java.base/java.lang.LinkageError.<init>(LinkageError.java:55)
        at java.base/java.lang.ClassCircularityError.<init>(ClassCircularityError.java:53)
        at java.base/java.lang.Throwable.<init>(Throwable.java:272)
        at java.base/java.lang.Error.<init>(Error.java:70)
        at java.base/java.lang.LinkageError.<init>(LinkageError.java:55)
        at java.base/java.lang.ClassCircularityError.<init>(ClassCircularityError.java:53)
        at java.base/java.lang.Throwable.<init>(Throwable.java:272)
        at java.base/java.lang.Error.<init>(Error.java:70)
......

Then, I get jstack about my application process and found below information:

"agent-init-thread" #12 prio=5 os_prio=0 cpu=2608.86ms elapsed=31.78s tid=0x00007f91203ff800 nid=0xef runnable  [0x00007f90f88bd000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileOutputStream.writeByexittes(java.base@11.0.15.1/Native Method)
        at java.io.FileOutputStream.write(java.base@11.0.15.1/FileOutputStream.java:354)
        at java.io.BufferedOutputStream.flushBuffer(java.base@11.0.15.1/BufferedOutputStream.java:81)
        at java.io.BufferedOutputStream.flush(java.base@11.0.15.1/BufferedOutputStream.java:142)
        - locked <0x00000000e0048db8> (a java.io.BufferedOutputStream)
        at java.io.PrintStream.write(java.base@11.0.15.1/PrintStream.java:561)
        - locked <0x00000000e0048d90> (a java.io.PrintStream)
        at sun.nio.cs.StreamEncoder.writeBytes(java.base@11.0.15.1/StreamEncoder.java:233)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(java.base@11.0.15.1/StreamEncoder.java:312)
        at sun.nio.cs.StreamEncoder.flushBuffer(java.base@11.0.15.1/StreamEncoder.java:104)
        - locked <0x00000000e0048f10> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.flushBuffer(java.base@11.0.15.1/OutputStreamWriter.java:181)
        at java.io.PrintStream.write(java.base@11.0.15.1/PrintStream.java:606)
        - locked <0x00000000e0048d90> (a java.io.PrintStream)
        at java.io.PrintStream.print(java.base@11.0.15.1/PrintStream.java:745)
        at java.io.PrintStream.println(java.base@11.0.15.1/PrintStream.java:899)
        - locked <0x00000000e0048d90> (a java.io.PrintStream)
        at java.lang.Throwable$WrappedPrintStream.println(java.base@11.0.15.1/Throwable.java:752)
        at java.lang.Throwable.printStackTrace(java.base@11.0.15.1/Throwable.java:662)
        - locked <0x00000000e0048d90> (a java.io.PrintStream)
        at java.lang.Throwable.printStackTrace(java.base@11.0.15.1/Throwable.java:648)
        at java.lang.Throwable.printStackTrace(java.base@11.0.15.1/Throwable.java:639)
        at com.alibaba.ttl.threadpool.agent.internal.logging.Logger$StdErrorLogger.log(Logger.java:70)
        at com.alibaba.ttl.threadpool.agent.TtlTransformer.transform(TtlTransformer.java:67)
        at java.lang.instrument.ClassFileTransformer.transform(java.instrument@11.0.15.1/ClassFileTransformer.java:246)
        at sun.instrument.TransformerManager.transform(java.instrument@11.0.15.1/TransformerManager.java:188)
        at sun.instrument.InstrumentationImpl.transform(java.instrument@11.0.15.1/InstrumentationImpl.java:563)
        at java.lang.Throwable.<init>(java.base@11.0.15.1/Throwable.java:272)
        at java.lang.Error.<init>(java.base@11.0.15.1/Error.java:70)
        at java.lang.LinkageError.<init>(java.base@11.0.15.1/LinkageError.java:55)
        at java.lang.IncompatibleClassChangeError.<init>(java.base@11.0.15.1/IncompatibleClassChangeError.java:55)
        at java.lang.NoSuchMethodError.<init>(java.base@11.0.15.1/NoSuchMethodError.java:58)
        at java.lang.invoke.MethodHandleNatives.resolve(java.base@11.0.15.1/Native Method)
        at java.lang.invoke.MemberName$Factory.resolve(java.base@11.0.15.1/MemberName.java:1070)
        at java.lang.invoke.MemberName$Factory.resolveOrNull(java.base@11.0.15.1/MemberName.java:1113)
        at java.lang.invoke.InvokerBytecodeGenerator.resolveFrom(java.base@11.0.15.1/InvokerBytecodeGenerator.java:625)
        at java.lang.invoke.InvokerBytecodeGenerator.lookupPregenerated(java.base@11.0.15.1/InvokerBytecodeGenerator.java:680)
        at java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(java.base@11.0.15.1/InvokerBytecodeGenerator.java:689)
        at java.lang.invoke.LambdaForm.compileToBytecode(java.base@11.0.15.1/LambdaForm.java:871)
        at java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(java.base@11.0.15.1/DirectMethodHandle.java:287)
        at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.base@11.0.15.1/DirectMethodHandle.java:216)
        at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.base@11.0.15.1/DirectMethodHandle.java:201)
        at java.lang.invoke.DirectMethodHandle.make(java.base@11.0.15.1/DirectMethodHandle.java:94)
        at java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(java.base@11.0.15.1/MethodHandles.java:2322)
        at java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(java.base@11.0.15.1/MethodHandles.java:2278)
        at java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(java.base@11.0.15.1/MethodHandles.java:2520)
        at java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(java.base@11.0.15.1/MethodHandles.java:2466)
        at java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(java.base@11.0.15.1/MethodHandleNatives.java:592)
        at jdk.jfr.internal.PlatformRecorder.startDiskMonitor(jdk.jfr@11.0.15.1/PlatformRecorder.java:386)
        at jdk.jfr.internal.PlatformRecorder.<init>(jdk.jfr@11.0.15.1/PlatformRecorder.java:83)
        at jdk.jfr.FlightRecorder.getFlightRecorder(jdk.jfr@11.0.15.1/FlightRecorder.java:182)
        - locked <0x00000000e3e801b0> (a java.lang.Class for jdk.jfr.internal.PlatformRecorder)
        at jdk.jfr.Recording.<init>(jdk.jfr@11.0.15.1/Recording.java:97)
        at jdk.jfr.Recording.<init>(jdk.jfr@11.0.15.1/Recording.java:121)
        at com.alibaba.cpc.jfr.JFRContinuousProfileRecorder.init(JFRContinuousProfileRecorder.java:45)
......

at com.alibaba.cpc.jfr.JFRContinuousProfileRecorder.init(JFRContinuousProfileRecorder.java:45) is the code from our Java Agent. It seems our Java Agent use jdk.jfr.* class and then TTL Agent transformed them failed and cause relevant thread print a lot of logs and can't stop.

Is there any way to exclude some package that TTL don't transform? Thank you very much!

oldratlee commented 1 month ago

Is there any way to exclude some package that TTL don't transform?

@steverao

At present there's no option of TTL Agent to exclude some packages from TTL Agent class transformation.

Providing the option of TTL Agent(for example: -D option) is a nice function, so user can workaround the problems caused by class transformation.

steverao commented 1 month ago

Providing the option of TTL Agent(for example: -D option) is a nice function, so user can workaround the problems caused by class transformation.

Yes, if we can provide relevant ability for users, it will be better.