Closed luffy1849 closed 3 weeks ago
raphw
commented
1 year ago The lock occurs as two threads attempt to load a class which requires a lock for looking up a class file. The same issue can however occur during regular class loading. I would not know that this is related to class loading and is likely a bug. Are you running an outdated version of Tomcat?
luffy1849
commented
1 year ago The tomcat version is not out of date And I'm not sure if this problem can be avoided by excluding bytebuddy from loading some class
using catalina jvm flags: -Dsun.net.spi.nameservice.provider.1=dns,dnsjava -Dsun.net.spi.nameservice.provider.2=default -Xmx6g -Xms6g -Xmn3g -Xss1m -javaagent:/data1/insight-agent/insight-agent.jar -Xss1m -XX:MaxTenuringThreshold=4 -XX:+UseConcMarkSweepGC -XX:SurvivorRatio=8 -XX:CMSInitiatingOccupancyFraction=70 -XX:+ExplicitGCInvokesConcurrent -server -DServer=live-stream-dispatcher -XX:-OmitStackTraceInFastThrow -XX:+PrintFlagsFinal -XX:+PrintCommandLineFlags -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:../gclogs/gc.log.20230426_101614 -XX:CICompilerCount=6
Using CATALINA_BASE: /data1
Using CATALINA_HOME: /data1
Using CATALINA_TMPDIR: /data1/temp
Using JRE_HOME: /usr/jdk1.8.0_121
Using CLASSPATH: /data1/bin/bootstrap.jar:/data1/bin/tomcat-juli.jar
Server version: Apache Tomcat/8.0.46
Server built: Aug 10 2017 10:10:31 UTC
Server number: 8.0.46.0
OS Name: Linux
OS Version: 4.19.91-21.2.al7.x86_64
Architecture: amd64
JVM Version: 1.8.0_121-b13
JVM Vendor: Oracle CorporationByte Buddy is not loading a class, it is looking up a class file. This happens during instrumentation, I am not really sure how this can trigger a deadlock for Tomcat.
amar1019
commented
1 year ago Facing the same on Server version: Apache Tomcat/9.0.58. It does not happen everytime though.
amar1019
commented
1 year ago Seems like it is happening when agent's own classes are getting loaded. It is not consistent. Resolved it by excluding all agent's classes. Earlier i was excluding only bytebuddy package.
luffy1849
commented
1 year ago Seems like it is happening when agent's own classes are getting loaded. It is not consistent. Resolved it by excluding all agent's classes. Earlier i was excluding only bytebuddy package.
I haven't solved this yet. I don't quite understand. For example, which classes are excluded?
amar1019
commented
1 year ago In my case, i am excluding all the agents' classes including bytebuddy( com.test., which includes com.test.bytebuddy. and com.test.advice. and com.test.util.). Earlier i was giving com.test.bytebuddy.*
raphw
commented
1 year ago You should not instrument the agent from within its own class loader, which should factor out the agent.
Ideally, you bundle the agent in a custom jar and when loading the agent, you create a new class loader which then loads the agent within its own class loader. From this agent, exclude this dedicated class loader.
luffy1849
commented
1 year ago I solved this problem by excluding sun.net.www.protocol
This seems to be caused by tomcat's listener JreMemoryLeakPreventionListener
I also found other people on the Internet who had the same problem as me https://discuss.elastic.co/t/use-javaagent-happened-deadlock/320132
Other similar deadlock details