server agent crash

[eostermueller]

I just got the InTrace agent to crash again....I'm still working on exact steps to reproduce.

The control thread is alive and will display the names of the classes that are instrumented. The UI can connect and disconnect.

When healthy, there is a continuous stream of events from the trace class (example.FirstTraceExample -- here is the URL: https://github.com/eostermueller/headlessInTraceClient/blob/master/src/main/java/example/FirstTraceExample.java).

But when in this broken condition, if you log onto the server agent and the instrumented class is full dialed in, then there are no events logged.

Oddly enough, this situation persists even after restarting FirstTraceExample.java. Twice, I have fixed this problem by cleaning/compiling FirstTraceExample.java. I have even rebooted my laptop and can start FirstTraceExample.java and the error still shows up.

I got a thread dump of this process in the 'broken' state. I'm working on attaching the logs.

--Erik

[eostermueller]

broken thread dump: https://gist.github.com/eostermueller/516b648409376063bb45 healthy thread dump: https://gist.github.com/eostermueller/a3c55050134fe4c63902

Both thread dumps were taken while no clients were connected. A few things to note:

• The broken thread dump as a few threads labeled as BLOCKED...healthy one has zero threads labeled as BLOCKED.
• The "Instrumentation Status Updates" thread is only present in the healthy thread dump.
• The "TraceServer" thread is only present in the healthy thread dump.
• The exception below is from the server process shows a java.lang.VerifyError on example.FirstTraceExample. This seems like a likely cause of the problem, by can't find a way to reproduce.

main: [java] ## Loaded InTrace Agent. [java] objc[1359]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre/bin/java and /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined. [java] ## InTrace Agent listening on port 9123 [java] java.lang.VerifyError [java] at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method) [java] at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144) [java] at org.intrace.agent.ClassTransformer.instrumentKlasses(ClassTransformer.java:702) [java] at org.intrace.agent.ClassTransformer.getResponse(ClassTransformer.java:575) [java] at org.intrace.agent.server.AgentClieexception transforming [example.FirstTraceExample] [java] ntConnection.runMethod(AgentClientConnection.java:118) [java] at org.intrace.output.InstruRunnable.run(InstruRunnable.java:18) [java] at java.lang.Thread.run(Thread.java:744)

~
~

[eostermueller]

This crash and issue 38 perhaps are the exact same thing. The intrace-agent.jar before Issue 31 enhancement has these traits: 1a) Transformation for one class of ~3000 total classes in my sample app is nearly instantaneous. Less than a full second. 2a) permgen growth is barely noticeable in JVisualVM...perhaps 2mb at most. a single of the ~3000 classes is being instrumented.

The intrace-agent.jar with the Issue 31 enhancement has these traits: 1b) Transformation of the same one of ~3000 classes takes dozens of seconds. 2b) permgen grows more than 25mb for the single same class above. See screenshot for details.

The class that is instrumented is javax.servlet.http.HttpServlet. The sample application is available here: https://github.com/eostermueller/headlessInTraceClient/tree/master/example.webapp

For all this testing, the InTrace GUI was the client used, not my headlessInTraceClient.

Arguments used for the sample-app:

-XX:MaxPermSize=300M -javaagent:/Users/erikostermueller/Documents/src/jdist/intrace/etoRelease-1/intrace-agent.jar=[serverport-9125[gzip-true[exit-stack-trace-true

java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)

...on osx mavericks.

--Erik

[eostermueller]

I forgot to mention....in the screenshot above, there are 3 25mb climbs in the perm gen. Here are the events that triggered these 3 increases: a) adding instrumentation for javax.servlet.http.HttpServlet b) Removing instrumentation for this class. c) adding back instrumentation the exact same class.