search

mchr3k / org.intrace

Java tracing agent and live trace client
http://mchr3k.github.com/org.intrace/
73 stars 18 forks source link

Feature request: Listener API to eavesdrop on trace events #12

Open eostermueller opened 12 years ago

eostermueller commented 12 years ago

Expose a set of interfaces that allows a programmer to write a program (called the Implementor) that receives all InTrace trace events.

The classes java.util.Observable and java.util.Observer could be used as a model for some, but not all, of the interaction between InTrace and an Implementor.

Each Implementor would run in the same JVM as the InTrace GUI. Some kind of configuration would store class names of all Implementors. -D parameters would be acceptable for a "first attempt." Once InTrace has all the configuration for a single Implementor, that Implementor will be considered "registered"

The interface must allow the InTrace GUI to ask the Implementor for a human-friendly name that will be used for display in the InTrace GUI.

The interface must allow the InTrace GUI to ask the Implementor for an Icon unique for that Implementor. If none is provided, a default (but unique one) will be used.

At GUI startup time, InTrace will ask each registered "Implementor" for a list of -- classes to be instrumented. Include/exclude functionality should be incorporated. -- Filters to be used. Include/exclude functionality should be incorporated. Once the above and any other initialization is complete, InTrace GUI should invoke some kind of "startup" method on each Implementor.

When the end user closes the GUI, InTrace should notify each Implementor of the shutdown request.

Implementor display/output

Each implementor will have its own "section" in the InTrace GUI.
When InTrace UI starts, the end user will need a menu option to open a particular Implementor with a particular host/port.

Example Implementors

Each of these ideas imitates functionality found in commercially offered APM (Application Performance Monitoring) tools, like CompuWare DynaTrace and CA Introscope.

1) Facility to display Gantt charts for each request of a SOA application being traced. Gantt chart will display response time for each traced method. Important to show "nested" methods. 2) Response time/throughput graphs of instrumented classes 3) List of all SQL statements executed by the application being traced.

Option 1), above, could be expanded to show the following for each SOA request: a) input xml b) response xml c) all SQL requests made during the request d) all exceptions thrown during the request.

mchr3k commented 12 years ago

Hi eostermueller,

Thanks for this feature suggestion (and the other two you raised). I am pleased to see that you are interested in InTrace.

Can you give me a bit more background about yourself and these suggestion you have raised? I would be interested to know all of the following.

eostermueller commented 12 years ago

mchr3k wrote:

Have you used InTrace much or have you just had a quick play? Just a quick play. I verified that I can capture SQL statements from the Java PetStore application. I like the way you can dynamically add/remove the instrumentation and the filters.

What are you using InTrace for? Personal/commercial projects? Both commercial and personal, actually. The company I work for relies heavily on big-dollar APM tools for a number of things. InTrace can't completely replace those tools, but it could reduce the number of licenses we need for those tools. What kind of big dollar tools? CompuWare DynaTrace and CA Introscope. They cost tens of thousands of USD. So that is the business side....on the personal side, I'm toying with the idea of writing a little book on performance testing. That book would use some techniques that partially rely on the InTrace feature requests I've made.

Are you interested in getting involved in the development of these features or are you hoping I will implement these? Do you have a timeline in mind? This month? This year? Yes, I'm ready to get involved. When? How about tomorrow? But I do have a few concerns, before getting started.

a) I need to run some tests with an IBM jvm on linux. If I can get those tests to work (hopefully by tomorrow), I'll be ready start helping.

b) If InTrace is monitoring a remote application, it seems that a separate network request is made for every trace event. Is that right? -- I haven't looked closely enough at the code to see. If that is the case, I don't think that kind of model will scale when medium amounts of load is applied. Perhaps we could consider some kind of burst mode where trace events collect in a queue and are sent to the GUI every X seconds? CA Introscope, for instance, has a 15 second delay. I'd be happy with 5.

c) The systems we deploy have multiple, identical JVMs that process traffic (often, but not always, deployed on separate machines). I'd like to see a single InTrace GUI capture events from multiple JVMs. Obviously, the traffic from each would have to be somehow labeled so the user can distinguish traffic from each JVM.

d) You have support for millisecond response time. I'd really like to provide support (perhaps optionally) for nanoTime http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime()

Only item a) is a show stopper for me. I need to make sure that is resolved before investing any time. I'd like to hear your opinion on the other two. If needed, I'd be ok to add support for b) and c) and d) at a later date.

Since this is your code base, perhaps you could propose a design for these enhancements. We could agree on the design then you could slice off a few pieces for me to work on. What do you think?

So I mentioned these three "plugins" or "implementors":

1) Facility to display Gantt charts for each request of a SOA application being traced. Gantt chart will display response time for each traced method. Important to show "nested" methods. 2) Response time/throughput graphs of instrumented classes 3) List of all SQL statements executed by the application being traced.

For plugin 1), I've found two decent libraries we could use to create Gantt charts: http://www.jfree.org/jfreechart/ http://www.simile-widgets.org/timeline/

The second tool, Simile-Timeline has better scrolling/navigation, but it is in JavaScript which would make things messy. So I'd like to start with JFreeChart.

In regards to plugin 2). I'm sick of tools that use "averages" to plot response time. Averages hide outliers that are often very important. So instead, I'd like to somehow display the counts of responses in various time ranges.

For instance, say we had the following response times for 3 requests in a single 5 second interval:

500ms 95ms 105ms

The average of these numbers is about 233 -- which doesn't some close to representing the 500, the 95 or the 105ms.

Instead, I'd like the graph to somehow show the counts of fast and slow responses in various ranges, something like this:

2 requests faster than 128ms 0 requests between 128 and 256ms 1 request between 256 and 512ms.

The ranges will need to be configurable, but this gives you the idea. We'd have to work out whether we used stacked histograms, or perhaps color to show these counts.

Let me know what you think.

--Erik

On Mon, Jan 2, 2012 at 2:36 PM, mchr3k < reply@reply.github.com

wrote:

Hi eostermueller,

Thanks for this feature suggestion (and the other two you raised). I am pleased to see that you are interested in InTrace.

Can you give me a bit more background about yourself and these suggestion you have raised? I would be interested to know all of the following.

  • Have you used InTrace much or have you just had a quick play?
  • What are you using InTrace for? Personal/commercial projects?
  • Are you interested in getting involved in the development of these features or are you hoping I will implement these?
  • Do you have a timeline in mind? This month? This year?

Reply to this email directly or view it on GitHub: https://github.com/mchr3k/org.intrace/issues/12#issuecomment-3333367

eostermueller commented 12 years ago

mchr3k,

Rats. I'm having problems getting an IBM jdk working with InTrace. When I start your example.TraceExample program with the javaagent parameter, I'm getting the stack trace below. Suggestions? I really need to have IBM jdk support before I can help with anything else. It looks like this is the source that is throwing the error:

http://www.docjar.com/html/api/sun/instrument/InstrumentationImpl.java.html*

How much InTrace functionality relies on "retransforming" ?

Perhaps removing InTrace support for "on the fly" instrumentation on IBM jdk would help?

--Erik

Buildfile: build.xml****

init:****

run:****

 [java] ## Loaded InTrace Agent.****

 [java] Exception in thread "main"

java.lang.UnsupportedOperationException: cannot get the capability, performing dispose of the retransforming environment****

 [java]     at

sun.instrument.InstrumentationImpl.isRetransformClassesSupported0(Native Method)****

 [java]     at

sun.instrument.InstrumentationImpl.isRetransformClassesSupported(InstrumentationImpl.java:124)

 [java]     at

sun.instrument.InstrumentationImpl.addTransformer(InstrumentationImpl.java:79)

 [java]     at org.intrace.agent.AgentInit.initialize(AgentInit.java:41)
 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

 [java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at org.intrace.agent.Agent.initialize(Agent.java:66)****

 [java]     at org.intrace.agent.Agent.premain(Agent.java:26)****

 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

 [java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:335)

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:350)

 [java] Exception in thread "main"

java.lang.reflect.InvocationTargetException****

 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

 [java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:335)

 [java]     at

sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:350)

 [java] Caused by: java.lang.RuntimeException:

java.lang.reflect.InvocationTargetException****

 [java]     at org.intrace.agent.Agent.initialize(Agent.java:70)****

 [java]     at org.intrace.agent.Agent.premain(Agent.java:26)****

 [java]     ... 6 more****

 [java] Caused by: java.lang.reflect.InvocationTargetException****

 [java]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native

Method)****

 [java]     at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)

 [java]     at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)

[java]     at java.lang.reflect.Method.invoke(Method.java:600)****

 [java]     at org.intrace.agent.Agent.initialize(Agent.java:66)****

 [java]     ... 7 more****

 [java] Caused by: java.lang.UnsupportedOperationException: adding

retransformable transformers is not supported in this environment****

 [java]     at

sun.instrument.InstrumentationImpl.addTransformer(InstrumentationImpl.java:80)

 [java]     at org.intrace.agent.AgentInit.initialize(AgentInit.java:41)
 [java]     ... 12 more****
 [java] Fatal error: processing of -javaagent failed****

BUILD SUCCESSFUL****

Total time: 0 seconds

On Mon, Jan 2, 2012 at 5:55 PM, Erik Ostermueller eostermueller@gmail.comwrote:

mchr3k wrote:

Have you used InTrace much or have you just had a quick play? Just a quick play. I verified that I can capture SQL statements from the Java PetStore application. I like the way you can dynamically add/remove the instrumentation and the filters.

What are you using InTrace for? Personal/commercial projects? Both commercial and personal, actually. The company I work for relies heavily on big-dollar APM tools for a number of things. InTrace can't completely replace those tools, but it could reduce the number of licenses we need for those tools. What kind of big dollar tools? CompuWare DynaTrace and CA Introscope. They cost tens of thousands of USD. So that is the business side....on the personal side, I'm toying with the idea of writing a little book on performance testing. That book would use some techniques that partially rely on the InTrace feature requests I've made.

Are you interested in getting involved in the development of these features or are you hoping I will implement these? Do you have a timeline in mind? This month? This year? Yes, I'm ready to get involved. When? How about tomorrow? But I do have a few concerns, before getting started.

a) I need to run some tests with an IBM jvm on linux. If I can get those tests to work (hopefully by tomorrow), I'll be ready start helping.

b) If InTrace is monitoring a remote application, it seems that a separate network request is made for every trace event. Is that right? -- I haven't looked closely enough at the code to see. If that is the case, I don't think that kind of model will scale when medium amounts of load is applied. Perhaps we could consider some kind of burst mode where trace events collect in a queue and are sent to the GUI every X seconds? CA Introscope, for instance, has a 15 second delay. I'd be happy with 5.

c) The systems we deploy have multiple, identical JVMs that process traffic (often, but not always, deployed on separate machines). I'd like to see a single InTrace GUI capture events from multiple JVMs. Obviously, the traffic from each would have to be somehow labeled so the user can distinguish traffic from each JVM.

d) You have support for millisecond response time. I'd really like to provide support (perhaps optionally) for nanoTime

http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/System.html#nanoTime()

Only item a) is a show stopper for me. I need to make sure that is resolved before investing any time. I'd like to hear your opinion on the other two. If needed, I'd be ok to add support for b) and c) and d) at a later date.

Since this is your code base, perhaps you could propose a design for these enhancements. We could agree on the design then you could slice off a few pieces for me to work on. What do you think?

So I mentioned these three "plugins" or "implementors":

1) Facility to display Gantt charts for each request of a SOA application being traced. Gantt chart will display response time for each traced method. Important to show "nested" methods. 2) Response time/throughput graphs of instrumented classes 3) List of all SQL statements executed by the application being traced.

For plugin 1), I've found two decent libraries we could use to create Gantt charts: http://www.jfree.org/jfreechart/ http://www.simile-widgets.org/timeline/

The second tool, Simile-Timeline has better scrolling/navigation, but it is in JavaScript which would make things messy. So I'd like to start with JFreeChart.

In regards to plugin 2). I'm sick of tools that use "averages" to plot response time. Averages hide outliers that are often very important. So instead, I'd like to somehow display the counts of responses in various time ranges.

For instance, say we had the following response times for 3 requests in a single 5 second interval:

500ms 95ms 105ms

The average of these numbers is about 233 -- which doesn't some close to representing the 500, the 95 or the 105ms.

Instead, I'd like the graph to somehow show the counts of fast and slow responses in various ranges, something like this:

2 requests faster than 128ms 0 requests between 128 and 256ms 1 request between 256 and 512ms.

The ranges will need to be configurable, but this gives you the idea. We'd have to work out whether we used stacked histograms, or perhaps color to show these counts.

Let me know what you think.

--Erik

On Mon, Jan 2, 2012 at 2:36 PM, mchr3k < reply@reply.github.com

wrote:

Hi eostermueller,

Thanks for this feature suggestion (and the other two you raised). I am pleased to see that you are interested in InTrace.

Can you give me a bit more background about yourself and these suggestion you have raised? I would be interested to know all of the following.

  • Have you used InTrace much or have you just had a quick play?
  • What are you using InTrace for? Personal/commercial projects?
  • Are you interested in getting involved in the development of these features or are you hoping I will implement these?
  • Do you have a timeline in mind? This month? This year?

Reply to this email directly or view it on GitHub: https://github.com/mchr3k/org.intrace/issues/12#issuecomment-3333367

mchr3k commented 12 years ago

How much InTrace functionality relies on "retransforming" ?

The dynamic selection of which classes to trace at runtime depends on retransforming. However, it would be possible to modify InTrace to specify all classes to trace at startup.

Without retransforming support it won't be possible to instrument any Java classes which are loaded before InTrace. However, application classes should be possible to instrument.

Perhaps removing InTrace support for "on the fly" instrumentation on IBM jdk would help?

This sounds like it is necessary for an IBM JDK. This will require code and build changes.

I can provide limited amounts of time/support about what changes need to be made. However, InTrace is something which I do in my spare time so I won't be able to promise you much.

I wonder if you would be interested in agreeing to a basic contract for me to provide a higher level of time/support for implementing the changes you need. If this is something you want to discuss further then it may be better to do it directly over email. You can contact me at mchr3k at gmail.com.