MXBean to dynamically configure JVM Diagnostic Options

[chandrams]

Problem determination becomes difficult when there is no diagnostic information. A running OpenJ9 JVM includes mechanisms for producing different types of diagnostic data when different events occur. In general, the production of this data happens by default and can also be controlled by using the command line JVM options such as -Xdump or -Xtrace at JVM startup or by dynamically setting them using the com.ibm.jvm.Dump or com.ibm.jvm.Trace API

Instead of using these APIs, we can include a MXBean (which in turn calls the methods of Dump/Trace API) to dynamically configure these JVM options while remotely monitoring an application running in a remote server or container. This MXBean can be used in jconsole (or by any other monitoring tool / admin console) to dynamically configure the diagnostic options while monitoring the application, without having to actually restart the application.

We can include the following methods in the MXBean:

1. configureDumpOptions(String dumpOption) – sets the specified dump options
2. setTraceOptions(String traceOption) - sets the trace options
3. resetDumpOption(String option) – resets the dump options to what it was at JVM initialization
4. triggerDump(String dumpType) – triggers a dump agent based on dump type requested
5. triggerDumpToFile(String dumpType, String fileNamePattern) - triggers a dump agent based on the dump type, writes to the file name specified by the user and returns the filename to which the dump was actually written to
6. triggerClassicHeapDump() – triggers a heap dump in the classic format

We can also extend the functionality to include few most commonly used options for capturing diagnostic information, for example,

JavaDumpOnThreadBlock - setting this would trigger a Java dump on a thread block.

Above option is just an example, we need to find if there are any such frequently used options.

Usage

  try {
        mxbeanName = new ObjectName("com.ibm.lang.management:type=Diagnostics");
   } catch (MalformedObjectNameException e) {
        // Exception Handling
   }

  try {
         MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer();
         if (true != mbeanServer.isRegistered(mxbeanName)) {
                // DiagnosticsMXBean not registered
         }
         DiagnosticsMXBean bean = JMX.newMXBeanProxy(mbeanServer, mxbeanName, DiagnosticsMXBean.class);
   } catch (Exception e) {
          // Exception Handling
   }
  bean.configureDumpOptions("heap+java:events=vmstop"); // trigger heap & java dumps on vmstop
  bean.configureDumpOptions("heap:none"); // Remove all heap dump agents
  bean.resetDumpOption();

  bean.triggerDump(“java”); // trigger a java dump

  // trigger a heap dump and written the file name to which the dump was actually written to
  String filename = bean.triggerDumpToFile(“heap”, "heapdump.%pid.phd"); 

Opening this issue to discuss on this requirement, please feel free to comment.

[andrewcraik]

given that the dump options can be hard to compose into a command line would it make more sense to expose an API that helps you build the dump option you want and have that generate and set the JVM option so that the interface is more user friendly? Not sure if that would make the builder become stale too quickly, but making it easier to build the options string would seem to be a nice usability feature since the syntax is pretty complicated. Thoughts?

[ashu-mehra]

expose an API that helps you build the dump option you want and have that generate and set the JVM option so that the interface is more user friendly

+1 for that. If some wants to use this API, he shouldn't have to go through the user guide to understand the syntax to be able to pass correct option.

[chandrams]

Thought of having a mapping for few frequently used options, for example, JavaDumpOnThread would have a mapping internally to its equivalent -Xdump option. But having it generated for all cases would be useful.

[paulcheeseman]

I have written a javascript tool for building Xdump options (internal IBM link). I hope to do the same for Xtrace when I have time, although I think Xtrace would be significantly more complicated.

Maybe we could integrate something like this into the OpenJ9 documentation?

[pshipton]

@paulcheeseman By all means, please open a Pull Request on http://github.com/eclipse/openj9-website to contribute this. I'm sure the docs team will help integrate it into the website.

[paulcheeseman]

I've opened an issue against the openj9-website repo.

[pshipton]

Sorry, copy/pasted the wrong link by accident, the correct website link is http://github.com/eclipse/openj9-website (corrected in previous comment as well).

[chandrams]

Tested with a liberty application running in a docker container on a remote host and was able to trigger dumps using jconsole to a shared volume.

[chandrams]

Below are few observations and queries:

1) Dump.setDumpOptions() did not work when I passed the below option: "java:events=throw,filter=java/lang/NullPointerException#DumpTest.main*,range=1..1"

After investigation, I found -Xdump:dynamic has to be specified if throw/catch events are configured through Dump.setDumpOptions() when JIT is ON, whereas this option is not required when we set the above option on command line using -Xdump.

Shouldn't this be documented in Dump API or somewhere else? I could not find any documentation on this.

2) Trace.set() on the below option returned 0 which as per javadoc means Success, but I do not see any trace information generated. Isn't it misleading to return success when the trace option is not actually honored?

"methods=java/lang/String.*,print=mt"

3) As per the javadoc for Trace API, all trace options supported on command line are not supported through the API, but there are no details as to which options aren't supported. Shouldn't this be documented?