search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 721 forks source link

Split DTFJ interface classes into separate module #2084

Open pshipton opened 6 years ago

pshipton commented 6 years ago

The DTFJ interface classes used to be in a separate jar file in Java 8. As part of the work to move to OpenJ9 and Java 9 modules, the DTFJ interface classes were all moved into a single dtfj.jar file, and the openj9.dtfj module for Java 9 and later. It seems they need to be separated again so they can be used by the Eclipse DTFJ plugin.

The Eclipse DTFJ plugin [1], used by Eclipse MAT [2] and perhaps other things, needs the interface classes. The interface classes were separated out and included in the plugin. When trying to use the plugin with MAT, https://bugs.eclipse.org/bugs/show_bug.cgi?id=535351 results:

Opening a core dump using the DTFJ reader fails under OpenJ9 Java 10:

java.lang.ClassCastException: com.ibm.dtfj.image.j9.ImageFactory incompatible with com.ibm.dtfj.image.ImageFactory
    at org.eclipse.mat.dtfj.DTFJIndexBuilder.getDynamicDTFJDump(DTFJIndexBuilder.java:8529)
    at org.eclipse.mat.dtfj.DTFJIndexBuilder.getUncachedDump(DTFJIndexBuilder.java:8492)
    at org.eclipse.mat.dtfj.DTFJIndexBuilder.getDump(DTFJIndexBuilder.java:8305)
    at org.eclipse.mat.dtfj.DTFJIndexBuilder.fill(DTFJIndexBuilder.java:888)
    at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:222)
    at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.openSnapshot(SnapshotFactoryImpl.java:126)
    at org.eclipse.mat.snapshot.SnapshotFactory.openSnapshot(SnapshotFactory.java:145)
    at org.eclipse.mat.ui.snapshot.ParseHeapDumpJob.run(ParseHeapDumpJob.java:83)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:60)

This seems to be because of a class loader problem – ImageFactory came from a VM class loader, and j9.ImageFactory from a plugin:
factory com.ibm.dtfj.image.j9.ImageFactory@da9ca2cc class com.ibm.dtfj.image.j9.ImageFactory org.eclipse.osgi.internal.loader.EquinoxClassLoader@b4a04aa7[com.ibm.dtfj.j9:1.12.29003.201710051509(id=3)]
interface com.ibm.dtfj.image.ImageFactory jdk.internal.loader.ClassLoaders$AppClassLoader@8405452d

@ajohnson1 what is the test case to recreate this?

[1] http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/ [2] Memory Analyzer standalone https://www.eclipse.org/mat/snapshotBuilds.php

robbinspg commented 6 years ago

I'm not convinced this is related to splitting out the interface classes. The current DTFJ Feature contains an interface jar in the api plugin which has the correct classes in.

When running MAT ALL the DTFJ classes should be coming from the plugins so all should be using the plugin classloader.

Are the DTFJ classes also on the runtime classpath in OpenJ9 Java 10?

ajohnson1 commented 6 years ago

The test case is to take the latest Memory Analyzer nightly build from https://www.eclipse.org/mat/snapshotBuilds.php, start it under OpenJ9 using

MemoryAnalyzer -vm <path to jvm>

install DTFJ using Help > Install New Software > 'IBM Diagnostic Tool Framework for Java' http://public.dhe.ibm.com/ibmdl/export/pub/software/websphere/runtimes/tools/dtfj/

Restart Memory Analyzer. Create a core dump using File > Acquire Heap Dump or choose an existing one and open it.

I am not certain that splitting the interface classes is the way to solve this problem.

When the DTFJ feature is installed into Memory Analyzer (or Eclipse when the MAT feature is installed) then Memory Analyzer should use this version of DTFJ, even if it is an OpenJDK or Oracle JVM. This is how it worked with non-modular JVMs, and with Oracle Java 10 which doesn't have a JVM DTFJ.

The DTFJ feature is already split into plugins - a com.ibm.dtfj.api plugin which holds the interfaces and which the org.eclipse.mat.dtfj plugin is compiled against. The DTFJ feature also provides some implementation plugins, com.ibm.dtfj.sov and com.ibm.dtfj.j9. The j9 plugin then also has access to com.ibm.java.diagnostics.blobs, com.ibm.java.diagnostics.core, com.ibm.java.diagnostics.ddr, com.ibm.java.diagnostics.ddr.core, but MAT doesn't deal directly with any implementation plugins.

When Memory Analyzer RCP runs under OpenJ9 Java 10 the following variable is set Help > About memory Analyzer > Installation Details > Configuration:

org.osgi.framework.system.packages=com.ibm.cuda, com.ibm.dataaccess, com.ibm.dtfj.image, com.ibm.dtfj.java, com.ibm.dtfj.runtime, com.ibm.gpu, com.ibm.jvm, com.ibm.lang.management, com.ibm.le.conditionhandling, com.ibm.oti.shared, com.ibm.virtualization.management, com.sun.jarsigner, com.sun.java.accessibility.util, com.sun.javadoc, com.sun.jdi, com.sun.jdi.connect, com.sun.jdi.connect.spi, com.sun.jdi.event, com.sun.jdi.request, com.sun.management, com.sun.net.httpserver, com.sun.net.httpserver.spi, com.sun.nio.file, com.sun.nio.sctp, com.sun.security.auth, com.sun.security.auth.callback, com.sun.security.auth.login, com.sun.security.auth.module, com.sun.security.jgss, com.sun.source.doctree, com.sun.source.tree, com.sun.source.util, com.sun.tools.attach, com.sun.tools.attach.spi, com.sun.tools.javac, com.sun.tools.javadoc, com.sun.tools.jconsole, java.applet, java.awt, java.awt.color, java.awt.datatransfer, java.awt.desktop, java.awt.dnd, java.awt.event, java.awt.font, java.awt.geom, java.awt.im, java.awt.im.spi, java.awt.image, java.awt.image.renderable, java.awt.print, java.beans, java.beans.beancontext, java.io, java.lang, java.lang.annotation, java.lang.instrument, java.lang.invoke, java.lang.management, java.lang.module, java.lang.ref, java.lang.reflect, java.math, java.net, java.net.spi, java.nio, java.nio.channels, java.nio.channels.spi, java.nio.charset, java.nio.charset.spi, java.nio.file, java.nio.file.attribute, java.nio.file.spi, java.rmi, java.rmi.activation, java.rmi.dgc, java.rmi.registry, java.rmi.server, java.security, java.security.acl, java.security.cert, java.security.interfaces, java.security.spec, java.sql, java.text, java.text.spi, java.time, java.time.chrono, java.time.format, java.time.temporal, java.time.zone, java.util, java.util.concurrent, java.util.concurrent.atomic, java.util.concurrent.locks, java.util.function, java.util.jar, java.util.logging, java.util.prefs, java.util.regex, java.util.spi, java.util.stream, java.util.zip, javax.accessibility, javax.annotation.processing, javax.crypto, javax.crypto.interfaces, javax.crypto.spec, javax.imageio, javax.imageio.event, javax.imageio.metadata, javax.imageio.plugins.bmp, javax.imageio.plugins.jpeg, javax.imageio.plugins.tiff, javax.imageio.spi, javax.imageio.stream, javax.lang.model, javax.lang.model.element, javax.lang.model.type, javax.lang.model.util, javax.management, javax.management.loading, javax.management.modelmbean, javax.management.monitor, javax.management.openmbean, javax.management.relation, javax.management.remote, javax.management.remote.rmi, javax.management.timer, javax.naming, javax.naming.directory, javax.naming.event, javax.naming.ldap, javax.naming.spi, javax.net, javax.net.ssl, javax.print, javax.print.attribute, javax.print.attribute.standard, javax.print.event, javax.rmi.ssl, javax.script, javax.security.auth, javax.security.auth.callback, javax.security.auth.kerberos, javax.security.auth.login, javax.security.auth.spi, javax.security.auth.x500, javax.security.cert, javax.security.sasl, javax.smartcardio, javax.sound.midi, javax.sound.midi.spi, javax.sound.sampled, javax.sound.sampled.spi, javax.sql, javax.sql.rowset, javax.sql.rowset.serial, javax.sql.rowset.spi, javax.swing, javax.swing.border, javax.swing.colorchooser, javax.swing.event, javax.swing.filechooser, javax.swing.plaf, javax.swing.plaf.basic, javax.swing.plaf.metal, javax.swing.plaf.multi, javax.swing.plaf.nimbus, javax.swing.plaf.synth, javax.swing.table, javax.swing.text, javax.swing.text.html, javax.swing.text.html.parser, javax.swing.text.rtf, javax.swing.tree, javax.swing.undo, javax.tools, javax.transaction.xa, javax.xml, javax.xml.catalog, javax.xml.crypto, javax.xml.crypto.dom, javax.xml.crypto.dsig, javax.xml.crypto.dsig.dom, javax.xml.crypto.dsig.keyinfo, javax.xml.crypto.dsig.spec, javax.xml.datatype, javax.xml.namespace, javax.xml.parsers, javax.xml.stream, javax.xml.stream.events, javax.xml.stream.util, javax.xml.transform, javax.xml.transform.dom, javax.xml.transform.sax, javax.xml.transform.stax, javax.xml.transform.stream, javax.xml.validation, javax.xml.xpath, jdk.dynalink, jdk.dynalink.beans, jdk.dynalink.linker, jdk.dynalink.linker.support, jdk.dynalink.support, jdk.javadoc.doclet, jdk.jshell, jdk.jshell.execution, jdk.jshell.spi, jdk.jshell.tool, jdk.nashorn.api.scripting, jdk.nashorn.api.tree, jdk.net, jdk.security.jarsigner, netscape.javascript, openj9.lang.management, org.ietf.jgss, org.w3c.dom, org.w3c.dom.bootstrap, org.w3c.dom.css, org.w3c.dom.events, org.w3c.dom.html, org.w3c.dom.ls, org.w3c.dom.ranges, org.w3c.dom.stylesheets, org.w3c.dom.traversal, org.w3c.dom.views, org.w3c.dom.xpath, org.xml.sax, org.xml.sax.ext, org.xml.sax.helpers, sun.misc, sun.reflect

which might be what is making the JVM version of DTFJ visible, as packages 'com.ibm.dtfj.image, com.ibm.dtfj.java, com.ibm.dtfj.runtime' are mentioned. I don't know how to change this (remove packages) though adding packages is possible via org.osgi.framework.system.packages.extra - which would only work for Memory Analyzer RCP.

However Memory Analyzer does need access to the com.sun.tools.attach and com.sun.tools.attach.spi packages from the JVM for the org.eclipse.mat.ibmdumps plugin which uses that API to acquire dumps from Oracle and IBM JVMs using the Attach API.

Running under Oracle Java 8 1.8.0_112, the packages in that list are much fewer, and don't include the com.sun classes. Although the com.sun.tools.attach packages are not listed the classes are available via reflection.

org.osgi.framework.system.packages=java.applet,java.awt,java.awt.color,java.awt.datatransfer,java.awt.dnd,java.awt.event,java.awt.font,java.awt.geom,java.awt.im,java.awt.im.spi,java.awt.image,java.awt.image.renderable,java.awt.print,java.beans,java.beans.beancontext,java.io,java.lang,java.lang.annotation,java.lang.instrument,java.lang.invoke,java.lang.management,java.lang.ref,java.lang.reflect,java.math,java.net,java.nio,java.nio.channels,java.nio.channels.spi,java.nio.charset,java.nio.charset.spi,java.nio.file,java.nio.file.attribute,java.nio.file.spi,java.rmi,java.rmi.activation,java.rmi.dgc,java.rmi.registry,java.rmi.server,java.security,java.security.acl,java.security.cert,java.security.interfaces,java.security.spec,java.sql,java.text,java.text.spi,java.time,java.time.chrono,java.time.format,java.time.temporal,java.time.zone,java.util,java.util.concurrent,java.util.concurrent.atomic,java.util.concurrent.locks,java.util.function,java.util.jar,java.util.logging,java.util.prefs,java.util.regex,java.util.spi,java.util.stream,java.util.zip,javax.accessibility,javax.activation,javax.activity,javax.annotation,javax.annotation.processing,javax.crypto,javax.crypto.interfaces,javax.crypto.spec,javax.imageio,javax.imageio.event,javax.imageio.metadata,javax.imageio.plugins.bmp,javax.imageio.plugins.jpeg,javax.imageio.spi,javax.imageio.stream,javax.jws,javax.jws.soap,javax.lang.model,javax.lang.model.element,javax.lang.model.type,javax.lang.model.util,javax.management,javax.management.loading,javax.management.modelmbean,javax.management.monitor,javax.management.openmbean,javax.management.relation,javax.management.remote,javax.management.remote.rmi,javax.management.timer,javax.naming,javax.naming.directory,javax.naming.event,javax.naming.ldap,javax.naming.spi,javax.net,javax.net.ssl,javax.print,javax.print.attribute,javax.print.attribute.standard,javax.print.event,javax.rmi,javax.rmi.CORBA,javax.rmi.ssl,javax.script,javax.security.auth,javax.security.auth.callback,javax.security.auth.kerberos,javax.security.auth.login,javax.security.auth.spi,javax.security.auth.x500,javax.security.cert,javax.security.sasl,javax.sound.midi,javax.sound.midi.spi,javax.sound.sampled,javax.sound.sampled.spi,javax.sql,javax.sql.rowset,javax.sql.rowset.serial,javax.sql.rowset.spi,javax.swing,javax.swing.border,javax.swing.colorchooser,javax.swing.event,javax.swing.filechooser,javax.swing.plaf,javax.swing.plaf.basic,javax.swing.plaf.metal,javax.swing.plaf.multi,javax.swing.plaf.nimbus,javax.swing.plaf.synth,javax.swing.table,javax.swing.text,javax.swing.text.html,javax.swing.text.html.parser,javax.swing.text.rtf,javax.swing.tree,javax.swing.undo,javax.tools,javax.transaction,javax.transaction.xa,javax.xml,javax.xml.bind,javax.xml.bind.annotation,javax.xml.bind.annotation.adapters,javax.xml.bind.attachment,javax.xml.bind.helpers,javax.xml.bind.util,javax.xml.crypto,javax.xml.crypto.dom,javax.xml.crypto.dsig,javax.xml.crypto.dsig.dom,javax.xml.crypto.dsig.keyinfo,javax.xml.crypto.dsig.spec,javax.xml.datatype,javax.xml.namespace,javax.xml.parsers,javax.xml.soap,javax.xml.stream,javax.xml.stream.events,javax.xml.stream.util,javax.xml.transform,javax.xml.transform.dom,javax.xml.transform.sax,javax.xml.transform.stax,javax.xml.transform.stream,javax.xml.validation,javax.xml.ws,javax.xml.ws.handler,javax.xml.ws.handler.soap,javax.xml.ws.http,javax.xml.ws.soap,javax.xml.ws.spi,javax.xml.ws.spi.http,javax.xml.ws.wsaddressing,javax.xml.xpath,org.ietf.jgss,org.omg.CORBA,org.omg.CORBA_2_3,org.omg.CORBA_2_3.portable,org.omg.CORBA.DynAnyPackage,org.omg.CORBA.ORBPackage,org.omg.CORBA.portable,org.omg.CORBA.TypeCodePackage,org.omg.CosNaming,org.omg.CosNaming.NamingContextExtPackage,org.omg.CosNaming.NamingContextPackage,org.omg.Dynamic,org.omg.DynamicAny,org.omg.DynamicAny.DynAnyFactoryPackage,org.omg.DynamicAny.DynAnyPackage,org.omg.IOP,org.omg.IOP.CodecFactoryPackage,org.omg.IOP.CodecPackage,org.omg.Messaging,org.omg.PortableInterceptor,org.omg.PortableInterceptor.ORBInitInfoPackage,org.omg.PortableServer,org.omg.PortableServer.CurrentPackage,org.omg.PortableServer.POAManagerPackage,org.omg.PortableServer.POAPackage,org.omg.PortableServer.portable,org.omg.PortableServer.ServantLocatorPackage,org.omg.SendingContext,org.omg.stub.java.rmi,org.w3c.dom,org.w3c.dom.bootstrap,org.w3c.dom.css,org.w3c.dom.events,org.w3c.dom.html,org.w3c.dom.ls,org.w3c.dom.ranges,org.w3c.dom.stylesheets,org.w3c.dom.traversal,org.w3c.dom.views,org.w3c.dom.xpath,org.xml.sax,org.xml.sax.ext,org.xml.sax.helpers

So any solution should work with: Oracle/OpenJDK Java 8 and Java 10 IBM/OpenJ9 Java 8 and Java 10 Memory Analyzer RCP and Memory Analyzer installed into Eclipse Photon, for the Acquire Heap Dumps ibmdump Attach API and Attach API using helper VM, and for the MAT DTFJ plugin with the latest IBM DTFJ feature for opening IBM/OpenJ9 core dumps.

robbinspg commented 6 years ago

I'm not up to date with Java9/10 and modularity but is this caused by the openj9.dtfj/module-info.java exporting the 3 dtfj packages:

module openj9.dtfj { requires transitive java.desktop; requires transitive java.logging; requires java.xml; exports com.ibm.dtfj.image; exports com.ibm.dtfj.image.j9 to openj9.dtfjview; exports com.ibm.dtfj.java; exports com.ibm.dtfj.runtime; exports com.ibm.dtfj.utils.file to openj9.dtfjview; exports com.ibm.java.diagnostics.utils to openj9.dtfjview; exports com.ibm.java.diagnostics.utils.commands to openj9.dtfjview; exports com.ibm.java.diagnostics.utils.plugins to openj9.dtfjview; }

is that automatically adding these to the osgi system package list?

robbinspg commented 6 years ago

I think this is the relevant eclipse change: https://bugs.eclipse.org/bugs/show_bug.cgi?id=527353

@ajohnson1 I assume you are running Photon? Does Oxygen MAT work for this test?