Class retransformation doesn't work for dynamic agent on Java 11

[SerVB]

Hey!

It seems like dynamic retransformations of classes work for me only on Java 8 but not on Java 11. In the latter case, I get exceptions from javassist about different not found standard Java classes, for example, the ones directly referenced by me or even from the signature of the method-to-transform.

What should I do to fix that on Java 11? I want to transform classes dynamically here too.

For illustration purposes, I've created a repro file. Here I retransform two classes: one is my own, another is system. I've created both agentmain and premain to compare. Dynamic variant is executed when a main argument is passed to the app (I pass it as just "o"). After the retransformation, I call two methods (of my own class and of the system one). When the transformation is successful, I receive additional logging ("hi-" and "scaled!").

// MyMain.java
package mypackage;

import com.sun.tools.attach.VirtualMachine;
import javassist.*;
import javassist.bytecode.AccessFlag;
import sun.java2d.SunGraphics2D;
import sun.java2d.SurfaceData;

import javax.swing.*;
import java.awt.*;
import java.awt.geom.AffineTransform;
import java.awt.image.BufferedImage;
import java.io.IOException;
import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.lang.instrument.UnmodifiableClassException;
import java.lang.management.ManagementFactory;
import java.security.ProtectionDomain;

public class MyMain {

    public static void premain(String args, Instrumentation inst) {
        System.out.println("premain start");
        inst.addTransformer(new MyFormer(), true);
        try {
            inst.retransformClasses(MyMain.class);
        } catch (UnmodifiableClassException e) {
            e.printStackTrace();
        }
        System.out.println("premain end");
    }

    public static void agentmain(String args, Instrumentation inst) {
        System.out.println("agentmain start");
        inst.addTransformer(new MyFormer(), true);
        try {
            inst.retransformClasses(MyMain.class);
        } catch (UnmodifiableClassException e) {
            e.printStackTrace();
        }
        System.out.println("agentmain end");
    }

    public static void main(String[] args) {
        if (args.length > 0) {
            attachToThisVm();
        }

        Frame f = new JFrame();
        f.setVisible(true);

        System.out.println(new MyMain().hi());

        SunGraphics2D system = new SunGraphics2D(SurfaceData.getPrimarySurfaceData(new BufferedImage(10, 10, BufferedImage.TYPE_INT_ARGB)), Color.BLACK, Color.WHITE, Font.getFont("System"));
        system.drawRenderedImage(null,new AffineTransform() {

            @Override
            public void setToScale(double sx, double sy) {
                super.setToScale(sx, sy);
                System.out.println("scaled!");
            }
        });
    }

    public static void attachToThisVm() {
        System.out.println("dynamically loading javaagent");
        String name = ManagementFactory.getRuntimeMXBean().getName();
        int p = name.indexOf('@');
        String pid = name.substring(0, p);

        try {
            VirtualMachine vm = VirtualMachine.attach(pid);
            vm.loadAgent("javaAgentTest-1.0-SNAPSHOT.jar", null);
            vm.detach();
        }
        catch (Exception e) {
            throw new RuntimeException(e);
        }
        System.out.println("dynamically loaded javaagent");
    }

    public int hi() {
        return 3;
    }

    public static class MyFormer implements ClassFileTransformer {

        @Override
        public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {
            return transformClass(className, classfileBuffer);
        }

        private byte[] transformClass(String className, byte[] buffer) {
            if ("mypackage/MyMain".equals(className)) {
                System.out.println(className);

                ClassPool cp = ClassPool.getDefault();
                String name = className.replace("/", ".");
                cp.insertClassPath(new ByteArrayClassPath(name, buffer));
                try {
                    CtClass clazz = cp.get(name);
                    CtBehavior[] declaredBehaviors = clazz.getDeclaredBehaviors();
                    for (CtBehavior db : declaredBehaviors) {
                        if ("hi".equals(db.getName())) {
                            if ((db.getMethodInfo().getAccessFlags() & AccessFlag.STATIC) != 0) {
                                System.out.println("bad access flags, skipping...");
                                return buffer;
                            }

                            System.out.println("Forming hi...");
                            db.insertBefore("System.out.print(\"hi-\");");  // crashes on 11, direct usage case, even referencing to java.lang.Object will crash
                        }
                    }

                    return clazz.toBytecode();
                } catch (Throwable e) {
                    e.printStackTrace();
                    System.out.println("error");
                    return buffer;
                }
            }

            if ("sun/java2d/SunGraphics2D".equals(className)) {
                System.out.println(className);
                ClassPool cp = ClassPool.getDefault();
                String name = className.replace("/", ".");
                cp.insertClassPath(new ByteArrayClassPath(name, buffer));
                try {
                    CtClass clazz = cp.get(name);
                    CtBehavior[] declaredBehaviors = clazz.getDeclaredBehaviors();
                    for (CtBehavior db : declaredBehaviors) {
                        if ("sun.java2d.SunGraphics2D.drawRenderedImage(java.awt.image.RenderedImage,java.awt.geom.AffineTransform)".equals(db.getLongName())) {
                            if ((db.getMethodInfo().getAccessFlags() & AccessFlag.STATIC) != 0) {
                                System.out.println("bad access flags, skipping...");
                                return buffer;
                            }

                            System.out.println("Forming drawRenderedImage...");
                            db.insertBefore("$2.setToScale(2.0, 2.0);");  // crashes on 11, signature case
                        }
                    }
                    return clazz.toBytecode();
                } catch (NotFoundException | CannotCompileException | IOException e) {
                    e.printStackTrace();
                    return buffer;
                }
            }

            return buffer;
        }
    }
}

I build jar via Gradle using Java 11:

// build.gradle, module name is javaAgentTest
plugins {
    id 'java'
}

group 'org.example'
version '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

sourceCompatibility = "1.8"
targetCompatibility = "1.8"

def inline = { deps -> deps.collect { it.isDirectory() ? it : zipTree(it) } }

jar {
    manifest {
        attributes(
                "Can-Redefine-Classes": true,
                "Can-Retransform-Classes": true,
                "Premain-Class": "mypackage.MyMain",
                "Agent-Class": "mypackage.MyMain",
        )
    }

    from {
        inline(configurations.runtimeClasspath)  // fat jar
    }
}

dependencies {
    implementation "org.javassist:javassist:3.27.0-GA"
}

On Java 8, both static and dynamic variants work:

$ java -version
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~20.04-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)
$ java -cp javaAgentTest-1.0-SNAPSHOT.jar -javaagent:javaAgentTest-1.0-SNAPSHOT.jar mypackage.MyMain
premain start
mypackage/MyMain
Forming hi...
premain end
sun/java2d/SunGraphics2D
Forming drawRenderedImage...
hi-3
scaled!
$ java -cp javaAgentTest-1.0-SNAPSHOT.jar:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar mypackage.MyMain o
dynamically loading javaagent
agentmain start
mypackage/MyMain
Forming hi...
agentmain end
dynamically loaded javaagent
sun/java2d/SunGraphics2D
Forming drawRenderedImage...
hi-3
scaled!

On Java 11, dynamic variant doesn't work (it will fork for hi method if there is no reference to System.out, for example, just db.insertBefore("return 22;");):

$ java -version
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)
$ java -cp javaAgentTest-1.0-SNAPSHOT.jar -javaagent:javaAgentTest-1.0-SNAPSHOT.jar mypackage.MyMain
premain start
mypackage/MyMain
Forming hi...
premain end
sun/java2d/SunGraphics2D
Forming drawRenderedImage...
hi-3
scaled!
$ java -cp javaAgentTest-1.0-SNAPSHOT.jar -Djdk.attach.allowAttachSelf=true mypackage.MyMain o
dynamically loading javaagent
agentmain start
mypackage/MyMain
Forming hi...
javassist.CannotCompileException: [source error] no such class: System.out
        at javassist.CtBehavior.insertBefore(CtBehavior.java:806)
        at javassist.CtBehavior.insertBefore(CtBehavior.java:766)
        at mypackage.MyMain$MyFormer.transformClass(MyMain.java:112)
        at mypackage.MyMain$MyFormer.transform(MyMain.java:91)
        at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
        at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
        at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
        at mypackage.MyMain.agentmain(MyMain.java:38)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
Caused by: compile error: no such class: System.out
        at javassist.compiler.MemberResolver.searchImports(MemberResolver.java:479)
        at javassist.compiler.MemberResolver.lookupClass(MemberResolver.java:422)
        at javassist.compiler.MemberResolver.lookupClassByJvmName(MemberResolver.java:329)
        at javassist.compiler.TypeChecker.atCallExpr(TypeChecker.java:711)
        at javassist.compiler.JvstTypeChecker.atCallExpr(JvstTypeChecker.java:170)
        at javassist.compiler.ast.CallExpr.accept(CallExpr.java:49)
        at javassist.compiler.CodeGen.doTypeCheck(CodeGen.java:266)
        at javassist.compiler.CodeGen.atStmnt(CodeGen.java:360)
        at javassist.compiler.ast.Stmnt.accept(Stmnt.java:53)
        at javassist.compiler.Javac.compileStmnt(Javac.java:578)
        at javassist.CtBehavior.insertBefore(CtBehavior.java:786)
        ... 15 more
error
agentmain end
dynamically loaded javaagent
sun/java2d/SunGraphics2D
Forming drawRenderedImage...
javassist.CannotCompileException: cannot find java.awt.image.RenderedImage
        at javassist.CtBehavior.insertBefore(CtBehavior.java:803)
        at javassist.CtBehavior.insertBefore(CtBehavior.java:766)
        at mypackage.MyMain$MyFormer.transformClass(MyMain.java:140)
        at mypackage.MyMain$MyFormer.transform(MyMain.java:91)
        at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
        at java.desktop/sun.java2d.loops.GraphicsPrimitiveMgr.<clinit>(GraphicsPrimitiveMgr.java:56)
        at java.desktop/sun.java2d.loops.Blit.<clinit>(Blit.java:114)
        at java.desktop/sun.java2d.xr.XRPMBlitLoops.register(XRPMBlitLoops.java:46)
        at java.desktop/sun.java2d.xr.XRSurfaceData.initXRSurfaceData(XRSurfaceData.java:106)
        at java.desktop/sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:124)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.desktop/sun.awt.X11GraphicsEnvironment.<clinit>(X11GraphicsEnvironment.java:61)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:315)
        at java.desktop/java.awt.GraphicsEnvironment$LocalGE.createGE(GraphicsEnvironment.java:101)
        at java.desktop/java.awt.GraphicsEnvironment$LocalGE.<clinit>(GraphicsEnvironment.java:83)
        at java.desktop/java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:129)
        at java.desktop/java.awt.Window.initGC(Window.java:487)
        at java.desktop/java.awt.Window.init(Window.java:507)
        at java.desktop/java.awt.Window.<init>(Window.java:549)
        at java.desktop/java.awt.Frame.<init>(Frame.java:423)
        at java.desktop/java.awt.Frame.<init>(Frame.java:388)
        at java.desktop/javax.swing.JFrame.<init>(JFrame.java:180)
        at mypackage.MyMain.main(Unknown Source)
Caused by: javassist.NotFoundException: java.awt.image.RenderedImage
        at javassist.ClassPool.get(ClassPool.java:430)
        at javassist.bytecode.Descriptor.toCtClass(Descriptor.java:571)
        at javassist.bytecode.Descriptor.getParameterTypes(Descriptor.java:424)
        at javassist.CtBehavior.getParameterTypes(CtBehavior.java:323)
        at javassist.CtBehavior.insertBefore(CtBehavior.java:781)
        ... 25 more
3

Please take a look. It was initially asked here: https://stackoverflow.com/questions/64340794/class-retransformation-doesnt-work-for-dynamic-agent-on-java-11. Looks like a bug to me and to a community member.

[SerVB]

I've investigated a bit. There is a difference in System classpath appending:

https://github.com/jboss-javassist/javassist/blob/1c4e31b9677d020a1e89aeebc6686396f9e6c68a/src/main/javassist/ClassPoolTail.java#L250-L255

With static agent, this code is executed in Thread[main,5,main] and its cl is ClassLoaders$AppClassLoader.

With dynamic agent, this code is executed in Thread[Attach Listener,9,system] and its cl is null.

So in the second case, indeed there are no system classes visible by Javassist.

As the result, I've come up with placing the following code to beginning of agentmain:

try {
    Method m = Thread.class.getDeclaredMethod("getThreads");
    m.setAccessible(true);
    Thread[] threads = (Thread[]) m.invoke(null);

    for (Thread t : threads) {
        if ("main".equals(t.getName())) {
            ClassPool.getDefault().appendClassPath(new LoaderClassPath(t.getContextClassLoader()));
            System.out.println("cp from main thread is appended");
            break;
        }
    }
} catch (IllegalAccessException | InvocationTargetException | NoSuchMethodException e) {
    e.printStackTrace();
}

Now dynamic variant works, so it can be used as a workaround 😉

What do you think, should this be fixed in the library or is it designed behavior?

[kriegaex]

I think I solved your problem by refactoring for separation of concerns, see my answer on StackOverflow.

It might still be worth for the maintainer to investigate this issue on the Javassist side, but I think this is optional.

[kriegaex]

Method m = Thread.class.getDeclaredMethod("getThreads");
m.setAccessible(true);

The idea to access a the private method Thread.getThreads() is not so nice. There are alternatives I want to mention just in case the Javassist maintainer wants to improve the situation for the attach listener situation discussed here:

1. Enumerate threads in thread groups

We can utilise the knowledge that the Attach Listener thread is in the system thread group and the main thread in its own main thread group which has the system group as a parent. There is no direct way to iterate over child groups, but several ThreadGroup methods to copy child thread groups and/or child threads (optionally recursively) into predefined arrays. So it would be possible to

• determine if the currently running thread is indeed Attach Listener / system,
• enumerate all threads recursively (or optionally find child thread group main first and enumerate only there),
• filter for the main / main thread,
• get its context class loader.

A similar approach is described here, you can easily adjust it to your needs.

2. Get a set of threads indirectly (inefficient, but easy to code)

You can use Thread.getAllStackTraces().keySet() and then filter the set via stream API in order to find the main thread. I took this idea from here.

[SerVB]

Thanks! I think at least there should be a clear message describing possible reasons why this is happened and how one can try to solve it.

[kkofler]

You can use Thread.getAllStackTraces().keySet() and then filter the set via stream API in order to find the main thread.

You don't even need the stream API, you can just copy&paste the ranged for loop that was posted, it works just as well on a Set<Thread>, i.e.:

    Set<Thread> threads = Thread.getAllStackTraces().keySet();
    for (Thread t : threads) {
        if ("main".equals(t.getName())) {
            ClassPool.getDefault().appendClassPath(new LoaderClassPath(t.getContextClassLoader()));
            System.out.println("cp from main thread is appended");
            break;
        }
    }

(and the try is not needed).

[kkofler]

That said, this is only a workaround and this really ought to be fixed within javassist.

[kriegaex]

You can use Thread.getAllStackTraces().keySet() and then filter the set via stream API in order to find the main thread.

You don't even need the stream API

I did not say you need the stream API, I said you can use it. It is just a different, more functional programming style. If you prefer a more procedural approach with for loops and if conditions inside instead of filtering a stream, be my guest. I am just wondering how your post adds value to this discussion.