Help need when agent and remote spring boot app use same classes

[gaoxingliang]

Hi, I'm using byte buddy to intercept a remote spring boot application about SQL queries. The agent will use "fastjson" to compress data and send it out to a remote endpoint. The problem is: both the agent and the remote spring boot app used same library of fastjson. and it caused problem:

My main agent logic:

        new AgentBuilder.Default()
                .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
                .disableClassFormatChanges()
                .with( //new AgentBuilder.Listener.WithErrorsOnly(
                        new AgentBuilder.Listener.WithTransformationsOnly(
                                AgentBuilder.Listener.StreamWriting.toSystemOut()))
                .type(ElementMatchers.isSubTypeOf(Statement.class))
                .transform(new AgentBuilder.Transformer.ForAdvice()

                        .advice(ElementMatchers.namedOneOf("executeQuery", "execute", "executeUpdate").and(ElementMatchers.isPublic()),
                                ExecuteAdvice.class.getName())).installOn(inst);

and then starts a thread to report fastjson to another remote server.

and the error on remote springboot app is:

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.IllegalAccessError: tried to access class com.fasterxml.jackson.databind.type.TypeBindings$TypeParamStash from class com.fasterxml.jackson.databind.type.TypeBindings
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1076) ~[spring-webmvc-5.3.2.jar!/:5.3.2]
        at org.springframework.web.servlet.DispatcherServlet.doSe

and then I found https://github.com/raphw/byte-buddy/issues/597. This seems the same issue after I changed the code to:

        new AgentBuilder.Default()
                .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION)
                .disableClassFormatChanges()
                .with( //new AgentBuilder.Listener.WithErrorsOnly(
                        new AgentBuilder.Listener.WithTransformationsOnly(
                                AgentBuilder.Listener.StreamWriting.toSystemOut()))
                .type(ElementMatchers.isSubTypeOf(Statement.class))
                .transform(new AgentBuilder.Transformer.ForAdvice()

                        .advice(ElementMatchers.namedOneOf("executeQuery", "execute", "executeUpdate").and(ElementMatchers.isPublic()),
                                ExecuteAdvice.class.getName())).installOn(inst);

        try {
            inst.appendToBootstrapClassLoaderSearch(new JarFile(main.getJarFile()));
        }
        catch (IOException e) {
            TraceLog.error("Fail to load jar " + main.getJarFile(), e);
        }

it's still got below error and didn't know what to fix this:

[Byte Buddy] ERROR com.mysql.cj.jdbc.ConnectionImpl$7 [org.springframework.boot.loader.LaunchedURLClassLoader@377dca04, null, loaded=false]
java.lang.IllegalAccessError: tried to access method net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TypeContainment$WithinMethod.<init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V from class net.bytebuddy.pool.TypePool$Default$TypeExtractor
        at net.bytebuddy.pool.TypePool$Default$TypeExtractor.visitOuterClass(TypePool.java:7746)
        at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:562)
        at net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:394)
        at net.bytebuddy.pool.TypePool$Default.parse(TypePool.java:681)
        at net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:665)
        at net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:745)
        at net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:843)Exception in thread "Thread-2" 
        at net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:912)
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection

[gaoxingliang]

from the comment https://github.com/raphw/byte-buddy/issues/795#issuecomment-567220948 and https://stackoverflow.com/questions/60237664/classpath-problems-while-instrumenting-springboot-application so do I need to write all classes needed in my own agent without import any additional jars eg (JSON) by repackaging all classes named org.json to com.xxx.json?

[raphw]

I suggest that you use a shader. If you append a library to the boot path, all applications will by default use that particular version. A Java agent is ideally not interferring with the application it is monitoring.

Another alternative is to use a trampoline agent where you attach your agent to an isolated class loader upon startup, for example by creating a URLClassLoader to which you attach your jars, instead of loading your jars as part of the actual agent.

[gaoxingliang]

@raphw Yes. I created a standalone classloader to load my agent class yesterday. but it caused No class defined error in my XXXXAdvice code. it's like the below steps:

1. create a standalone class loader and load my XXXAgent.class
2. in the XXXAgent.class, it instruments the code and added the ExecuteAdvice.
3. in the ExecuteAdvice, it tries to send out the open tracing span but it failed because the class (example Tracer) not found.
public class ExecuteAdvice {

    @Advice.OnMethodEnter(suppress = Throwable.class)
    static Span enter(@Advice.This Statement statement,
                      @Advice.AllArguments Object[] args){
        TraceLog.debug("Got sql " + Arrays.asList(args));
        // target remote url:
        try {
            if (sqlMain == null) {
                sqlMain = new SqlMain();
            }
            sqlMain.l.add("called");
            TraceLog.debug("passed sql " + Arrays.asList(args));

        catch (Throwable throwables) {
            TraceLog.error("erroor ", throwables);
            return null;
        }

    }

    @Advice.OnMethodExit(suppress = Throwable.class, onThrowable = Throwable.class)
    static void exit(@Advice.Enter Span span, @Advice.Thrown Throwable th){
        if (span != null) {
            // send span out
        }
    }
}

I guess it's caused by the advice is executing in the spring's class loader and while my Tracer is built on the standalone classloader. do I need to put all my classes (used in the XXXAdvice) to the bootstrap class loader? and is this the only way to solve this?

[raphw]

You need to distinguish here. You should isolate your class loader, but at the same time you should put shared infrastructure like your TraceLog into a seperate jar that is loaded by the boot loader. Instrumentation allows you to append classes to it, otherwise, isolate your code as much as possible.

[gaoxingliang]

@raphw Thanks for your idea. I solved this problem by: (1) minimal the core classes and load into the bootstrap classloader. (2) for other classes, loaded into another normal standalone classloader which loaded other agent classes. (referred https://github.com/alibaba/arthas)

[XhstormR]

@raphw Thanks for your idea. I solved this problem by: (1) minimal the core classes and load into the bootstrap classloader. (2) for other classes, loaded into another normal standalone classloader which loaded other agent classes. (referred https://github.com/alibaba/arthas)

@gaoxingliang Hi, I got the same error but I don't know how to fix it, can you paste a sample code?

[gaoxingliang]

@XhstormR here is the repo https://github.com/gaoxingliang/tracing-research/ and the main class & idea:

1. the spy.java which records the loaded plugins: https://github.com/gaoxingliang/tracing-research/blob/main/spy/src/main/java/com/zoomphant/agent/trace/spy/Spy.java
2. the bootstrap.java : https://github.com/gaoxingliang/tracing-research/blob/main/bootstrap/src/main/java/com/zoomphant/agent/trace/boost/Bootstrap.java