How to reduce our startup time overhead of instrumentation

[natfitz]

We are working on a Java agent to inject tracing information in various RPC libraries, and we have been getting reports that the agent is causing significant increase in startup times for larger applications. In some anecdotal cases start up used to be 30 seconds but is now 10 minutes with the agent installed.

The initialization of the transformers

        new HttpServletTransformer(this.instrumentation);
        new RequestDispatcherTransformer(this.instrumentation);

        //SQL DATABASE
        new ConnectionTransformer(this.instrumentation);
        new PreparedStatementTransformer(this.instrumentation);
        new PreparedInsertStatementTransformer(this.instrumentation);
        new PreparedExecuteStatementTransformer(this.instrumentation);

        //Apache HTTP CLIENT
        new HttpClientTransformer(this.instrumentation);
        new HttpClientUriRequestTransformer(this.instrumentation);

        //OkHttp Client
        new OkHttpBuilderTransformer(this.instrumentation);
        new OkHttpCallTransformer(this.instrumentation);

        //HttpURLConnection & RestTemplate
        new HttpURLConnectionConnectTransformer(this.instrumentation);
        new HttpURLConnectionGetInputStreamTransformer(this.instrumentation);
        new HttpURLConnectionGetOutputStreamTransformer(this.instrumentation);

        //JMS Queues
        new JMSProducerTransformer(this.instrumentation);
        new JMSConsumerTransformer(this.instrumentation);

Example of a transformer

public class HttpURLConnectionGetInputStreamTransformer implements AgentBuilder.Transformer {

    public HttpURLConnectionGetInputStreamTransformer(Instrumentation instrumentation) {

        new AgentBuilder.Default()
                .ignore(ElementMatchers.nameStartsWith("net.bytebuddy."))
                .type(ElementMatchers.isSubTypeOf(java.net.HttpURLConnection.class))
                .transform(this)
                .installOn(instrumentation);
    }

    @Override
    public DynamicType.Builder<?> transform(DynamicType.Builder<?> builder, TypeDescription typeDescription, ClassLoader classLoader, JavaModule module) {
        return builder.visit(Advice.to(HttpURLConnectionGetInputStreamAdvice.class).on(ElementMatchers.named("getInputStream")));
    }
}

From what we can tell each transformer adds a small bit of overhead but with us creating so many the time is adding up to be quite significant. Is there a way to accomplish this without scanning all of the classes for each transformer?

[raphw]

The agent builders can be chained, simply add .asDecorator() to the end after .transform() and only install a single transformer. Every transformer has a large overhead and you should avoid piling them up.

[natfitz]

Interesting...I was able to get a decent reduction in overhead with removing some duplication of transformers like so

   @Override
    public DynamicType.Builder<?> transform(DynamicType.Builder<?> builder, TypeDescription typeDescription, ClassLoader classLoader, JavaModule module) {

        return builder
                .visit(Advice.to(HttpURLConnectionGetInputStreamAdvice.class)
                        .on(ElementMatchers.named("getInputStream")))
                .visit(Advice.to(HttpURLConnectionConnectAdvice.class)
                        .on(ElementMatchers.named("connect")))
                .visit(Advice.to(HttpURLConnectionGetOutputStreamAdvice.class)
                        .on(ElementMatchers.named("getOutputStream")));
    }

So am I understanding correctly that the suggestion is to have one transformer that contains all of the .visit() and I would reuse it across the agent builder like so

        AgentBuilder agentBuilder =
                new AgentBuilder.Default()
                    .ignore(none())
                .type(named("javax.servlet.http.HttpServlet")).and(isPublic())
                    .transform(new UniversalTransformer(instrumentation))
                    .asDecorator()
                .type(named("javax.servlet.RequestDispatcher"))
                    .transform(new UniversalTransformer(instrumentation))
                    .asDecorator()

[raphw]

Yes, doing so, Byte Buddy does not need to parse the byte code multiple times and can reuse a single serialization.

[natfitz]

Thanks for the help, with the changes in place I'm seeing reduction of at least 50% on start up time of my small test apps.

[maneeshbhunwal123]

Hi @raphw ,

i am not getting .asDecorator() as a method. I even checked this file. Can you please help me with if this is still supported?

[raphw]

It's the default now and you can use asTerminal() to get the previous behaviour in recent versions.