Investigate adding the Java agent at runtime

[MatthewVaadin]

According to the Java documentation, it should be possible to add a Java agent JAR to the classpath at runtime. Doing this would allow O11y Kit to be deployed as a single Maven dependency.

[MatthewVaadin]

In the bakery app, I have attempted to create a dependency jar that adds the O11y Kit agent at runtime. I have tried using two methods:

1. Java attach API

   VirtualMachine vm = VirtualMachine.attach(processId);
   vm.loadAgent(agentJar);
   vm.detatch();

   Which gives the following error: java.lang.RuntimeException: java.io.IOException: Can not attach to current VM

2. Byte buddy agent

   ByteBuddyAgent.attach(agentJar, processId);

   Which gives the following error: java.lang.IllegalStateException: Could not self-attach to current VM using external process

These look like the same issue to me.

[MatthewVaadin]

The above issue can be resolved by setting the environment variable JDK_JAVA_OPTIONS to the value -Djdk.attach.allowAttachSelf=true.

[mcollovati]

A way to work around the self-attach issue, may be to launch from the application an external java process that will do the agent load stuff.

This approach seems to work, give the following conditions

• There should be a java class with a main method that takes the agent jar path and a JVM process id as arguments and loads the agent into the process.
• The agent jar must be deployed along with the application, e.g. as part of the fat jar for Spring Boot application or in the WEB-INF/lib for WAR deployment
• The Kit starter should be able to find the agent jar, for example by searching for the OpenTelemetryAgent class file on classpath (Classloader.getResources())
• If the agent jar is nested in a far jar, it should be extracted to a temporary file
• The Kit starter should be able to find the java executable. There are multiple ways to do this: for example, it can try to get it from the current process command line, or by composing the path with the java.home system property or by using OS tools like which/where as does Flow to search npm/pnpm.
• The Kit starter should execute the java process to run the agent attachment (e.g. java -classpath xxxxx com.vaadin.extension.AgentLoader /path/to/agent.jar PID)
• The agent configuration should be provided as a properties file path to the AgentLoader class.

Some pitfalls

• It seems it is not possible to configure the agent using environment variables or system properties other than otel.javaagent.configuration-file from the AgentLoader. Environment variables and OpenTelemetry system properties seem to work only if provided ~at boot to the~ by the application process at bootstrap or at runtime.
• Executing an external process may be prevented by JVM configuration (e.g, SecurityManager)
• Even if it should be possible to access to a ResettableClassFileTransformer to "unload" the agent (through OpenTelemetry ClassFileTransformerHolder), this may not work due to JVM restrictions
• Correctly detecting java executable for every OS may be tricky

[mcollovati]

Additional note: the attach API is present only on JDK. If a JRE is used at runtime, the above code cannot be executed

[knoobie]

FYI: https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/046868.html

[mcollovati]

@knoobie thank you for the information

[heruan]

Closing this given the research results and the discussed security implications.