OutOfMemoryError at sun.instrument.InstrumentationImpl.appendToClassLoaderSearch0(Native Method)

[xiongchun]

Hey,

i install my opentelemetry agent to websphere v8.5.5.5.21. when i startup the OutOfMemoryError occured, BUT the target system is startup normally.

Some info as follows:

ERROR io.opentelemetry.javaagent.OpenTelemetryAgent
java.lang.OutOfMemoryError
    at sun.instrument.InstrumentationImpl.appendToClassLoaderSearch0(Native Method)
    at sun.instrument.InstrumentationImpl.appendToBootstrapClassLoaderSearch(InstrumentationImpl.java:214)
    at io.opentelemetry.javaagent.OpenTelemetryAgent.installBootstrapJar(OpenTelemetryAgent.java:86)
    at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:54)
    at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
    at java.lang.reflect.Method.invoke(Method.java:508)
    at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:408)
    at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:423)

BTW: setted -Xms8000m -Xmx8000m -XX:MetaspaceSize=1024m -XX:MaxMetaspaceSize=2048m -Xmn512m -Xss512k

someone can give me some advice? Thanks a lot!

[laurit]

@xiongchun where did you add the jvm arguments to enable the agent?

[xiongchun]

@xiongchun where did you add the jvm arguments to enable the agent?

In websphere console ui. And i am sure jvm arguments have been saved to server.xml, and be loaded normally.

Add some information: my agent.jar is about 40M package by gradle. it is OutOfMemoryError all the time when i startup. BUT, if i unzip the agent.jar and then rezip it，the agent.jar is decrease to 20M, the server Started with no OutOfMemoryError.

i don't know whether the OutOfMemoryError is throwed by c/c++ called by appendToClassLoaderSearch0(Native Method) ???

[laurit]

If this only happens with your custom agent and not with the vanilla otel agent then there isn't much we can do to help you. I'd suggest you firstly check whether you get any diagnostic dumps from the jvm when the oom happens (if I remember correctly ibm vm prints to stdout/stderr when it generates the dump) and see if you can figure out something form there. Consider contacting your jvm vendor (I presume it is IBM) for assistance. Alternatively you could attempt to reproduce the same issue on openj9 and if it reproduces report issue to openj9. As openj9 source code is available you could also build your own version of openj9 and debug it to understand why it ooms. If you believe that it is the size of the jar that causes this you could attempt to reduce the size of your custom agent jar. You could also experiment with adding the agent jar to boot class path with -Xbootclasspath/pand modify OpenTelemetryAgent to skip calling appendToBootstrapClassLoaderSearch when it is already on boot class path (OpenTelemetryAgent.class.getClassLoader() == null)

[breedx-splk]

Also @xiongchun please indicate the version and flavor of the JVM that you are using. Do you have any other agents running alongside the java agent? Why would re-zipping the jar result in something half the size? That's fishy....

[xiongchun]

@laurit @breedx-splk Thanks for your professional advice. I compressed the file in the packaging process with gradle, and let the agent jar was about 20M, so there would be no error. I'll do more in-depth research under your advice later.