search

elastic / apm-agent-java

https://www.elastic.co/guide/en/apm/agent/java/current/index.html
Apache License 2.0
566 stars 321 forks source link

`IndyBootstrapDispatcher` cannot be properly loaded in the Resin environment #3807

Open dogourd opened 1 month ago

dogourd commented 1 month ago

Describe the bug

When starting Resin 3 using the provided startup script ($RESIN_HOME/bin/httpd.sh), the runtime is unable to load java.lang.IndyBootstrapDispatcher via the DynamicClassLoader.

After some investigation, I found that the startup script ultimately calls the com.caucho.boot.WatchdogProcess#createProcess method (located in resin.jar) to launch Resin. This method hardcodes the startup argument -Djava.system.class.loader=com.caucho.loader.SystemClassLoader, which replaces the default system class loader.

As a result, when Resin tries to load IndyBootstrapDispatcher, the DynamicClassLoader delegates the loading to SystemClassLoader, which is unable to load it. In Resin 3, the resin.conf file provides a tree-node element to configure special JAR file paths, allowing SystemClassLoader to access the target files. However, the internal implementation is hardcoded to fetch the target entry from the JAR file, like configuredJars.getJarEntry("java/lang/IndyBootstrapDispatcher.class").

The issue is that Elastic has shaded the class, and its actual resource path is represented as agent/bootstrap/java/lang/IndyBootstrapDispatcher.esclazz. This mismatch causes the class loading to fail, leading to malfunctioning agent functionality.

All the above is specific to Resin 3. Resin 4 might have the same problem, but I haven't tested it yet.

Steps to reproduce

  1. In the resin.conf file, add the following configuration 
    <jvm-arg>-javaagent:/path/to/elastic-apm-agent.jar</jvm-arg>
<jvm-arg>-Delastic.apm.log_level=DEBUG</jvm-arg>
<jvm-arg>-Delastic.apm.log_file=/path/to/logs/elastic-apm-agent.log</jvm-arg>
  2. download the vulns.jar to $RESIN_HOME/webapps
  3. RUN $RESIN_HOME/bin/httpd.sh start
  4. Access http://localhost:7040/vulns through your browser
JonasKunz commented 1 month ago

IndyBootstrapDispatcher is in fact not supposed to be loaded by the system-classloader. It is injected by the agent into the bootstrap-classlaoder. We specifically shade it to agent/bootstrap/java/lang/IndyBootstrapDispatcher.esclazz so that system-classloaders don't find it and then normally should delegate to the bootstrap classloader.

I tried reproducing with your instructions, unfortunately I couldn't get resin to work quickly (likely due to MacOs / some env requirements). It would be best if you could provide a github repo with a dockerfile as a self contained and portable way to reproduce.

Alternatively we could start out by you provding us the logs here, maybe this already helps me spot the issue.

dogourd commented 1 month ago

IndyBootstrapDispatcher is in fact not supposed to be loaded by the system-classloader.

Yes, but when I mentioned the system-classloader, I was referring to Resin's specific implementation. It doesn't seem to follow the parent delegation principle. As a result, when it attempts to load IndyBootstrapDispatcher, it doesn't delegate to the bootstrap-classloader, which ultimately causes the class loading to fail.

I’ve created a Dockerfile to reproduce this issue, and you can download it directly for testing. dockerfile.zip

When the /opt/resin3/resin-3.1.16/log/jvm-default.log file appears inside the container and contains the message Resin started in xxms, it indicates that Resin has started successfully. At this point, you can use tail -f to continuously monitor the log file's output. Then, visit localhost:8080/vulns in your browser to observe the file's output. Eventually, you will see the error message:

java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: java/lang/IndyBootstrapDispatcher.

Here’s the log output I observed in my environment (in case it helps):

java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: java/lang/IndyBootstrapDispatcher
    at _jsp._index__jsp.init(_index__jsp.java:80)
    at com.caucho.jsp.JspManager.loadPage(JspManager.java:404)
    at com.caucho.jsp.JspManager.compile(JspManager.java:247)
    at com.caucho.jsp.JspManager.createPage(JspManager.java:171)
    at com.caucho.jsp.JspManager.createPage(JspManager.java:150)
    at com.caucho.jsp.PageManager.getPage(PageManager.java:248)
    at com.caucho.jsp.PageManager.getPage(PageManager.java:178)
    at com.caucho.jsp.PageManager.getPage(PageManager.java:161)
    at com.caucho.jsp.QServlet.getSubPage(QServlet.java:295)
    at com.caucho.jsp.QServlet.getPage(QServlet.java:210)
    at com.caucho.server.dispatch.PageFilterChain.compilePage(PageFilterChain.java:238)
    at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:145)
    at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:187)
    at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:265)
    at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:273)
    at com.caucho.server.port.TcpConnection.run(TcpConnection.java:682)
    at com.caucho.util.ThreadPool$Item.runTasks(ThreadPool.java:743)
    at com.caucho.util.ThreadPool$Item.run(ThreadPool.java:662)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NoClassDefFoundError: java/lang/IndyBootstrapDispatcher
    ... 19 more
Caused by: java.lang.ClassNotFoundException: java.lang.IndyBootstrapDispatcher in DynamicClassLoader[SimpleLoader[/opt/resin3/resin-3.1.16/webapps/vulns/WEB-INF/work]]
    at com.caucho.loader.DynamicClassLoader.loadClass(DynamicClassLoader.java:1224)
    at com.caucho.loader.DynamicClassLoader.loadClass(DynamicClassLoader.java:1203)
    ... 19 more