OpenLiberty versions with Jakarta 9+ load some application jar files from the wrong location

[yogregg]

I have a Jakarta EE 9+ war file, and am testing it on the Liberty versions that should support it (21/22/23 configured with the appropriate webProfile-N feature -- specific versions are listed below). Our web applications have a defensive feature where they check where each of the jar files in their WEB-INF/lib directory get loaded from, since we've had problems on some appserver versions that we support with the appserver loading some jar files from its own locations instead of the one that we include in WEB-INF/lib, and the application malfunctions as a result.

I've never had this problem show up on Liberty before, but now that I am testing on Liberty versions 21/22/23 with webProfile-9.1/10, I am seeing some of the jar files being loaded from a location other than our WEB-INF/lib and the application refuses to start due to the action of our checker. Expected behavior is that it loads classes that exist in the war file's WEB-INF/lib from the jar files included in WEB-INF/lib.

Specifically, in versions 21 and 22, I am seeing two cases where the jar file isn't being loaded from WEB-INF/lib:

1. WEB-INF/lib includes jakarta.xml.bind-api-3.0.1.jar, but when our checker loads a trial class from that jar (the specific one it uses is jakarta.xml.bind.Binder), it reports that it didn't get loaded from that WEB-INF/lib jar file, but instead was loaded from "bundleresource://68.fwk296737429/jakarta/xml/bind/Binder.class", which I am guessing is some version that Liberty itself contains.
2. WEB-INF/lib includes reactive-streams-1.0.3.jar, but when our checker loads a trial class from that jar (the specific one it uses is org.reactivestreams.Publisher), it reports that it didn't get loaded from that WEB-INF/lib jar file, but instead was loaded from "bundleresource://216.fwk296737429/org/reactivestreams/Publisher.class".

In Liberty 23, I am also having this issue, but in that case it is only with reactive-streams, not jakarta.xml.bind-api.

I have essential the same war file that uses the javax namespace rather than the jakarta namespace, and I've tried deploying that to the same three Liberty versions but using the webProfile-8.0 feature. I am not seeing any classloading problems in that case.

Liberty versions tested:

• 21.0.0.12 with feature webProfile-9.1
• 22.0.0.1 with feature webProfile-9.1
• 23.0.0.7 with feature webProfile-10.0 java.version = 17.0.1 java.runtime = IBM Semeru Runtime Open Edition (17.0.1+12)

Here is my server.xml from the 21.0.0.12 case. The others are similar but use the webProfile feature versions noted above.

<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">

    <!-- Enable features -->
    <featureManager>
        <feature>webProfile-9.1</feature>
    </featureManager>

    <!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
    <httpEndpoint id="defaultHttpEndpoint"
                  host="*"
                  httpPort="9380"
                  httpsPort="9343" />

    <!-- Automatically expand WAR files and EAR files -->
    <applicationManager autoExpand="true"/>

    <!-- Default SSL configuration enables trust for default certificates from the Java runtime --> 
    <ssl id="defaultSSLConfig" trustDefaultCerts="true" />

    <webContainer deferServletLoad="false"/>
        <mimeTypes>
        <type>svg=image/svg+xml</type>
        <type>svgz=image/svg+xml</type>
        <type>css=text/css</type>
    </mimeTypes>
</server>

[jhanders34]

The default Liberty classloader policy is the same as Java's default policy, i.e. parentFirst. parentFirst means that it will use the Liberty runtime classloader and APIs exposed by the Liberty runtime clasloader before the application once since the liberty runtime is the parent classloader to an application. If you want to use a different policy, you need to configure it in your server.xml to do parentLast. With webProfile 9.1 the XML Binding APIs are added to the runtime for applications to use so that is why they are loaded from Liberty runtime instead of from your application. This has been corrected in webProfile 10 to not expose the XML Binding APIs with the webProfile 10 convenience feature. You can read about parentLast configuration for your application's classloader in the Open Liberty documentation here: https://openliberty.io/docs/latest/reference/config/application.html

[yogregg]

Thank you @jhanders34 for the very quick reply. I've now tried setting classloader to parentLast in my server.xml (on 21.0.0.12 with feature webProfile-9.1) and am still seeing reactive-stream.jar not being loaded from WEB-INF/lib.

I tried two different syntaxes:

<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">
    <!-- Enable features -->
    <featureManager>
        <feature>webProfile-9.1</feature>
    </featureManager>

    ....
    <classloader delegation="parentLast"/>
    ...
</server>

and also this:

<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">
    <!-- Enable features -->
    <featureManager>
        <feature>webProfile-9.1</feature>
    </featureManager>

    ....
    <application location="myApp.war" name="myApp">
      <classloader delegation="parentLast"/>
    </application>
    ...
</server>

Per the documentation, both of these look valid. Neither of these worked. Did I get the syntax right? After making the changes, I stopped and restarted Liberty.

By the way, of these two, the second one where you must list specific applications really wouldn't be ok with our customers, who expect to just be able to deploy the war files without having to do other application-specific reconfiguration of their application servers. But I tried it anyways.

When I saw in the documentation that the classloader element is also allowed at the top level of server.xml, my assumption was that that is supposed to set the default for all applications, which would be much better. But the doc doesn't really say. Is that what it means when classloader is used at the top level?

What's the correct syntax? If the above are both right, something else must be going on here.

[yogregg]

After having run into a very similar issue in #28414 I'm adding some additional details here along with an easy way to reproduce the problem I'm seeing. To recap, when using webProfile-10.0 in 23.0.0.7, classes in the reactive-streams-1.0.3.jar that is in my web application's WEB-INF/lib are getting loaded out of an internal Liberty location instead of the (different) version of the jar in my WEB-INF/lib. If instead I configure my server with a smaller set of features (this is undesirable) that includes only servlet-6.0, pages-3.1 and websocket-2.1, it correctly loads the classes from the jar in WEB-INF/lib.

(My original problem report above also talks about a problem with jakarta.xml.bind-api. Please disregard that, it is not the aspect of the problem that I'm concerned about. It is the reactive-streams issue that is an ongoing problem for me.)

After conversations with @neuwerk in #28414 and seeing other similar classloading problems deemed to be bugs not features, I now see that it really isn't valid for Liberty to load random classes like the ones in org.reactivestreams:reactive-streams from its own internal classes rather than from the web application's WEB-INF/lib. The Jakarta EE spec allows application servers to load things that are part of the Jakarta EE spec from their own copies of those classes, but the same isn't true for arbitrary other classes like the ones in reactive-streams. After @neuwerk 's excellent work in fixing the rather similar problem in the other issue, I'd love to see this one get fixed too, but it has languished for nearly a year now.

I've attached two things to help reproduce this. First is openliberty-issue-25838.zip, which contains sources for a very simple maven war file project that allows you to reproduce the issue. Unpack that, do "mvn clean package" and deploy the resulting target/liberty_problem_202307-1.0-SNAPSHOT.war file to a Liberty in the version range I tried that has just the webProfile-10.0 feature enabled and that has -verbose:class in its jvm.options to turn on classloader tracing. When the web application starts, you will see a line like this in console.log:

class load: org.reactivestreams.Publisher from: file:/C:/apps/liberty-24.0.0.6-nightly-snapshot-cl240720240605-1201/wlp/lib/../dev/api/stable/com.ibm.websphere.org.reactivestreams.reactive-streams.1.0_1.1.90.jar

I have also attached the full console.log for starting the web application. To confirm that this problem still exists in current code, I used a nightly build from today to reproduce this.

openliberty-issue-25838.zip liberty_problem_202307_console.log

[jhanders34]

Thanks for the new details @yogregg. The reason that using webProfile-9.1 causes this problem is that the org.reactivestreams package is exposed to the application classloader with the restfulWSClient-3.0 and restfulWS-3.0 features. I believe this was done for some of the RESTEasy function that supports reactive functions. I will defer to @jim-krueger and @WhiteCat22 to provide details on why that package is exposed with type of internal. If it is not needed to be exposed to the application class loader that would resolve your problem.

Liberty now has MicroProfile Reactive Streams 3.0 function that works with Jakarta EE 9 and 10 so you do not need to include it in your application. If it is an option for you to use this new function, you can add the mpReactiveStreams-3.0 feature to your server.xml file.

Regarding your previous question about configuring the classloader delegation, I don't believe you can configure it at the top level and you need to do it for each application. @jaridk can you confirm that?

[yogregg]

Hi @jhanders34 thanks for the quick reply. I'm not so concerned with webProfile-9.1 anymore, since we've moved on to webProfile-10.0 in our applications. The problem exists there as I described in my last entry. Maybe the reason is the same. I can't remove reactive-streams.jar from our war file because our product support many different application server products and it isn't a standard part of Jakarta EE 10's Web Profile and so is not present in all appservers.

My main point is that the fact that Liberty is not loading the versions of these classes that ship in my war file's WEB-INF/lib is a bug, and that I can see from the issues history that the Liberty team has fixed a number of similar pollution of the web application's class loader before, and so I'd like to see this fixed. As I mentioned, a similar issue I had recently reported in #28414 was just fixed. It can in general cause quite a bit of trouble for a web application when the application server loads its own version of classes rather than what ships with the application. The versions of the classes can be different, and the ones in the application server product can be incompatible with how the web application is built to use the library. It makes it impossible to know whether a web application will actually run as designed, which of course is completely counter to the point of having standards like Jakarta EE in the first place. The Jakarta EE specs are very clear that compliant implementations must use the classes from the war/ear/rar files. For example, https://jakarta.ee/specifications/platform/10/jakarta-platform-spec-10.0#a3046 has a long list of web container class loading requirements that in part specifies what classes and resource the application must have access to. That list includes this item: "The content of all jar files in the WEB-INF/lib directory of the containing war file, but not any subdirectories". I have reactive-streams.jar in my WEB-INF/lib, but Liberty is not giving my application access to the classes in it.

The Servlet 6 specification reinforces this requirement. https://jakarta.ee/specifications/servlet/6.0/jakarta-servlet-spec-6.0#web-application-class-loader says, in part, "The container should not allow applications to override or access the container's implementation classes." (the bolding is mine).

I can work around this by not including all of webProfile-10.0 features in server.xml, instead just specifying a smaller subset that I require that apparently doesn't include whatever triggers this issue. But I shouldn't have to, because I've written an application that complies with Jakarta EE 10, and it should run in an appserver that claims Jakarta EE 10 compliance. Moreover, it is awkward to have to tell my customers that they have to reconfigure their Liberty servers to run our applications, and inconvenient for them to have to do that. That isn't true for any of the other application server products that we support, and it isn't a good look for Liberty. I'd like to see that fixed, because Liberty is otherwise one of the easier to use application servers, and I'd like to be able to promote it as a solid choice for our customers.

[jim-krueger]

Hi @yogregg. Unlike the other Liberty situations you reference, JAXRS is not in a position to remove these classes from being exposed to the application. The Jakarta Rest (JAXRS) specification has the following, documenting this support and including an example: https://jakarta.ee/specifications/restful-ws/3.1/jakarta-restful-ws-spec-3.1#reactive_api_extensions. In addition, mpRestClient 2.0 and 3.0 expose the API as stable.

We therefore have no choice other than to continue to expose these classes. To do otherwise would cause backward compatibility issues and break existing users who rely on this support.

As stated earlier in this issue, the best option available to you is to use parent last classloading to ensure your copy of the API is available to your application, or reduce your feature set, as is normally advised, to keep your server right sized to your application needs.