CharacterEscapeHandler is not working on liberty

[soilworker01]

Describe the bug
When using the jaxb-2.2 feature of liberty it's not possible to add a CharacterEscapeHandler(com.sun.xml.bind.marshaller.CharacterEscapeHandler) to the marshaller. Sample Code

        JAXBContext jaxbContext     = JAXBContext.newInstance( My.class );
        Marshaller jaxbMarshaller   = jaxbContext.createMarshaller(); 
        jaxbMarshaller.setProperty("com.sun.xml.bind.marshaller.CharacterEscapeHandler",
                new MyCharacterEscapeHandler());

MarshallerImpl.setProperty

 public void setProperty(String name, Object value) throws PropertyException {
        if( INDENT_STRING.equals(name) ) {
            checkString(name, value);
            indent = (String)value;
            return;
        }
        if( ENCODING_HANDLER.equals(name) || ENCODING_HANDLER2.equals(name)) {
            if(!(value instanceof CharacterEscapeHandler)) 
                throw new PropertyException(
                    Messages.MUST_BE_X.format(
                            name,
                            CharacterEscapeHandler.class.getName(),
                            value.getClass().getName() ) );
            escapeHandler = (CharacterEscapeHandler)value;
            return;
        }

if(!(value instanceof CharacterEscapeHandler)) This will always fail on liberty with activated jaxb-2.2 feature because the Marshaller(Impl) is loaded with the equinox classloader of openliberty and MyCharacterEscapeHandler is loaded with the application class loader.

It leads to: javax.xml.bind.PropertyException:property "com.sun.xml.bind.marshaller.CharacterEscapeHandler" must be an instance of type com.sun.xml.bind.marshaller.CharacterEscapeHandler, not ...MyCharacterEscapeHandler

Diagnostic information:

• OpenLiberty Version: [24.0.0.1+]
• Affected feature(s) [jaxb-2.2]
• Java Version: [several openjdk 11 distributions]

Additional context
Workaround was to don't use the jaxb-2.2 feature of liberty. We added the relevant jaxb jars to our webapplication. This was working fine until we have upgraded liberty to 24.0.0.2. After the upgrade it was not possible to start the webapplication because no JaxbContextFactory could be found.

Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBContextFactory cannot be found by com.ibm.ws.jaxb.tools_1.0.86.cl240220240212-1928

[neuwerk]

@soilworker01 thanks for bringing this to our attention!

So it sounds like two issues:

1. CharacterEscapeHandler not being available to implement through the jaxb-2.2 feature

You may, or may not, be aware that this is an intentional part of the jaxb-2.2 feature design that the implementation classes aren't available to the application class path. That being said, it seems reasonable to expect that the API Marshaller.setProperty should respect the property and allow you to set it to the RI implementation. This is a commonly enough asked for functionality, and in the past when asked to support this, we've typically pointed to the fact that the property is implementation specific, and if you need it, just don't use our feature. But a much greyer area in terms of how its invoked through a specification API. I think it would make a great candidate as a new Feature request? It would mean opening a new git issue and using the Feature request template and then fill out with the template with the required info. Would you open one if you'd like us to support this with our features?

2. The ClassNotFoundException: javax.xml.bind.JAXBContextFactory when the jaxb-2.2 feature is NOT installed.

What's odd is that look up of the class is being done by our internal tools bundle. It makes me wonder if some feature is trying to load JAXB and because it's not installed its failing. Can you post what features you have enabled in your server.xml?

[soilworker01]

Thanks for your response! Here are the features from our server.xml:

<feature>jsp-2.3</feature>
<feature>jms-2.0</feature>
<feature>jdbc-4.2</feature>
<feature>mpMetrics-2.3</feature>
<feature>cdi-2.0</feature>
<feature>el-3.0</feature>
<feature>distributedMap-1.0</feature>
<feature>jndi-1.0</feature>
<feature>servlet-4.0</feature>
<!-- deactivated -->
<!--<feature>jaxb-2.2</feature> -->
<feature>concurrent-1.0</feature>

[soilworker01]

As 24.0.0.1 is working on our side we have replaced the com.ibm.ws.jaxb.tools_1.0.86 jar in 24.0.0.2 with com.ibm.ws.jaxb.tools.2.2.10_1.0.85(24.0.0.1) -> No exception, works again.

Cross-check: Replacing the com.ibm.ws.jaxb.tools.2.2.10_1.0.85 jar with com.ibm.ws.jaxb.tools_1.0.86 jar in 24.0.0.1 the error occures again.

Caused by: java.lang.ClassNotFoundException: javax.xml.bind.JAXBContextFactory cannot be found by com.ibm.ws.jaxb.tools_1.0.86.cl240220240212-1928

It looks like something changed in jaxb tools with version 24.0.0.2, but I can't figure out what.

[neuwerk]

This is great info. Thank you for trying this and providing updates.

Still its an odd one for sure. None of the features enabled are using that particular bundle/jar, so I can't explain why it would be getting loaded by the runtime in the first place. You just shouldn't see anything happening with this jar, because none of features load it. What's even odder, is that the cdi-2.0 feature is enabling the JAXB spec bundles, which, AFAIK, should make the javax.xml.bind.JAXBContextFactory class available to any of our other bundles.

Comparing the changes between com.ibm.ws.jaxb.tools.2.2.10_1.0.85 and com.ibm.ws.jaxb.tools_1.0.86.jar I see little that could be causing change. But its interesting that replacing the jar removes the error. If any of the features you're using were enabling the com.ibm.ws.jaxb.tools_1.0.86.jar bundle jar, by replacing the jar, you would see bundle activation errors because the new bundle is no longer there for the feature to activate. That being said, are you doing anything other than replacing the jar files in the lib directory? I assume not, but does your environment/application use this jar directly in any way?

I can try to recreate on my end, would you mind telling me what version of JAXB your application is using since you're not using our feature?

[soilworker01]

"That being said, are you doing anything other than replacing the jar files in the lib directory? I assume not, but does your environment/application use this jar directly in any way?" We just replaced the jaxb tools jar in the lib folder of liberty, we haven't added it to our application or something like that. Additional information here: Just removing the jaxb tool jar from the lib folder also fixes the problem

"I can try to recreate on my end, would you mind telling me what version of JAXB your application is using since you're not using our feature?" Generally we use JAXB 2.3 and we have tried a lot of variants

• jaxb-impl 2.3.3, jakarta.xml.bind api 2.3.3
• jaxb-runtime 2.3.6, jakarta.xml.bind api 2.3.3
• jaxb-api 2.3.1, jaxb-impl 2.3.3, jaxb-core 2.3.0
• jaxb-api 2.3.1, jaxb-impl 2.3.3, jaxb-core 2.3.0, jaxb-runtime 2.3.6

We have also tried JAXB 2.2 (2.2.11)

Thanks in advance!

[neuwerk]

Perfect! Thank you, I'll give it a go and see what I can find out. I'll get back to you to shortly.

[neuwerk]

Quick update, I was able to recreate. Now to find the problem...

[soilworker01]

I don't know if this is helping but with the VM parameter

-Djavax.xml.bind.JAXBContextFactory=com.sun.xml.bind.v2.ContextFactory

there happens some interesting classloading.

First of all apache camel loads the classes from the correct libs

[err] März 28, 2024 1:32:10 NACHM. javax.xml.bind
FINE: Resolved classes from context path: [class org.apache.camel.BeanScope, class org.apache.camel.model.AggregateDefinition, class org.apache.camel.model.cloud.BlacklistServiceCallServiceFilterConfiguration, class org.apache.camel.model.config.BatchResequencerConfig, class org.apache.camel.model.dataformat.ASN1DataFormat, class org.apache.camel.model.errorhandler.DeadLetterChannelDefinition, class org.apache.camel.model.language.CSimpleExpression, class org.apache.camel.model.loadbalancer.CustomLoadBalancerDefinition, class org.apache.camel.model.rest.ApiKeyDefinition, class org.apache.camel.model.transformer.CustomTransformerDefinition, class org.apache.camel.model.validator.CustomValidatorDefinition, class org.apache.camel.core.xml.CamelJMXAgentDefinition, class org.apache.camel.spring.xml.CamelBeanPostProcessor]
[err] März 28, 2024 1:32:10 NACHM. javax.xml.bind
FINE: Checking system property javax.xml.bind.JAXBContextFactory
[err] März 28, 2024 1:32:10 NACHM. javax.xml.bind
FINE:   found com.sun.xml.bind.v2.ContextFactory
[77.169s][info][class,load  ] javax.xml.bind.ServiceLoaderUtil source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/usr/servers/defaultServer/apps/expanded/myapp.war/WEB-INF/lib/jakarta.xml.bind-api-2.3.3.jar
[77.172s][info][class,load  ] com.sun.xml.bind.v2.ContextFactory source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/usr/servers/defaultServer/apps/expanded/myapp.war/WEB-INF/lib/jaxb-impl-2.3.3.jar
[77.174s][info][class,load  ] com.sun.xml.bind.api.JAXBRIContext source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/usr/servers/defaultServer/apps/expanded/myapp.war/WEB-INF/lib/jaxb-impl-2.3.3.jar
[77.178s][info][class,load  ] com.sun.xml.bind.v2.runtime.JAXBContextImpl source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/usr/servers/defaultServer/apps/expanded/myapp.war/WEB-INF/lib/jaxb-impl-2.3.3.jar
[77.180s][info][class,load  ] com.sun.xml.bind.v2.model.annotation.AnnotationReader source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/usr/servers/defaultServer/apps/expanded/myapp.war/WEB-INF/lib/jaxb-impl-2.3.3.jar
[77.180s][info][class,load  ] com.sun.xml.bind.v2.model.annotation.RuntimeAnnotationReader source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/usr/servers/defaultServer/apps/expanded/myapp.war/WEB-INF/lib/jaxb-impl-2.3.3.jar
[77.181s][info][class,load  ] javax.xml.bind.JAXBContextFactory source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/usr/servers/defaultServer/apps/expanded/myapp.war/WEB-INF/lib/jakarta.xml.bind-api-2.3.3.jar
[77.181s][info][class,load  ] jdk.internal.reflect.GeneratedMethodAccessor839 source: __JVM_DefineClass__
[77.182s][info][class,load  ] jdk.internal.reflect.GeneratedConstructorAccessor1657 source: __JVM_DefineClass__

But if we try to marshall something the classes get loaded from com.ibm.ws.jaxb.tools_1.0.86.jar and, that's really interesting, also from com.ibm.websphere.javaee.jaxb.2.2_1.0.86.jar which I think is the lib of the openliberty jaxb feature. The jaxb feature is not activated, but it still loads classes from there?

[err] März 28, 2024 1:35:31 NACHM. javax.xml.bind
FINE: Searching jaxb.properties
[err] März 28, 2024 1:35:31 NACHM. javax.xml.bind
FINE: Checking system property javax.xml.bind.JAXBContextFactory
[err] März 28, 2024 1:35:31 NACHM. javax.xml.bind
FINE:   found com.sun.xml.bind.v2.ContextFactory
[277.938s][info][class,load  ] com.sun.xml.bind.v2.ContextFactory source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/lib/com.ibm.ws.jaxb.tools_1.0.86.jar
[err] März 28, 2024 1:35:31 NACHM. javax.xml.bind
FINE: loaded com.sun.xml.bind.v2.ContextFactory from bundleresource://200.fwk377662424/com/sun/xml/bind/v2/ContextFactory.class
[277.940s][info][class,load  ] javax.xml.bind.JAXBContext source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/dev/api/spec/com.ibm.websphere.javaee.jaxb.2.2_1.0.86.jar
[277.940s][info][class,load  ] com.sun.xml.bind.api.JAXBRIContext source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/lib/com.ibm.ws.jaxb.tools_1.0.86.jar
[277.942s][info][class,load  ] com.sun.xml.bind.v2.runtime.JAXBContextImpl source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/lib/com.ibm.ws.jaxb.tools_1.0.86.jar
[277.943s][info][class,load  ] javax.xml.bind.JAXBException source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/dev/api/spec/com.ibm.websphere.javaee.jaxb.2.2_1.0.86.jar
[277.944s][info][class,load  ] com.sun.xml.bind.v2.model.annotation.AnnotationReader source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/lib/com.ibm.ws.jaxb.tools_1.0.86.jar
[277.944s][info][class,load  ] com.sun.xml.bind.v2.model.annotation.RuntimeAnnotationReader source: file:/C:/Users/myuser/IdeaProjects/my-webapp/build/wlp/lib/com.ibm.ws.jaxb.tools_1.0.86.jar
[277.944s][info][class,load  ] jdk.internal.reflect.GeneratedMethodAccessor1402 source: __JVM_DefineClass__
[277.944s][info][class,load  ] jdk.internal.reflect.GeneratedConstructorAccessor2836 source: __JVM_DefineClass__

[neuwerk]

I really appreciate you sending in what you've found on your own, I was playing with this as well over the last couple of days. I wasn't seeing the java.lang.ClassNotFoundException when the JDK property was set, which is good in a way because it at least shows that the JAXB algorithm for Discovery of JAXB implementation is respected. I don't think it should be required in this case, and marshaling shouldn't load any of our internals.

This bundle com.ibm.websphere.javaee.jaxb.2.2_1.0.86.jar is part of our jaxb-2.2 feature, but to fully explain what's happening here: most of the components that make up our jaxb-2.2 feature are being enabled by different features. The cdi-2.0 feature will pull in com.ibm.websphere.javaee.jaxb.2.2 as a dependency, but JAXB RI is also activated, com.ibm.ws.jaxb.tools_1.0.86.jar, which is being enabled by a long feature dependency chain that eventually leads to jms-2.0. What does this all mean? The features you need, depend on our internal JAXB feature, but it was written in such a way, that the RI should be shielded from your application classpath. That's clearly broken from 24.0.0.1 and on.

What I suspect happened with the JAXB update is that the mechanism for looking up the Context impl has changed in such a way, that a JAXBContext on the application classpath can identify our "internal" ContextImpl as the impl to use, but since its on a different classpath it can't find the spec JAXBContext class to load. This is just speculation from my debugging. I'm not 100% sure what the fix will look like, but I am getting closer. I'll keep you posted as I know more.

[soilworker01]

In the course of the analysis, I noticed that actually a JAXB-2.3 feature was planned for Liberty. However, I couldn't find why it was removed or why it didn't make it into any release. I'm also not sure if a more compatible jaxb version will help us here.

[neuwerk]

Hey @soilworker01 sorry for the delay getting back to you, I've actually got a fix in progress. I just need to expand our tests to make sure I don't regress anything with the fix.

[neuwerk]

I was discussing my fix with my coworker, and they pointed out that even with the change I was thinking, you'll need to update some of your server configuration. To prevent a clash with the internal versions of JAXB that we use with other features, we'd normally suggest changing Liberty's classloader delegation to parentLast. Can you try doing this in your server.xml:

<classloader delegation="parentLast"/>

This can be added directly to the server.xml or configured on the application or module level using the application configuration. This is the doc that describes the classloader config. This should prevent the JAXB look up algorithm from finding our internal implementation. The fix I was thinking would remove one of these steps in the lookup algorithm, but its by no means conclusive and the only way around that is by setting the classloader delegation. Can you try it and let me know if you still see the issue? I tried myself and it resolved it for me without my fix.

[soilworker01]

Thanks for your response! We've already tried parentLast for our application. Unfortunately this is also not working:

[FEHLER ] CWWKZ0002E: Beim Starten der Anwendung myapp ist eine Ausnahme eingetreten. Ausnahmenachricht: com.ibm.ws.container.service.state.StateChangeException: java.lang.RuntimeException: Service class org.apache.camel.cdi.CdiCamelExtension didn't implement the required interface

I also think it's not best practice to use parentLast in general as you want to use the features provided by the application server/liberty. You don't want to package and maintain all dependencies in your application yourself.

[neuwerk]

I'm not surprised you've tried it already! Still, that's good to know. It'd worked for me because I don't have nearly as complex of a dependency chain as your applications probably do. The parentLast is recommended in certain instances, but you're right you can run into problems when the application is depending on Liberty features as well. I'm sure you've already covered this edge case, but I just want to make sure: the application/module that requires JAXB is the one generating the StateChangeException? Classloader delegation can be modularized, so I just want to make sure when you added the classloader config you added as scoped as possible to the application that's packaging JAXB?

[soilworker01]

There is only one application deployed on our liberty and that's the configuration:

    <application type="war" id="myapp" location="myapp.war"
                 name="myapp">
        <classloader delegation="parentLast" apiTypeVisibility="spec, ibm-api, api, stable"
                     commonLibraryRef="OracleLib,ibmMq"/>
    </application>

[neuwerk]

I'm by no means a classloading expert, but after discussing this further with actual experts, it seems likely that some of your dependencies are overlapping with what Liberty is providing through other features. If your JAXB impl was the only dependency that overlaps with Liberty provided APIs, then setting parentLast shouldn't cause any issues.

Obviously you're application is using Apache Camel, is the interface implemented by CdiCamelExtension being included in your dependencies? Whatever interface it is implementing, it seems likely that is conflicting with one being provided by Liberty itself.

[neuwerk]

If you could post a list of all the dependencies you're pulling in with your app, we should be able to figure out what in there would conflict with anything Liberty is providing.

[soilworker01]

I'm not sure if you really want to see this list :)

abdera-core-1.1.3.jar
abdera-i18n-1.1.3.jar
abdera-parser-1.1.3.jar
apache-mime4j-core-0.8.7.jar
apicurio-data-models-1.1.26.jar
asm-9.4.jar
axiom-api-1.2.14.jar
axiom-impl-1.2.14.jar
bcmail-jdk18on-1.72.jar
bcpg-jdk18on-1.72.jar
bcpkix-jdk18on-1.72.jar
bcprov-jdk18on-1.72.jar
bcutil-jdk18on-1.72.jar
beanio-2.1.0.jar
c3p0-0.9.5.5.jar
camel-api-3.20.2.jar
camel-atom-3.20.2.jar
camel-attachments-3.20.2.jar
camel-base-3.20.2.jar
camel-base-engine-3.20.2.jar
camel-bean-3.20.2.jar
camel-browse-3.20.2.jar
camel-cdi-3.20.2.jar
camel-cloud-3.20.2.jar
camel-cluster-3.20.2.jar
camel-controlbus-3.20.2.jar
camel-core-3.20.2.jar
camel-core-catalog-3.20.2.jar
camel-core-engine-3.20.2.jar
camel-core-languages-3.20.2.jar
camel-core-model-3.20.2.jar
camel-core-processor-3.20.2.jar
camel-core-reifier-3.20.2.jar
camel-core-xml-3.20.2.jar
camel-crypto-3.20.2.jar
camel-cxf-common-3.20.2.jar
camel-cxf-soap-3.20.2.jar
camel-cxf-spring-common-3.20.2.jar
camel-cxf-spring-soap-3.20.2.jar
camel-cxf-spring-transport-3.20.2.jar
camel-cxf-transport-3.20.2.jar
camel-dataformat-3.20.2.jar
camel-dataset-3.20.2.jar
camel-direct-3.20.2.jar
camel-directvm-3.20.2.jar
camel-dsl-support-3.20.2.jar
camel-exec-3.20.2.jar
camel-file-3.20.2.jar
camel-ftp-3.20.2.jar
camel-health-3.20.2.jar
camel-http-3.20.2.jar
camel-http-base-3.20.2.jar
camel-http-common-3.20.2.jar
camel-jackson-3.20.2.jar
camel-jaxb-3.20.2.jar
camel-jdbc-3.20.2.jar
camel-jetty-3.20.2.jar
camel-jetty-common-3.20.2.jar
camel-jms-3.20.2.jar
camel-jpa-3.20.2.jar
camel-language-3.20.2.jar
camel-ldap-3.20.2.jar
camel-log-3.20.2.jar
camel-mail-3.20.2.jar
camel-main-3.20.2.jar
camel-management-3.20.2.jar
camel-management-api-3.20.2.jar
camel-mock-3.20.2.jar
camel-quartz-3.20.2.jar
camel-ref-3.20.2.jar
camel-rest-3.20.2.jar
camel-rest-openapi-3.20.2.jar
camel-rss-3.20.2.jar
camel-saga-3.20.2.jar
camel-scheduler-3.20.2.jar
camel-seda-3.20.2.jar
camel-servlet-3.20.2.jar
camel-soap-3.20.2.jar
camel-spring-3.20.2.jar
camel-spring-boot-3.20.2.jar
camel-spring-main-3.20.2.jar
camel-spring-xml-3.20.2.jar
camel-sql-3.20.2.jar
camel-stub-3.20.2.jar
camel-support-3.20.2.jar
camel-swagger-java-3.20.2.jar
camel-timer-3.20.2.jar
camel-tooling-model-3.20.2.jar
camel-util-3.20.2.jar
camel-util-json-3.20.2.jar
camel-validator-3.20.2.jar
camel-vm-3.20.2.jar
camel-xml-io-util-3.20.2.jar
camel-xml-jaxb-3.20.2.jar
camel-xml-jaxb-dsl-3.20.2.jar
camel-xml-jaxp-3.20.2.jar
camel-xmlsecurity-3.20.2.jar
camel-xpath-3.20.2.jar
camel-xslt-3.20.2.jar
camel-xstream-3.20.2.jar
camel-zipfile-3.20.2.jar
cdi-api-2.0.jar
checker-qual-3.12.0.jar
classgraph-4.8.138.jar
classmate-1.5.1.jar
commons-beanutils-1.9.4.jar
commons-codec-1.15.jar
commons-collections-3.2.2.jar
commons-collections4-4.4.jar
commons-compress-1.21.jar
commons-digester-1.8.jar
commons-exec-1.3.jar
commons-io-2.11.0.jar
commons-lang-2.6.jar
commons-lang3-3.12.0.jar
commons-logging-1.2.jar
commons-math3-3.6.1.jar
commons-net-3.9.0.jar
commons-pool-1.6.jar
commons-text-1.10.0.jar
cryptacular-1.2.4.jar
curvesapi-1.07.jar
cxf-core-3.5.5.jar
cxf-rt-bindings-soap-3.5.5.jar
cxf-rt-bindings-xml-3.5.5.jar
cxf-rt-databinding-jaxb-3.5.5.jar
cxf-rt-features-logging-3.5.5.jar
cxf-rt-frontend-jaxrs-3.5.5.jar
cxf-rt-frontend-jaxws-3.5.5.jar
cxf-rt-frontend-simple-3.5.5.jar
cxf-rt-rs-client-3.5.5.jar
cxf-rt-rs-service-description-common-openapi-3.5.5.jar
cxf-rt-rs-service-description-openapi-v3-3.5.5.jar
cxf-rt-rs-service-description-swagger-ui-3.5.5.jar
cxf-rt-security-3.5.5.jar
cxf-rt-security-saml-3.5.5.jar
cxf-rt-transports-http-3.5.5.jar
cxf-rt-ws-addr-3.5.5.jar
cxf-rt-wsdl-3.5.5.jar
cxf-rt-ws-policy-3.5.5.jar
cxf-rt-ws-security-3.5.5.jar
dss-alert-5.11.1.jar
dss-crl-parser-5.11.1.jar
dss-detailed-report-jaxb-5.11.1.jar
dss-diagnostic-jaxb-5.11.1.jar
dss-document-5.11.1.jar
dss-enumerations-5.11.1.jar
dss-i18n-5.11.1.jar
dss-jaxb-common-5.11.1.jar
dss-jaxb-parsers-5.11.1.jar
dss-model-5.11.1.jar
dss-policy-jaxb-5.11.1.jar
dss-simple-certificate-report-jaxb-5.11.1.jar
dss-simple-report-jaxb-5.11.1.jar
dss-spi-5.11.1.jar
dss-utils-5.11.1.jar
dss-utils-apache-commons-5.11.1.jar
dss-xades-5.11.1.jar
ehcache-3.9.6.jar
el-ri-1.0.jar
error_prone_annotations-2.11.0.jar
excel-streaming-reader-4.0.0.jar
failureaccess-1.0.1.jar
FastInfoset-1.2.18.jar
fontbox-1.8.12.jar
geronimo-jpa_2.1_spec-1.0-alpha-1.1.jar
geronimo-jta_1.1_spec-1.1.1.jar
geronimo-ws-metadata_2.0_spec-1.1.3.jar
gmbal-4.0.1.jar
gson-2.10.1.jar
guava-31.1-android.jar
h2-2.1.212.jar
ha-api-3.1.12.hk2-jar
HdrHistogram-2.1.12.jar
hibernate-validator-7.0.1.Final.jar
HikariCP-java7-2.4.13.jar
httpclient-4.5.13.jar
httpcore-4.4.13.jar
httpmime-4.5.9.jar
istack-commons-runtime-3.0.12.jar
j2objc-annotations-1.3.jar
jackson-annotations-2.14.2.jar
jackson-core-2.14.2.jar
jackson-databind-2.14.2.jar
jackson-dataformat-xml-2.14.2.jar
jackson-dataformat-yaml-2.14.2.jar
jackson-datatype-jdk8-2.14.2.jar
jackson-datatype-joda-2.14.2.jar
jackson-datatype-jsr310-2.14.2.jar
jackson-jaxrs-base-2.14.2.jar
jackson-jaxrs-json-provider-2.14.2.jar
jackson-module-jaxb-annotations-2.14.2.jar
jakarta.activation-1.2.2.jar
jakarta.activation-api-1.2.2.jar
jakarta.annotation-api-1.3.5.jar
jakarta.jws-api-2.1.0.jar
jakarta.mail-1.6.7.jar
jakarta.validation-api-3.0.0.jar
jakarta.ws.rs-api-2.1.6.jar
jakarta.xml.bind-api-2.3.3.jar
jakarta.xml.soap-api-1.4.2.jar
jakarta.xml.ws-api-2.3.3.jar
jasypt-1.9.3.jar
javaparser-1.0.8.jar
javassist-3.28.0-GA.jar
java-support-7.5.2.jar
javax.activation-1.2.0.jar
javax.activation-api-1.2.0.jar
javax.annotation-api-1.3.2.jar
javax.el-api-3.0.0.jar
javax.inject-1.jar
javax.interceptor-api-1.2.2.jar
javax.jws-3.1.2.2.jar
javax.mail-1.6.0.jar
javax.transaction-api-1.3.jar
javax.xml.soap-api-1.4.0.jar
jaxb2-basics-0.6.4.jar
jaxb2-basics-runtime-0.6.4.jar
jaxb2-basics-tools-0.6.4.jar
jaxb-api-2.3.1.jar
jaxb-core-2.3.0.jar
jaxb-impl-2.3.3.jar
jaxb-runtime-2.3.6.jar
jaxws-api-2.3.1.jar
jaxws-rt-2.3.3.jar
jboss-logging-3.4.1.Final.jar
jdom2-2.0.6.1.jar
jempbox-1.8.12.jar
jettison-1.5.2.jar
jetty-client-9.4.50.v20221201.jar
jetty-continuation-9.4.50.v20221201.jar
jetty-http-9.4.50.v20221201.jar
jetty-io-9.4.50.v20221201.jar
jetty-jmx-9.4.50.v20221201.jar
jetty-security-9.4.50.v20221201.jar
jetty-server-9.4.50.v20221201.jar
jetty-servlet-9.4.50.v20221201.jar
jetty-servlets-9.4.50.v20221201.jar
jetty-util-9.4.50.v20221201.jar
jetty-util-ajax-9.4.50.v20221201.jar
joda-time-2.10.10.jar
jsch-0.2.1.jar
jsr305-3.0.2.jar
jstl-1.1.0.jar
jul-to-slf4j-1.7.36.jar
LatencyUtils-2.0.3.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
log4j-api-2.20.0.jar
log4j-core-2.20.0.jar
log4j-jul-2.20.0.jar
log4j-slf4j-impl-2.20.0.jar
management-api-3.2.2.jar
mapstruct-1.4.2.Final.jar
mchange-commons-java-0.2.19.jar
metrics-core-3.1.5.jar
micrometer-core-1.9.17.jar
microprofile-metrics-api-2.3.jar
mimepull-1.9.13.jar
mxparser-1.2.2.jar
myfaces-api-2.1.18.jar
myfaces-impl-2.1.18.jar
neethi-3.2.0.jar
openjpa-jdbc-3.0.0.jar
openjpa-kernel-3.0.0.jar
openjpa-lib-3.0.0.jar
openjpa-persistence-3.0.0.jar
openjpa-persistence-jdbc-3.0.0.jar
opensaml-core-3.4.6.jar
opensaml-messaging-api-3.4.6.jar
opensaml-profile-api-3.4.6.jar
opensaml-saml-api-3.4.6.jar
opensaml-saml-impl-3.4.6.jar
opensaml-security-api-3.4.6.jar
opensaml-security-impl-3.4.6.jar
opensaml-soap-api-3.4.6.jar
opensaml-soap-impl-3.4.6.jar
opensaml-storage-api-3.4.6.jar
opensaml-xacml-api-3.4.6.jar
opensaml-xacml-impl-3.4.6.jar
opensaml-xacml-saml-api-3.4.6.jar
opensaml-xacml-saml-impl-3.4.6.jar
opensaml-xmlsec-api-3.4.6.jar
opensaml-xmlsec-impl-3.4.6.jar
org.osgi.annotation.versioning-1.0.0.jar
pdfbox-1.8.12.jar
pfl-basic-4.1.0.jar
pfl-tf-4.1.0.jar
poi-5.2.2.jar
poi-ooxml-5.2.2.jar
poi-ooxml-lite-5.2.2.jar
poi-shared-strings-2.5.3.jar
policy-2.7.10.jar
quartz-2.3.2.jar
reflections-0.10.2.jar
rome-1.18.0.jar
rome-utils-1.18.0.jar
saaj-impl-1.5.2.jar
serp-1.15.1.jar
slf4j-api-1.7.36.jar
snakeyaml-2.0.jar
SparseBitSet-1.2.jar
specs-trusted-list-5.11.1.jar
specs-validation-report-5.11.1.jar
specs-xades-5.11.1.jar
specs-xmldsig-5.11.1.jar
spring-aop-5.3.31.jar
spring-beans-5.3.31.jar
spring-boot-2.7.18.jar
spring-boot-actuator-2.7.18.jar
spring-boot-actuator-autoconfigure-2.7.18.jar
spring-boot-autoconfigure-2.7.18.jar
spring-boot-starter-2.7.18.jar
spring-boot-starter-actuator-2.7.18.jar
spring-boot-starter-log4j2-2.7.18.jar
spring-boot-starter-security-2.7.18.jar
spring-context-5.3.31.jar
spring-core-5.3.31.jar
spring-expression-5.3.31.jar
spring-jcl-5.3.31.jar
spring-jdbc-5.3.31.jar
spring-jms-5.3.31.jar
spring-messaging-5.3.31.jar
spring-orm-5.3.31.jar
spring-security-config-5.7.11.jar
spring-security-core-5.7.11.jar
spring-security-crypto-5.7.11.jar
spring-security-web-5.7.11.jar
spring-tx-5.3.31.jar
spring-web-5.3.31.jar
spring-webmvc-5.3.31.jar
stax2-api-4.2.1.jar
stax-ex-1.8.3.jar
streambuffer-1.5.9.jar
swagger-annotations-1.6.11.jar
swagger-annotations-2.1.13.jar
swagger-core-1.6.11.jar
swagger-core-2.1.13.jar
swagger-integration-2.1.13.jar
swagger-jaxrs-1.6.4.jar
swagger-jaxrs2-2.1.13.jar
swagger-models-1.6.11.jar
swagger-models-2.1.13.jar
swagger-parser-1.0.62.jar
tika-core-2.6.0.jar
trinidad-api-2.1.4.jar
trinidad-impl-2.1.4.jar
txw2-2.3.6.jar
validation-api-1.1.0.Final.jar
validation-policy-5.11.1.jar
velocity-1.7.jar
woodstox-core-6.5.0.jar
wsdl4j-1.6.3.jar
wss4j-bindings-2.4.1.jar
wss4j-policy-2.4.1.jar
wss4j-ws-security-common-2.4.1.jar
wss4j-ws-security-dom-2.4.1.jar
wss4j-ws-security-policy-stax-2.4.1.jar
wss4j-ws-security-stax-2.4.1.jar
xbean-asm6-shaded-4.8.jar
xmlbeans-5.0.3.jar
xmlpull-1.1.3.1.jar
xml-resolver-1.2.jar
xmlschema-core-2.3.0.jar
xmlsec-2.3.1.jar
xstream-1.4.20.jar

[neuwerk]

Haha, that is quite the list. There are a few on here that I've spotted that I want to make sure you're not including in your application or shared libraries, as they are already provided by features you have implemented in your server.xml.

el-3.0 - el-ri-1.0.jar, javax.el-api-3.0.0.jar

cdi-2.0 - javax.inject-1.jar, javax.interceptor-api-1.2.2.jar

jms-2.0 - javax.transaction-api-1.3.jar

mpMetrics-2.3 - microprofile-metrics-api-2.3.jar

I'm not sure I've spotted everything, but dropping these at least should help, especially since the failure seems to be a conflict with the CDI spec apis that cdi-2.0 provides.

[soilworker01]

I've already tried to cleanup some dependencies(especially cdi) and at least camel is starting up. I don't know yet if there is everything working properly. The actual problem is that there is no chance to get the UI working(jsf). It doesn't matter if I use the jsf feature from liberty or ship the dependencies with the app.

Things I have tried before your last post: removed dependencies:

cdi-api-2.0.jar
javax.inject-1.jar
saaj-impl-1.5.2.jar

I also considered the dependencies of your post and tried again: removed dependencies:

el-ri-1.0.jar
javax.el-api-3.0.0.jar
cdi-api-2.0.jar
javax.inject-1.jar
javax.interceptor-api-1.2.2.jar
javax.transaction-api-1.3.jar
microprofile-metrics-api-2.3.jar
saaj-impl-1.5.2.jar

Both variants lead(as far as I tried for now) to the same problem:

2024-04-16 09:53:09,871 [Default Executor-thread-15] INFO  - Apache Camel 3.20.2 (camel) started in 2s894ms (build:172ms init:671ms start:2s51ms)
[AUDIT   ] CWWKZ0001I: Application myapp started in 39,360 seconds.
[ERROR   ] SRVE0271E: Uncaught init() exception created by servlet [FacesServlet] in application [myapp]: java.lang.IllegalStateException: No Factories configured for this Application. This happens if the faces-initialization does not work at all - make sure that you properly include all configuration settings necessary for a basic faces application and that all the necessary libs are included. Also check the logging output of your web application and your container for any exceptions!
If you did that and find nothing, the mistake might be due to the fact that you use some special web-containers which do not support registering context-listeners via TLD files and a context listener is not setup in your web.xml.
A typical config looks like this;
<listener>
  <listener-class>org.apache.myfaces.webapp.StartupServletContextListener</listener-class>
</listener>

    at javax.faces.FactoryFinder._getFactory(FactoryFinder.java:310)
    at javax.faces.FactoryFinder.getFactory(FactoryFinder.java:230)
    at javax.faces.webapp.FacesServlet.init(FacesServlet.java:121)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:301)
    at [internal classes]

[ERROR   ] SRVE0276E: Error while initializing Servlet [FacesServlet]: javax.servlet.ServletException: SRVE0207E: Uncaught initialization exception created by servlet
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:370)
    at [internal classes]
Caused by: java.lang.IllegalStateException: No Factories configured for this Application. This happens if the faces-initialization does not work at all - make sure that you properly include all configuration settings necessary for a basic faces application and that all the necessary libs are included. Also check the logging output of your web application and your container for any exceptions!
If you did that and find nothing, the mistake might be due to the fact that you use some special web-containers which do not support registering context-listeners via TLD files and a context listener is not setup in your web.xml.
A typical config looks like this;
<listener>
  <listener-class>org.apache.myfaces.webapp.StartupServletContextListener</listener-class>
</listener>

    at javax.faces.FactoryFinder._getFactory(FactoryFinder.java:310)
    at javax.faces.FactoryFinder.getFactory(FactoryFinder.java:230)
    at javax.faces.webapp.FacesServlet.init(FacesServlet.java:121)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.init(ServletWrapper.java:301)
    ... 1 more

[volosied]

Looks like you're using your own JSF implementations:

myfaces-api-2.1.18.jar
myfaces-impl-2.1.18.jar

And have all the required dependencies ( commons*) along with the correct features (servlet & el)

You should be able to get it to work (if you're only using JSF Managed beans).

Have you tried added the StartupServletContextListener to the web.xml as suggested by the error?

However, if you're trying to get JSF-CDI integration (via cdi-2.0 feature) , then I recommend removing those JSF jars and using the jsf-2.2 feature instead.

[soilworker01]

I've already tried all of these options.

Using the jsf-2.2 feature results in a feature clash(servlet-3.1/servlet-4.0)

Using our own jsf dependencies and the StartupServletContextListener(without jsf-2.2 feature) results in an UI "working half of the time correctly".

• UI opens and it's possible to navigate
• somehow a lot slower
• some jsf components don't work properly
• a lot of jsf warnings in the log

In general when using parentLast there occured a lot of side effects e.g. closing connections doesn't work properly.

I know that our app has a "small" dependency hell problem, but I want to point out that it was working with liberty 24.0.0.1 and is now not working anymore since 24.0.0.2. Also in my opinion using parentLast is not really a good practice.

[volosied]

Does your app work with delegation="parentLast" on 24.0.0.1 if you try?

Could you get a trace for the working (24.0.0.1 ) and failing scenarios (24.0.0.2 w/ parentLast)? I'm not sure where to look, but I'm hoping the traces below might be a good starting point.

Stop your server, clear the logs, add this to your server.xml, and then start it up again.

<logging  traceSpecification="com.ibm.ws.jsf*=all:org.apache.myfaces*=all:com.ibm.ws.webcontainer*=all"
traceFileName="trace.log"
maxFileSize="20"
maxFiles="10"
traceFormat="BASIC" />

I'm not sure if we should add the classloading trace just yet -- may complicate the logs with all the various dependencies you have.

[neuwerk]

@soilworker01 one other thing you could try, if you really do not want to use parentLast, is removing the delegation property, and add the JAXB specific libraries to a new common library location and update the commonLibraryRef like so commonLibraryRef="OracleLib,ibmMq, jaxb23". The common libraries have no class loader delegation, so that might help in this situation. If that doesn't work, and you are still having trouble with JSF, I think @volosied suggestion is a good one.

[soilworker01]

We've already tried to use jaxb via commonLibraryRef, this also leads to "No JaxbContext found". As I've already mentioned, parentLast is not an option for us.

[soilworker01]

regression

[soilworker01]

@neuwerk is the fix for xmlBinding-4.0 tools also helpful for jaxb tools ?

[neuwerk]

Hey @soilworker01, sorry for delay getting back to you, as you've seen I've made a very similar fix for xmlBinding-4.0 and I am in the process of exploring a similar fix for our jaxb-2.2 implementation. I want to avoid introducing any further regressions, but testing the changes locally have so far been promising, and I have pushed those changes to #28642 and ran a wider sweep of tests against a remote build. That also looks promising, but I want to do more testing of our features that depend on jaxb-2.2 to make sure before committing to it.

[neuwerk]

As a further comment, the reason I addressed the xmlBinding-4.0 feature is we stopped exporting our internal specification apis needed by other features to prevent such conflict. That isn't the case for our older features, so parentLast is the recommend way around this. However, the changes to JAXB in its implementation lookup that went into 24.0.0.2 are the same that xmlBinding-4.0 use, and so if the fix works for that, it should work here as well if we are lucky.

[neuwerk]

Hey @soilworker01! Good news, I was able to merge #28642 which got included in our release branch this morning. It will go out in the next release, in the meantime if you want to verify it solves the issue you're seeing, you could try running against one of our nightly builds.

[krismarc]

Hey,

I believe, I've got related story with spring boot based application. Underlying cache implementation is unable to parse it's xml.

FFDC1015I: An FFDC Incident has been created: "org.springframework.context.ApplicationContextException: Unable to start web server; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setWebSecurityCustomizers' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ignoreCloudFoundryPathsWebSecurityCustomizer' defined in class path resource [org/springframework/boot/actuate/autoconfigure/cloudfoundry/servlet/CloudFoundryActuatorAutoConfiguration$IgnoredCloudFoundryPathsWebSecurityConfiguration.class]: Unsatisfied dependency expressed through method 'ignoreCloudFoundryPathsWebSecurityCustomizer' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'cloudFoundryWebEndpointServletHandlerMapping' defined in class path resource [org/springframework/boot/actuate/autoconfigure/cloudfoundry/servlet/CloudFoundryActuatorAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.autoconfigure.cloudfoundry.servlet.CloudFoundryWebEndpointServletHandlerMapping]: Factory method 'cloudFoundryWebEndpointServletHandlerMapping' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cachesEndpoint' defined in class path resource [org/springframework/boot/actuate/autoconfigure/cache/CachesEndpointAutoConfiguration.class]: Unsatisfied dependency expressed through method 'cachesEndpoint' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cacheManager' defined in class path resource [org/springframework/boot/autoconfigure/cache/JCacheCacheConfiguration.class]: Unsatisfied dependency expressed through method 'cacheManager' parameter 1; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jCacheCacheManager' defined in class path resource [org/springframework/boot/autoconfigure/cache/JCacheCacheConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.cache.CacheManager]: Factory method 'jCacheCacheManager' threw exception; nested exception is javax.cache.CacheException: org.ehcache.xml.exceptions.XmlConfigurationException: Error parsing XML configuration at file:/home/vcap/app/wlp/usr/servers/defaultServer/apps/app.war/WEB-INF/classes/ehcache.xml com.ibm.ws.webcontainer.osgi.DynamicVirtualHost startWebApp" at ffdc_24.06.13_13.30.10.2.log

Factory method 'jCacheCacheManager' threw exception; nested exception is javax.cache.CacheException: org.ehcache.xml.exceptions.XmlConfigurationException: Error parsing XML configuration at file:/home/vcap/app/wlp/usr/servers/defaultServer/apps/app.war/WEB-INF/classes/ehcache.xml 

I've compared manifests of jaxb lib.

The same application is fine using 24.0.0.1.

Best regards, K.M.

[neuwerk]

@krismarc is it possible for you try recreating the problem against our most recent nightly build which should have my fix for this JAXB (#28642) issue in it?

[krismarc]

@neuwerk I was planning to give it a try with the latest beta 24.0.0.6. Do you mean this one https://public.dhe.ibm.com/ibmdl/export/pub/software/openliberty/runtime/nightly/2024-06-13_1320/ ? If so, I'll try tomorrow.

Btw. I've opened new issue as this one was already closed https://github.com/OpenLiberty/open-liberty/issues/28745

[neuwerk]

Yep @krismarc, that would be the one! Good idea, just in case its not related, and the issue is still occurring after you run the nightly build.

[krismarc]

I've left an update here: https://github.com/OpenLiberty/open-liberty/issues/28745#issuecomment-2167686894