SSLSocketFactory.getDefault() behaves differently between webProfile-8.0 and webProfile-9.1

[c-koell]

Describe the bug
We are using the default SSLSocketFactory for a MQ Connection.

SocketFactory socketFactory = SSLSocketFactory.getDefault();
mqConnectionFactory.setSSLSocketFactory(socketFactory);

The server.xml looks like

<keyStore id="defaultKeyStore" location="${server.output.dir}/mqClient.p12" password="xxxx" />
<ssl id="defaultSSLConfig" keyStoreRef="defaultKeyStore" trustStoreRef="defaultKeyStore" trustDefaultCerts="true"/>

All works fine until we have tried to upgrade to JakartaEE-9.1. Now i have found that the SSLSocketFactory looks different. Both Features should use the same transportSecurity-1.0 Feature. Here you can see the SSLSocketFactory under webProfile-8.0. The KeyManager is the "defaultKeyStore" which is defined in the server.xml

Here you can see the same SSLSocketFactory with webProfile-9.1. As you can see it is not configured.

Is this a bug or a feature :-) ?

Steps to Reproduce
Same Code works fine under webProfile-8.0. Switching to webProfile-9.1 the KeyManager is not initialized as before.

Diagnostic information:

• OpenLiberty Version: 22.0.0.13
• Java Version: Eclipse OpenJ9 VM, Version 17.0.5+8

[acdemyers]

From your stacks above it looks like webProfile-9.1 is not using a Liberty SSLContext. Liberty uses a custom keyManager and trustManager, that do not seem to be used with webProfile-9.1. I suspect something is making a call to the get the JDK's default SSLContext before transportSecurity-1.0 and the SSL stuff is initialized. That will cause the Liberty custom socket factory not to be used and Liberty SSLContexts do not get picked up in some cases.

[c-koell]

@acdemyers you are right. I haven't seen that the TrustManager is also different. With webProfile-9.1 it is a X509TrustManagerImpl and with webProfile-8.0 it is a WSX509TrustManager

[c-koell]

We are facing now a similar Problem in a other Project. We use there a client certificate in a defined Keystore. After migrating to webProfile-9.1 we get SSL Handshake exception. Should /can we enable some trace to find the root cause ?

[acdemyers]

@c-koell Sorry I thought I had requested trace an a copy of your config. I attempted to recreate this myself but I was not able. I admit by environment we very simple so can you please gather trace and provide me with some configuration information. I'd like Liberty trace set to com.ibm.ws.ssl.=all and com.ibm.websphere.ssl.=all. I'd also like JSSE tracing enabled add -Djavax.net.debug=all in the server jvm.options file.

Please send the trace, console, and message logs. I'd also like a copy of you server.xml file and please let me know if there is anything else in your jvm.options file.

[c-koell]

I have taken 2 traces from a simple application from us. After starting the application i looked at SSLSocketFactory.getDefault() As you can see i have attached the variables from the debugger.

This is from webProfile-8.0 This is from webProfile-9.1

You can find messages-8.0.log and trace-8.0.log trace-8.0.log messages-8.0.log

and also messages-9.1.log and trace-9.1.log messages-9.1.log trace-9.1.log

This is the server.xml from webProfile-9.1. This file is the "same" as with webprofile 8.0. Only the versions are different.

<?xml version="1.0" encoding="UTF-8"?>
<server>
    <featureManager>
        <feature>mpConfig-3.0</feature>
        <feature>webProfile-9.1</feature>
        <feature>mail-2.0</feature>
        <feature>concurrent-2.0</feature>
    </featureManager>   
    <httpEndpoint host="*" httpPort="${server.httpPort}" httpsPort="${server.httpsPort}" id="defaultHttpEndpoint" remoteIpRef="defaultRemoteIp" />
    <include location="${liberty.server.file}"/>
    <include location="${db.datasources.file}" />
    <applicationMonitor updateTrigger="disabled"/>
    <keyStore id="defaultKeyStore" location="${server.output.dir}/mqClient.p12" password="xxx" />
        <ssl id="defaultSSLConfig" keyStoreRef="defaultKeyStore" trustStoreRef="defaultKeyStore" trustDefaultCerts="true"/>
    <logging traceSpecification="com.ibm.ws.ssl.=all:com.ibm.websphere.ssl.=all" traceFileName="trace.log" consoleLogLevel="INFO"/>
</server>

This is the jvm.options file. This file is with webProfile-8.0 and webProfile-9.1 the same

# Generated by liberty-maven-plugin
-Duser.language=de
-Duser.country=AT
-Dfile.encoding=UTF-8
-Xms256m
-Xmx1024m
-Xdump:none
-Ddvt.serverInfo=D/localhost/TST
-Dlog.environment=D
-Dlog.subservice=TST
-Doracle.jdbc.DateZeroTime=true
-Ddvt.profile.activate=on
-Ddvt.jpaExecuteTrace=true
-Ddvt.mandantenFileUrl=Mandanten.xml
-Djacoco
-Ddvt.crypto-key-file-notPresent
-Xjit:disableLockReservation
-Ddummy

greets claus

[acdemyers]

Well looking at the traces I see the difference. webProfile-9.1 is using transportSecurity-1.0 and webProfile-8.0 is not.

When transportSecurity-1.0 is enabled we setup a custom SSLSocketFactory with the java Security property ssl.SocketFactory.provider. The socket factory really services as a proxy that when used should direct you to use the correct Liberty SSLContext/socket factory needed for your connection.

If using ssl-1.0 like webProfile-8.0 is using Liberty's SSLContext is set as the process default SSLContext with SSLContext.setDefault(). In that case your call to get the default SSLSocketFactory will get the default SSLContext then get the socket factory it creates.

The design of the JDK does not allow for both, meaning I can't do a SSLContext.setDefault() and a setting java Security property ssl.SocketFactory.provider.

There is some information documenting the difference here, https://www.ibm.com/docs/en/was-liberty/base?topic=liberty-ssl-defaults-in

What is issue is this difference causing you?

[c-koell]

Thank you for the quick response !

So i have 2 use cases at the moment.

1. We call a HTTP Endpoint with Apache HttpClient and a client certificate must be present. With ssl-1.0 (webProfile-8.0) following configuration was sufficient to establish that connection

   <keyStore location="${server.output.dir}/ServerKeyFile.p12" password="xxxxxx"></keyStore>
   <ssl id="defaultSSLConfig" keyStoreRef="defaultKeyStore" trustStoreRef="defaultTrustStore" trustDefaultCerts="true"/>

   If i understand the documentation i have to define a sslDefault ?

   <keyStore location="${server.output.dir}/ServerKeyFile.p12" password="xxxxx"></keyStore>
   <ssl id="defaultSSLConfig" keyStoreRef="defaultKeyStore" trustStoreRef="defaultTrustStore" trustDefaultCerts="true"/>
   <sslDefault outboundSSLRef="defaultSSLConfig"/> 

   I have tried that but i still get a Handshake Exception ...

2. I need a SSLSocketFactory that i can use in a third party library Unfortunately it is not clear to me how do i get the configured one from Openliberty ? The documentaion (https://www.ibm.com/docs/en/was-liberty/base?topic=liberty-ssl-defaults-in) says -> The (transportSecurity-1.0) behavior -> The SSL runtime code does not set the SSLContext process default when you enable the transportSecurity-1.0 feature but also -> The program returns the Liberty custom SSL socket factory to any code that attempts to obtain the SSLSocketFactory class default with the javax.net.ssl.SSLSocketFactory.getDefault() property. Is that not contradictory?

[c-koell]

In the parent documentaion https://www.ibm.com/docs/en/was-liberty/base?topic=liberty-enabling-ssl-communication-in is following under point 1 -> Enable the transportSecurity-1.0 Liberty feature in the server.xml file

A call to SSLContext.getDefault() returns the default context SSLContext of the JSSE. 
A call to SSLSocketFactory.getDefault() returns an SSLSocketFactory that is based on the Liberty server custom socket factory provider that uses the Liberty SSLContext.

I would understand it that way. If i call SSLContext.getDefault() i will get the SSLContext based on the JSSE (cacerts file) but if i call SSLSocketFactory.getDefault() should return the Liberty Custom SocketFactory

...returns an SSLSocketFactory that is based on the Liberty server custom socket factory provider that uses the Liberty SSLContext

[acdemyers]

Yeah, the behavior is kind of strange but if with transportSecurity-1.0 we set a the java Security property ssl.SocketFactory.provider to point to Liberty's custom SSL socket factory. When set, if you call SSLSocketFactory.getDefault() it goes directly to what is set on the ssl.SocketFactory.provider Security property. When the property is not set, it will make a call to SSLContest.getDefault().getSocketFactory(). With ssl-1.0 we set SSLContext.setDefault() with a liberty SSLContext. Some java restrictions do not allow us to do both, either we set the ssl.SocketFactory.provider or set can set SSLContext.setDefault().

The trace you provided does not show me up to your handshake error. From want I see in the webProfile-9.1 trace all I see is a Liberty SSLContext getting setup. But there is something odd I see in the trace, I'm not sure what it is but there seems to be some SSL activity taking place before Liberty starts logging startup info like:

[03.02.23, 09:27:30:775 MEZ] 00000001 id=00000000 com.ibm.ws.kernel.launch.internal.FrameworkManager I CWWKE0002I: Der Kernel wurde nach 6,618 Sekunden gestartet. [03.02.23, 09:27:30:959 MEZ] 00000034 id=00000000 com.ibm.ws.kernel.feature.internal.FeatureManager I CWWKF0007I: Die Featureaktualisierung wurde gestartet. [03.02.23, 09:27:32:273 MEZ] 00000023 id=00000000 com.ibm.ws.security.ready.internal.SecurityReadyServiceImpl I CWWKS0007I: Der Sicherheitsservice wird gestartet... [03.02.23, 09:27:32:359 MEZ] 00000025 id=00000000 com.ibm.ws.app.manager.internal.monitor.DropinMonitor A CWWKZ0058I: dropins auf Anwendungen überwachen.

So perhaps a JSSE SSLContext was set before Liberty starts logging? If that is the case it can mess with our socket factory setting. Are you running some kind of agent? They have been know to mess ups WebSphere socket factories. Trying to find what is calling a JSSE SSLContext that early will be hard to figure out.

[c-koell]

Thank you for the clarification. So this is also what i would understand after reading the documentation but as written initially SSLSocketFactory.getDefault() does not return the Liberty configured SSLSocketFactory. The documenation is different than the reallity :-(

Please be not confused about the trace. The trace i have taken is from a very simple application from us without any Handshake Exception. But with this application i'm unable to get a valid SSLSocketFactory.

As written i'm using SSLSocketFactory.getDefault() but as you can see from the images the Trust/KeyManager are different with webProfile-8/9.1!

If i look at the code from SSLSocketfactory i found nothing about Security property ssl.SocketFactory.provider

    public static SocketFactory getDefault() {
        if (DefaultFactoryHolder.defaultFactory != null) {
            return DefaultFactoryHolder.defaultFactory;
        }

        try {
            return SSLContext.getDefault().getSocketFactory();
        } catch (NoSuchAlgorithmException | UnsupportedOperationException e) {
            return new DefaultSSLSocketFactory(e);
        }
    }

I do nothing special during startup (no agents). The application will be started with eclipse and liberty-maven-plugin but i see the same behaviour if i start the application without eclipse/liberty-maven-plugin.

@acdemyers do you have a simple application where SSLSocketFactory.getDefault() returns the custom SSLSocketFactory from Liberty ? Dou you also use Java 17 OpenJDK/openj9 (no ibm jdk) ?

[c-koell]

Hi @acdemyers !

So after a little bit of searching i found an interesting fact. My server.xml looks like this:

<server>
    <featureManager>
        <feature>mpConfig-3.0</feature>
        <feature>webProfile-9.1</feature>
        <feature>mail-2.0</feature>
        <feature>concurrent-2.0</feature>
    </featureManager>   
    <httpEndpoint host="*" httpPort="${server.httpPort}" httpsPort="${server.httpsPort}" id="defaultHttpEndpoint" remoteIpRef="defaultRemoteIp" />
    <include location="${liberty.server.file}"/>
    <include location="${db.datasources.file}" />
        <applicationMonitor updateTrigger="disabled"/>
    <keyStore id="defaultKeyStore" location="${server.output.dir}/mqClient.p12" password="xxxx" />
        <ssl id="defaultSSLConfig" keyStoreRef="defaultKeyStore" trustStoreRef="defaultKeyStore" trustDefaultCerts="true"/>
</server>

As you can see we include some external config files. The URL's of these are stage dependent. Here is an example of the bootstrap.properties file

db.datasources.file="${server.config.dir}/datasources.xml"
liberty.server.file="https://config.srv.internal/rest/Configuration/TLR/Files/Liberty/TST/D/server.xml"

So the server makes really early a https call to our config server to get the include file. If i remove that include to the https endpoint then i get a valid SSLSocketFactory in my application !

So i can say now that since webProfile-9.1 this early request makes problems.

What can we / you do here that we get a valid SSLSocketFactory again ?

[acdemyers]

Interesting. Now I know why my simple app did not see the same as your findings. I'm looking into what can be done here.

[c-koell]

Any news ? As a workaround for me i could use use the same Code used in SSLSocketFactory.DefaultFactoryHolder. Thats not really nice but it works.

[acdemyers]

I had to go look up what DefaultFactoryHolder does, and I can see why that work. And I understand why you really don't want to do that as a work around. I've still looking into the problem, just been busy with some other stuff. I'll try comment again later in the day.

[c-koell]

@acdemyers do you have found some time :-) ?

[c-koell]

Unfortunately we found another problem.

We have some servers where we have defined some configDropins like following

<?xml version="1.0" encoding="UTF-8"?>
<server>
    <keyStore id="serverKeyStore" location="${server.output.dir}/resources/security/server-key.p12" password="xxxxx"/>
    <ssl id="serverSSLConfig" keyStoreRef="serverKeyStore" trustStoreRef="serverKeyStore" trustDefaultCerts="true"/>
    <sslOptions id="serverSSLOptions" sslRef="serverSSLConfig"/>
</server>

The KeyStore holds a Server Certificate. The defined sslOptions can than be used like this.

<httpEndpoint host="*" httpPort="${server.httpPort}" sslOptionsRef="serverSSLOptions" httpsPort="${server.httpsPort}" id="defaultHttpEndpoint" remoteIpRef="defaultRemoteIp" />

This works fine until we switch with the application to webProfile-9.1 :-( Although the serverSSLOptions is not used at all wer get a exception in the application while it tries to open a HTTPS Connection with a RestClient.

We see following Stacktrace

Caused by: com.ibm.websphere.ssl.SSLException: SSLContext could not be created due to null SSL properties.
at com.ibm.websphere.ssl.JSSEHelper.getSSLContext(JSSEHelper.java:732) ~[com.ibm.ws.ssl_1.5.72.jar:?]
at [internal classes]
at io.openliberty.restfulWS.internal.ssl.component.SslClientBuilderListener.building(SslClientBuilderListener.java:65) ~[io.openliberty.restfulWS.internal.ssl_1.0.72.jar:?]
at io.openliberty.org.jboss.resteasy.common.client.LibertyResteasyClientBuilderImpl.lambda$build$1(LibertyResteasyClientBuilderImpl.java:55) ~[io.openliberty.org.jboss.resteasy.common.jakarta_1.0.72.jar:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:522) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:512) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:239) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
at io.openliberty.org.jboss.resteasy.common.client.OsgiFacade.invoke(OsgiFacade.java:86) ~[io.openliberty.org.jboss.resteasy.common.jakarta_1.0.72.jar:?]
at io.openliberty.org.jboss.resteasy.common.client.LibertyResteasyClientBuilderImpl.lambda$build$2(LibertyResteasyClientBuilderImpl.java:55) ~[io.openliberty.org.jboss.resteasy.common.jakarta_1.0.72.jar:?]
at java.util.Optional.map(Optional.java:260) ~[?:?]
at io.openliberty.org.jboss.resteasy.common.client.LibertyResteasyClientBuilderImpl.build(LibertyResteasyClientBuilderImpl.java:54) ~[io.openliberty.org.jboss.resteasy.common.jakarta_1.0.72.jar:?]
at [internal classes]...

The problem exists independently if we make the HTTPS call during server start or not.

It's really unhandy that a defined config that is not used makes problems after upgrading to a newer feature.

[c-koell]

After digging into the last Problem i have found some interesting facts. If i do not define

<keyStore id="serverKeyStore" location="${server.output.dir}/resources/security/server-key.p12" password="xxxxx"/>
<ssl id="serverSSLConfig" keyStoreRef="serverKeyStore" trustStoreRef="serverKeyStore" trustDefaultCerts="true"/>
<sslOptions id="serverSSLOptions" sslRef="serverSSLConfig"/>

the jsseHelper is null here and no exception will be thrown. https://github.com/OpenLiberty/open-liberty/blob/e09e937bd8e87ee850c82fc5b6f8a50a736edb28/dev/io.openliberty.restfulWS.internal.ssl/src/io/openliberty/restfulWS/internal/ssl/component/SslClientBuilderListener.java#L75-L76 If i define it the jssehelper is != null and the exception will be thrown here https://github.com/OpenLiberty/open-liberty/blob/e09e937bd8e87ee850c82fc5b6f8a50a736edb28/dev/com.ibm.ws.ssl/src/com/ibm/websphere/ssl/JSSEHelper.java#L714-L734

Normally (when we do not define a keyStore) we start the openliberty server without the Server-Env Entry keystore_password=xxxx This means for my understanding that the default KeyStore will not be created as described here https://openliberty.io/docs/latest/reference/feature/transportSecurity-1.0.html After setting the Server-Env Entry we do not get the descibed SSLException. This workaround must be known but it's okay for me. So we can focus here to the main problem ...

[c-koell]

@acdemyers any news here ? To workaround to get the valid SSLSocketFactory is temporary okay for me but is it possible to fix this if a HTTPS Call is made in that erly phase of server startup ?

[acdemyers]

@c-koell I finally got around to working this. I have tried a few things to try to get past the problem but I have been unsuccessfulIy. I need seek a little help here and I will keep trying.

[c-koell]

@acdemyers do you have found some help :-) ?

[c-koell]

@acdemyers i have created a other issue https://github.com/OpenLiberty/open-liberty/issues/25573 Maybe you can have a look at it .. thanks