Truststore loading failure with IBM J9 JDK

[cmgrote]

It appears that if attempting to use the IBM J9 JDK (here specifically version Java(TM) SE Runtime Environment (build 8.0.6.25 - pxa6480sr6fp25-20210115_01(SR6 FP25))) the truststore cannot be loaded. This appears to be an issue with the IBM JDK, as we have not been able to reproduce it with any other.

If there are any users of this JDK please let us know any suggestions on resolving it (obviously without breaking any of the others) -- otherwise capturing here primarily for reference. If you run into this error and do not need to use the IBM JDK, we would suggest using another (OpenJDK, etc).

Project Egeria - Open Metadata and Governance
    ____   __  ___ ___    ______   _____                                 ____   _         _     ___
   / __ \ /  |/  //   |  / ____/  / ___/ ___   ____ _   __ ___   ____   / _  \ / / __    / /  / _ /__   ____ _  _
  / / / // /|_/ // /| | / / __    \__ \ / _ \ / __/| | / // _ \ / __/  / /_/ // //   |  / _\ / /_ /  | /  _// || |
 / /_/ // /  / // ___ |/ /_/ /   ___/ //  __// /   | |/ //  __// /    /  __ // // /  \ / /_ /  _// / // /  / / / /
 \____//_/  /_//_/  |_|\____/   /____/ \___//_/    |___/ \___//_/    /_/    /_/ \__/\//___//_/   \__//_/  /_/ /_/
 :: Powered by Spring Boot (v2.3.3.RELEASE) ::
2021-03-10 03:28:01.916  INFO 20481 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 9443 (https)
2021-03-10 03:28:11.757 ERROR 20481 --- [           main] o.s.boot.SpringApplication               : Application run failed
org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.1.jar!/:5.3.1]
    at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.1.jar!/:5.3.1]
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.1.jar!/:5.3.1]
    at org.springframework.context.support.DefaultLifecycleProcessor$$Lambda$770/0x0000000075f5dd40.accept(Unknown Source) ~[na:na]
    at java.lang.Iterable.forEach(Iterable.java:86) ~[na:2.9 (12-18-2020)]
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.1.jar!/:5.3.1]
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.1.jar!/:5.3.1]
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:942) ~[spring-context-5.3.1.jar!/:5.3.1]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591) ~[spring-context-5.3.1.jar!/:5.3.1]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) [spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) [spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) [spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) [spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.odpi.openmetadata.serverchassis.springboot.OMAGServerPlatform.main(OMAGServerPlatform.java:93) [classes!/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90) ~[na:1.8.0]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[na:1.8.0]
    at java.lang.reflect.Method.invoke(Method.java:508) ~[na:1.8.0]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [egeria-server-chassis-spring.jar:na]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:109) [egeria-server-chassis-spring.jar:na]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [egeria-server-chassis-spring.jar:na]
    at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:466) [egeria-server-chassis-spring.jar:na]
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:229) ~[spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:43) ~[spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.1.jar!/:5.3.1]
    ... 24 common frames omitted
Caused by: java.lang.IllegalArgumentException: standardService.connector.startFailed
    at org.apache.catalina.core.StandardService.addConnector(StandardService.java:231) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:282) ~[spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:213) ~[spring-boot-2.3.3.RELEASE.jar!/:2.3.3.RELEASE]
    ... 26 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1067) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.catalina.core.StandardService.addConnector(StandardService.java:227) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    ... 28 common frames omitted
Caused by: java.lang.IllegalArgumentException: Invalid keystore format
    at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1141) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1227) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:603) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1064) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    ... 30 common frames omitted
Caused by: java.io.IOException: Invalid keystore format
    at com.ibm.crypto.provider.JavaKeyStore.engineLoad(Unknown Source) ~[ibmjceprovider.jar:8.0 build_20210113-389]
    at java.security.KeyStore.load(KeyStore.java:1456) ~[na:1.8.0]
    at org.apache.tomcat.util.security.KeyStoreUtil.load(KeyStoreUtil.java:69) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:216) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.SSLHostConfig.getTruststore(SSLHostConfig.java:722) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.SSLUtilBase.getTrustManagers(SSLUtilBase.java:423) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97) ~[tomcat-embed-core-9.0.40.jar!/:9.0.40]
    ... 36 common frames omitted

[planetf1]

This looks bad:

Caused by: java.io.IOException: Invalid keystore format

How are you invoking the chassis?

I've been using zuul locally, ubuntu openjdk, & the Redhat UBI in containers, but not J9 recently - I have experienced rather a few issues in the past. In each case I've raised an issue with the openJ9 team who have been able to investigate, but because of this I've not been running with J9 locally. Though not aware of other outstanding issues.

Does it fail entirely with our format of keystore? Java keystores can be in jks format or p12 - we're using the latter as more standard/cross platform - is it specific to that?

[planetf1]

Is this just an IBM JDK issue, or is it openJ9 generally. Can you try with jdks from adoptopenjdk.net - what happens with 8? 11? 15?

[planetf1]

I see an interesting entry in the stack trace

    at com.ibm.crypto.provider.JavaKeyStore.engineLoad(Unknown Source) ~[ibmjceprovider.jar:8.0 build_20210113-389]

not sure if that appears in openj9? I think it's an IBM extension.

I see a recent change (not this issue - but in that area) as per https://www.ibm.com/support/pages/apar/IJ04911

This looks like an IBM JVM issue - if you can reproduce a 'hello world' example with our .p12 I think it worth forwarding to their support?

Also am not familiar with the plugin providers - ie in terms of whether we've done anything wrong

Finally certs are replaceable, and in a real environment the self-signed ones shipped with egeria would not be used in any case

[planetf1]

This is a Java 7 article -> https://www.ibm.com/support/knowledgecenter/en/SSYKE2_7.0.0/com.ibm.java.security.component.70.doc/security-component/keytoolDocs/supportedkeystoretypes.html but this, and other articles indicate there may be some incompatibility even in pkcs12 keystores

There is indeed one use of 'keytool' (openjdk) in the process to generate the certs. Specifically the stage to create the truststore. I had been trying to use openssl tool for all steps but couldn't get that step as I wanted so used keytool instead (which most docs refer to in any case). I wasn't aware of compatibility issues as it's java.... That could be where an issue crept in.

One could run the 'gensamplecerts.sh' using the IBM jdk locally in the environment where you are running egeria, and see if those certs work ok. Also be interesting to see the response from the JDK team

If so, I wouldn't suggest we check these in, but I can look again at removing the keytool invocation and just using openssl

[cmgrote]

Only logged for posterity -- I'm not using this JDK, and the original reporter was happy to switch to another. So suggest we keep the issue to poll if anyone actually uses / plans to use this JDK, but otherwise do not believe we need to actively work on anything to progress it.

[planetf1]

If we don't have a current plan to fix, we could close. I do agree recording it as an issue is really helpful if someone else hits the issue - it can be reopened in future if required