Egeria chassis SSL context error - update docs

[planetf1]

The egeria chassis fails to launch in a default configuration when run from the 'target' directory.

jonesn:server-chassis-spring/ (master) $ LOGGING_LEVEL_ROOT=Info java -jar target/server-chassis-spring-2.4-SNAPSHOT.jar
 ODPi Egeria
    ____   __  ___ ___    ______   _____                                 ____   _         _     ___
   / __ \ /  |/  //   |  / ____/  / ___/ ___   ____ _   __ ___   ____   / _  \ / / __    / /  / _ /__   ____ _  _
  / / / // /|_/ // /| | / / __    \__ \ / _ \ / __/| | / // _ \ / __/  / /_/ // //   |  / _\ / /_ /  | /  _// || |
 / /_/ // /  / // ___ |/ /_/ /   ___/ //  __// /   | |/ //  __// /    /  __ // // /  \ / /_ /  _// / // /  / / / /
 \____//_/  /_//_/  |_|\____/   /____/ \___//_/    |___/ \___//_/    /_/    /_/ \__/\//___//_/   \__//_/  /_/ /_/

 :: Powered by Spring Boot (v2.3.3.RELEASE) ::

15:39:33.833 [main] INFO  o.o.o.s.s.OMAGServerPlatform - Starting OMAGServerPlatform on planetf1mac with PID 73802 (/Users/jonesn/IdeaProjects/egeria-maven/open-metadata-implementation/server-chassis/server-chassis-spring/target/server-chassis-spring-2.4-SNAPSHOT.jar started by jonesn in /Users/jonesn/IdeaProjects/egeria-maven/open-metadata-implementation/server-chassis/server-chassis-spring)
15:39:33.840 [main] INFO  o.o.o.s.s.OMAGServerPlatform - No active profile set, falling back to default profiles: default
15:39:38.333 [main] INFO  o.s.b.w.e.tomcat.TomcatWebServer - Tomcat initialized with port(s): 9443 (https)
15:39:38.367 [main] INFO  o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["https-jsse-nio-9443"]
15:39:38.367 [main] INFO  o.a.catalina.core.StandardService - Starting service [Tomcat]
15:39:38.368 [main] INFO  o.a.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.37]
15:39:38.651 [main] INFO  o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
15:39:41.950 [main] INFO  o.a.catalina.core.StandardService - Stopping service [Tomcat]
15:39:41.980 [main] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'healthIndicatorRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/LegacyHealthEndpointCompatibilityConfiguration.class]: Unsatisfied dependency expressed through method 'healthIndicatorRegistry' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'healthContributorRegistry' defined in class path resource [org/springframework/boot/actuate/autoconfigure/health/HealthEndpointConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.boot.actuate.health.HealthContributorRegistry]: Factory method 'healthContributorRegistry' threw exception; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'elasticsearchHealthContributor' defined in class path resource [org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class]: Unsatisfied dependency expressed through method 'elasticsearchHealthContributor' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestClient' defined in class path resource [org/springframework/boot/autoconfigure/elasticsearch/ElasticsearchRestClientConfigurations$RestClientFallbackConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.RestClient]: Factory method 'elasticsearchRestClient' threw exception; nested exception is java.lang.IllegalStateException: could not create the default ssl context

The full log is at https://gist.github.com/ffff1f08a18853f9d01297a57dce4075

The root issue appears to be Caused by: java.lang.IllegalStateException: could not create the default ssl context - but this is occuring when the java is launched directly ie:

LOGGING_LEVEL_ROOT=Info java -jar target/server-chassis-spring-2.4-SNAPSHOT.jar

which previously worked.

This may be a classpath issue or a change in default configuration

Specifically it's possible spring boot actuator is causing a problem here -- for example we see references to 'health contributor' and specifically org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class

[planetf1]

Removing spring actuator - initialization stillno ferrors with the health configuration (as expected) but still an SSL issue with elastic search

15:51:02.077 [main] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestClient' defined in class path resource [org/springframework/boot/autoconfigure/elasticsearch/ElasticsearchRestClientConfigurations$RestClientFallbackConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.RestClient]: Factory method 'elasticsearchRestClient' threw exception; nested exception is java.lang.IllegalStateException: could not create the default ssl context

later down the stacktrace we see:

Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
    at java.base/java.security.Provider$Service.newInstance(Provider.java:1900)
    at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
    at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
    at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:184)
    at java.base/javax.net.ssl.SSLContext.getDefault(SSLContext.java:110)
    at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:212)
    ... 37 common frames omitted
Caused by: java.security.KeyManagementException: problem accessing trust store

The above was with the zulu 15 jsk. Switching to openj9 11, or zulu 11 we get an extra 'caused' at the end:

Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:792)
    at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243)
    at java.base/java.security.KeyStore.load(KeyStore.java:1479)
    at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:365)
    at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:313)
    at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:55)
    at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49)
    ... 52 common frames omitted
Caused by: java.security.UnrecoverableKeyException: Password verification failed
    at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:790)

Running the chassis where the current working directory has a 'good' truststore works ok ie:

jonesn:resources/ (master) $ pwd                                                                                    [15:59:40]
/Users/jonesn/src/egeria/open-metadata-implementation/server-chassis/server-chassis-spring/src/main/resources
jonesn:resources/ (master) $ java -jar ../../../target/server-chassis-spring-2.4-SNAPSHOT.jar

We would assume explicit specification of the truststore/keystore would be good too.

This tends to suggest either the classpath has changed inside the spring app, or the resources files were included are bad/not being picked up

The assembly version of the app works ok.

This is probably worth cleaning up alongside completing #1403

[planetf1]

2.2 behaves the same in my environment, so at least not a regression for the release. Will check 2.0/1 later. Could be environment specific

[bogdan-sava]

It should work like that: provide trust-store or use strict.ssl=false.

Egeria internal clients are using the same mechanism as strict.ssl=false but without possibility to get rid of it. I suggest to be removed and use sttrict.ssl=false to avoid ssl check.

The problem is that we have a dependency that is trying to use an ssl connection which would be trusted with JVM truststore, but we override it with egeria one.

For starting server chassis with default configuration and without truststore, we can add a validation to check file exists. If not exists, we do not override existing JVM truststore. Platform will start, but, but not able to connect to other Egeria platform. Or maybe yes, as in the SpringRESTClientConnector is hardcoded disable SSL verification.

[bogdan-sava]

Another remark here. Hardcoding disabling SSL check mechanism in the constructor of SpringRESTClientConnector, beyond the fact that cannot be undo in production, it is also redundant, as we have multiple instances of SpringRESTClientConnector and the methods HttpsURLConnection.setDefaultHostnameVerifier and HttpsURLConnection.setDefaultSSLSocketFactory are static. Not needed to be called for all instances of SpringRESTClientConnector.

[planetf1]

Understand there's more work (at least docs) in this area which have been on hold due to other work - thanks for the comments - will assign to myself and look at it along with 1403

[planetf1]

• Agree that the strict-ssl 'setting' that we use client-side needs change. It MUST be possible (even the default when running the server) to have full certificate checking. This has to be done by some specific coding in clients (as we have currently) to modify verification, though that doesn't mean we must have our own config for everything else

I will refer to ongoing updates in #1403 but will use this issue to document current behaviour I will also raise specific issues for changes such as

• strict ssl client-side configuration
• including some certificate-generation code (not run in build, but as example -- this was recently needed for our node work)
• extend docs to ensure we include client-side configuration for node, go, https
• update any docs/tutorials to ensure appropriate client side invocation example is given

Finally to a) help anyone hitting this error now, and b) for your comments here is a launch script I use for the chassis:

#!/bin/sh

 BUILD=/Users/jonesn/src/egeria
 KS=${BUILD}/open-metadata-implementation/server-chassis/server-chassis-spring/src/main/resources/keystore.p12
 TS=${BUILD}/open-metadata-implementation/server-chassis/server-chassis-spring/src/main/resources/keystore.p12
 KSP=egeria
 TSP=egeria
 SKEY=tomcat

 VER=2.4-SNAPSHOT

 JAR=${BUILD}/open-metadata-implementation/server-chassis/server-chassis-spring/build/libs/server-chassis-spring-${VER}.jar

 # Assuming no configuration is done explcitly within Egeria itsef:
 #
 # Settings used by the Egeria 'server' (what we call the Platform) (not clients) - ie when accepting inbound connections
 #
 # server.ssl-key-store              -- Used by tomcat/spring boot to locate keys that identify the server
 # server.ssl-key-alias              -- Used by tomcat/spring boot to identify the alias of the key tomcat should use for itself
 # server.ssl.key-store-password     -- Used by tomcat/spring boot for the keystore password (2 way SSL)
 # server.ssl.trust-store            -- Used by tomcat/spring boot to understand what clients it can trust (2 way SSL)
 # server.ssl.trust-store-password   -- Used by tomcat/spring boot  for the password of the truststore (2 way SSL)
 # strict.ssl                        -- true/false -- sets full checking for all certs (server-side)
 #
 # Settings used by the Egeria 'server' -- and indeed our clients or other java programmes using egeria libraries, acting in the client role (for JAVA)
 #
 # javax.net.ssl.keyStore            -- keyStore for client to use (2 way SSL needs this)
 # javax.net.ssl.keyStorePassword    -- password for the keystore  (2 way SSL needs this)
 # javax.net.ssl.trustStore          -- trustStore for the client to use (always needs setting as egeria makes client calls)
 # javax.net.ssl.trustStorePassword  -- password for the truststore (always - as above)
 # Any specification of which cert a client used would need to be coded within the app, as is strict cert checking (currently disabled)
 #
 # Based on this for a fully specified launch of the chassis we would therefore use:
 java \
     -Dserver.ssl.key-store=${KS} \
     -Dserver.ssl.key-alias=${SKEY} \
     -Dserver.ssl.key-store-password=${KSP} \
     -Dserver.ssl.trust-store=${TS} \
     -Dserver.ssl.trust-store-password=${TSP} \
     -Djavax.net.ssl.keyStore=${KS} \
     -Djavax.net.ssl.keyStorePassword=${KSP} \
     -Djavax.net.ssl.trustStore=${KS} \
     -Djavax.net.ssl.trustStorePassword=${KSP} \
     -jar ${JAR}

[planetf1]

Should include harvesting documentation added in PR #3238

[planetf1]

I've added some docs in the PR above - review comments welcome.