Closed planetf1 closed 3 years ago
Removing spring actuator - initialization stillno ferrors with the health configuration (as expected) but still an SSL issue with elastic search
15:51:02.077 [main] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchRestClient' defined in class path resource [org/springframework/boot/autoconfigure/elasticsearch/ElasticsearchRestClientConfigurations$RestClientFallbackConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.elasticsearch.client.RestClient]: Factory method 'elasticsearchRestClient' threw exception; nested exception is java.lang.IllegalStateException: could not create the default ssl context
later down the stacktrace we see:
Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)
at java.base/java.security.Provider$Service.newInstance(Provider.java:1900)
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:184)
at java.base/javax.net.ssl.SSLContext.getDefault(SSLContext.java:110)
at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:212)
... 37 common frames omitted
Caused by: java.security.KeyManagementException: problem accessing trust store
The above was with the zulu 15 jsk. Switching to openj9 11, or zulu 11 we get an extra 'caused' at the end:
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:792)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243)
at java.base/java.security.KeyStore.load(KeyStore.java:1479)
at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:365)
at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:313)
at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:55)
at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49)
... 52 common frames omitted
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:790)
Running the chassis where the current working directory has a 'good' truststore works ok ie:
jonesn:resources/ (master) $ pwd [15:59:40]
/Users/jonesn/src/egeria/open-metadata-implementation/server-chassis/server-chassis-spring/src/main/resources
jonesn:resources/ (master) $ java -jar ../../../target/server-chassis-spring-2.4-SNAPSHOT.jar
We would assume explicit specification of the truststore/keystore would be good too.
This tends to suggest either the classpath has changed inside the spring app, or the resources files were included are bad/not being picked up
The assembly version of the app works ok.
This is probably worth cleaning up alongside completing #1403
2.2 behaves the same in my environment, so at least not a regression for the release. Will check 2.0/1 later. Could be environment specific
It should work like that: provide trust-store or use strict.ssl=false.
Egeria internal clients are using the same mechanism as strict.ssl=false but without possibility to get rid of it. I suggest to be removed and use sttrict.ssl=false to avoid ssl check.
The problem is that we have a dependency that is trying to use an ssl connection which would be trusted with JVM truststore, but we override it with egeria one.
For starting server chassis with default configuration and without truststore, we can add a validation to check file exists. If not exists, we do not override existing JVM truststore. Platform will start, but, but not able to connect to other Egeria platform. Or maybe yes, as in the SpringRESTClientConnector is hardcoded disable SSL verification.
Another remark here. Hardcoding disabling SSL check mechanism in the constructor of SpringRESTClientConnector, beyond the fact that cannot be undo in production, it is also redundant, as we have multiple instances of SpringRESTClientConnector and the methods HttpsURLConnection.setDefaultHostnameVerifier and HttpsURLConnection.setDefaultSSLSocketFactory are static. Not needed to be called for all instances of SpringRESTClientConnector.
Understand there's more work (at least docs) in this area which have been on hold due to other work - thanks for the comments - will assign to myself and look at it along with 1403
I will refer to ongoing updates in #1403 but will use this issue to document current behaviour I will also raise specific issues for changes such as
Finally to a) help anyone hitting this error now, and b) for your comments here is a launch script I use for the chassis:
#!/bin/sh
BUILD=/Users/jonesn/src/egeria
KS=${BUILD}/open-metadata-implementation/server-chassis/server-chassis-spring/src/main/resources/keystore.p12
TS=${BUILD}/open-metadata-implementation/server-chassis/server-chassis-spring/src/main/resources/keystore.p12
KSP=egeria
TSP=egeria
SKEY=tomcat
VER=2.4-SNAPSHOT
JAR=${BUILD}/open-metadata-implementation/server-chassis/server-chassis-spring/build/libs/server-chassis-spring-${VER}.jar
# Assuming no configuration is done explcitly within Egeria itsef:
#
# Settings used by the Egeria 'server' (what we call the Platform) (not clients) - ie when accepting inbound connections
#
# server.ssl-key-store -- Used by tomcat/spring boot to locate keys that identify the server
# server.ssl-key-alias -- Used by tomcat/spring boot to identify the alias of the key tomcat should use for itself
# server.ssl.key-store-password -- Used by tomcat/spring boot for the keystore password (2 way SSL)
# server.ssl.trust-store -- Used by tomcat/spring boot to understand what clients it can trust (2 way SSL)
# server.ssl.trust-store-password -- Used by tomcat/spring boot for the password of the truststore (2 way SSL)
# strict.ssl -- true/false -- sets full checking for all certs (server-side)
#
# Settings used by the Egeria 'server' -- and indeed our clients or other java programmes using egeria libraries, acting in the client role (for JAVA)
#
# javax.net.ssl.keyStore -- keyStore for client to use (2 way SSL needs this)
# javax.net.ssl.keyStorePassword -- password for the keystore (2 way SSL needs this)
# javax.net.ssl.trustStore -- trustStore for the client to use (always needs setting as egeria makes client calls)
# javax.net.ssl.trustStorePassword -- password for the truststore (always - as above)
# Any specification of which cert a client used would need to be coded within the app, as is strict cert checking (currently disabled)
#
# Based on this for a fully specified launch of the chassis we would therefore use:
java \
-Dserver.ssl.key-store=${KS} \
-Dserver.ssl.key-alias=${SKEY} \
-Dserver.ssl.key-store-password=${KSP} \
-Dserver.ssl.trust-store=${TS} \
-Dserver.ssl.trust-store-password=${TSP} \
-Djavax.net.ssl.keyStore=${KS} \
-Djavax.net.ssl.keyStorePassword=${KSP} \
-Djavax.net.ssl.trustStore=${KS} \
-Djavax.net.ssl.trustStorePassword=${KSP} \
-jar ${JAR}
Should include harvesting documentation added in PR #3238
I've added some docs in the PR above - review comments welcome.
The egeria chassis fails to launch in a default configuration when run from the 'target' directory.
The full log is at https://gist.github.com/ffff1f08a18853f9d01297a57dce4075
The root issue appears to be
Caused by: java.lang.IllegalStateException: could not create the default ssl context
- but this is occuring when the java is launched directly ie:which previously worked.
This may be a classpath issue or a change in default configuration
Specifically it's possible spring boot actuator is causing a problem here -- for example we see references to 'health contributor' and specifically
org/springframework/boot/actuate/autoconfigure/elasticsearch/ElasticSearchRestHealthContributorAutoConfiguration.class