search

odpi / egeria

Egeria core
https://egeria-project.org
Apache License 2.0
807 stars 261 forks source link

Starting egeria fails after update to 2.10(docs, usability) #5093

Closed exonianp closed 3 years ago

exonianp commented 3 years ago 
dimitrisv@dimitrisv-VirtualBox:~/egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target$ java -jar server-chassis-spring-2.10-SNAPSHOT.jar
 Project Egeria - Open Metadata and Governance
    ____   __  ___ ___    ______   _____                                 ____   _         _     ___
   / __ \ /  |/  //   |  / ____/  / ___/ ___   ____ _   __ ___   ____   / _  \ / / __    / /  / _ /__   ____ _  _
  / / / // /|_/ // /| | / / __    \__ \ / _ \ / __/| | / // _ \ / __/  / /_/ // //   |  / _\ / /_ /  | /  _// || |
 / /_/ // /  / // ___ |/ /_/ /   ___/ //  __// /   | |/ //  __// /    /  __ // // /  \ / /_ /  _// / // /  / / / /
 \____//_/  /_//_/  |_|\____/   /____/ \___//_/    |___/ \___//_/    /_/    /_/ \__/\//___//_/   \__//_/  /_/ /_/

 :: Powered by Spring Boot (v2.4.4) ::

2021-04-28 16:00:18.467  INFO 48842 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 9443 (https)
2021-04-28 16:00:22.212 ERROR 48842 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.5.jar!/:5.3.5]
    at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.5.jar!/:5.3.5]
    at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.5.jar!/:5.3.5]
    at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
    at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.5.jar!/:5.3.5]
    at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.5.jar!/:5.3.5]
    at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.5.jar!/:5.3.5]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.5.jar!/:5.3.5]
    at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:769) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1313) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1302) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.odpi.openmetadata.serverchassis.springboot.OMAGServerPlatform.main(OMAGServerPlatform.java:93) ~[classes!/:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[server-chassis-spring-2.10-SNAPSHOT.jar:na]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:107) ~[server-chassis-spring-2.10-SNAPSHOT.jar:na]
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[server-chassis-spring-2.10-SNAPSHOT.jar:na]
    at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467) ~[server-chassis-spring-2.10-SNAPSHOT.jar:na]
Caused by: org.springframework.boot.web.server.WebServerException: Unable to start embedded Tomcat server
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:229) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.web.servlet.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:43) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.5.jar!/:5.3.5]
    ... 23 common frames omitted
Caused by: java.lang.IllegalArgumentException: standardService.connector.startFailed
    at org.apache.catalina.core.StandardService.addConnector(StandardService.java:243) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.addPreviouslyRemovedConnectors(TomcatWebServer.java:282) ~[spring-boot-2.4.4.jar!/:2.4.4]
    at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.start(TomcatWebServer.java:213) ~[spring-boot-2.4.4.jar!/:2.4.4]
    ... 25 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Protocol handler start failed
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1074) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.catalina.core.StandardService.addConnector(StandardService.java:239) ~[tomcat-embed-core-9.0.45.jar!/:na]
    ... 27 common frames omitted
Caused by: java.lang.IllegalArgumentException: /home/dimitrisv/egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/truststore.p12 (No such file or directory)
    at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:99) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:71) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:258) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1204) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.AbstractEndpoint.start(AbstractEndpoint.java:1290) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.coyote.AbstractProtocol.start(AbstractProtocol.java:614) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.catalina.connector.Connector.startInternal(Connector.java:1071) ~[tomcat-embed-core-9.0.45.jar!/:na]
    ... 29 common frames omitted
Caused by: java.io.FileNotFoundException: /home/dimitrisv/egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/truststore.p12 (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method) ~[na:na]
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219) ~[na:na]
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157) ~[na:na]
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112) ~[na:na]
    at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86) ~[na:na]
    at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184) ~[na:na]
    at org.apache.catalina.startup.CatalinaBaseConfigurationSource.getResource(CatalinaBaseConfigurationSource.java:118) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.SSLUtilBase.getStore(SSLUtilBase.java:197) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.SSLHostConfig.getTruststore(SSLHostConfig.java:730) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.SSLUtilBase.getTrustManagers(SSLUtilBase.java:423) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) ~[tomcat-embed-core-9.0.45.jar!/:na]
    at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:97) ~[tomcat-embed-core-9.0.45.jar!/:na]

Issue with the truststore.p12: It does not exist in the egeria/open-metadata-implementation/server-chassis/server-chassis-spring/target/ directory. After copying them there it runs.

planetf1 commented 3 years ago

I agree that if the truststore.p12 is not found the startup will fail (and the error could be better...) but I don't think the code here has changed recently.

When we run the egeria server chassis for our FVTs, and in our container environment we typically launch it from the 'server' subdirectory ie:

jonesn:egeria-omag-2.10-SNAPSHOT/ (master*) $ pwd                                                                     
/Users/jonesn/src/egeria/open-metadata-distribution/open-metadata-assemblies/target/egeria-2.10-SNAPSHOT-distribution/egeria-omag-2.10-SNAPSHOT
jonesn:egeria-omag-2.10-SNAPSHOT/ (master*) $ java -jar server/server-chassis-spring-2.10-SNAPSHOT.jar                                        
 Project Egeria - Open Metadata and Governance

.... rest of server startup

In this case the certs (and in particular the truststore) should be picked up in the correct location

There's some more info on the TLS setup at https://egeria.odpi.org/open-metadata-implementation/admin-services/docs/user/omag-server-platform-transport-level-security.html - this covers the potential command line parameters that may be needed to reference the tls configuration in different locations

I do agree it's not particularly clear!! -- any suggestions would be welcome on improving that - but I'm intrigued if you believe something has changed. Since 2.9? The truststore.p12 wasn't in the 'target' directory in 2.9 either. There is however a copy in the source root directory. Could it be your current working directory is different to before?

I do expect we will make some further changes to the cert setup in future releases so any additional suggestions/input would be most welcome.

exonianp commented 3 years ago

I feel that egeria should have its own certificate issuing authority that will be totally independent of any other certificate generation in the organisation.

The paradigm set by Lotus Notes/Domino 30 years ago can be applied here as well. So rather than assume that the client will have in place a CA Authority etc. to issue certificates, Egeria (as a solution) should include a trusted cert generation mechanism. I guess that at the time Lotus included this infrastructure, simply because SSL/certs were a luxury by many companies. So they had it built in. If a specific OU was compromised you could just as well expire the cert of the OU and issue new ids to users.

Nowadays, we have the opposite problem. Everybody can have its certs to ensure the TLS communication but the certs per se lost the... actual authority once had to as part and parcel of the identity of the user during its creation (and revocation).

So Egeria upon building itself should have its own independent certificate and then each new "client" will be issued a certificate by Egeria in order to connect/authenticate with Egeria.

There are some open source projects like EJBCA that can be used to this end.

When a company gets compromised, you can instantly expire the certificates and ensure that no further access to data is happening.

It is obvious to all of us that the metadata repositories are/will be the "crown jewels" of any company and thus a primary target during attacks.

planetf1 commented 3 years ago

My own feeling is that certificate management overall is very much outside the scope of egeria. It is certainly a critical task within an organization. For larger orgs they will have chosen specific tools and have trained staff who know how to manage those certs properly.

This may be a difference between Egeria as an open source project, and a commercial distribution including egeria, where there may be integration to the rest of that 'stack' - whatever it may be - cloud or on-prem. There may be management there of egeria specific certs for example....

In Egeria we need to sure we provide all the correct docs, configuration, that they need to do this job well. Really well. It would be so useful to have someone experienced with this side involved in making sure we've done the best we can.

We also have smaller orgs who are more likely to use simpler/cheaper tools. At one extreme of this is the approach we're leaning towards currently which is to document / provide example of very basic scenario with out of the box openssl tools. Think more developer use case at this level. The certificate recovation example could be covered too -- good point.

There is another potential approach in-between which is to use some well known public certificate management infrastructures - let's encrypt etc. a detailed walkthrough (see blog below?) Also with infrastructure like 'let's encrypt' there is a possibility that certs can be dynamically created. This looks appealing for the simple org cases, but of course requires time to work through.

I do think improvements in the docs are very valuable, and some 'scenarios' where we work through the specifics with a certain provider in a blog post. This would need partnering with someone who uses that technology so that we can cover from both angles.

This could be a good discussion for a community call

planetf1 commented 3 years ago

The post above I think captures our current feeling around scope of egeria. If you would like to contribute improved docs (or report issues), scripting, or a walk through of configuring with another cert provider that would be most welcome. Closing for now as this does not appear to be a regression in current code