Security TLS configuration fails

ericklife28 commented 10 months ago

What is the bug?

When I enable tls, I have decided to use self signed certificates using the opensearch cluster CA for the transport layer. However I want to create a custom certificate for the http layer. I am using this configuration:

`security:

config:  # Everything related to the securityconfig

  securityConfigSecret:

    name:  ${security_config_secret}

  adminCredentialsSecret:

    name:  ${admin_credentials_secret}

tls:

  transport:

    generate: true

    perNode: true

  http:

    generate: false # --> here I disable certificate generation 

    secret:

      name: ${tls_rest_secret_name} # Name of the secret that contains the provided certificate`

However, the opensearch-cluster-bootstrap fails with this error:

init-sysctl vm.max_map_count = 262144 opensearch [2023-12-23T13:50:23,243][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". opensearch [2023-12-23T13:50:23,244][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: Defaulting to no-operation (NOP) logger implementation opensearch [2023-12-23T13:50:23,244][WARN ][stderr ] [opensearch-cluster-bootstrap-0] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. opensearch [2023-12-23T13:50:23,254][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-bootstrap-0] SSL dual mode is disabled opensearch [2023-12-23T13:50:23,254][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-bootstrap-0] OpenSearch Config path is /usr/share/opensearch/config opensearch [2023-12-23T13:50:23,501][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-bootstrap-0] JVM supports TLSv1.3 opensearch [2023-12-23T13:50:23,503][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-bootstrap-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively opensearch [2023-12-23T13:50:23,555][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-bootstrap-0] uncaught exception in thread [main] opensearch org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.3.0.jar:2.3.0] opensearch at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.3.0.jar:2.3.0] opensearch Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.node.Node.(Node.java:420) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.node.Node.(Node.java:347) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0] opensearch ... 6 more opensearch Caused by: java.lang.reflect.InvocationTargetException opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] Stream closed EOF for opensearch-cluster/opensearch-cluster-bootstrap-0 (init) Stream closed EOF for opensearch-cluster/opensearch-cluster-bootstrap-0 (init-sysctl) opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?] opensearch at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] opensearch at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?] opensearch at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?] opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.node.Node.(Node.java:420) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.node.Node.(Node.java:347) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0] opensearch uncaught exception in thread [main] opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0] opensearch ... 6 more opensearch Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested. opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?] opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255) ~[?:?] opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:176) ~[?:?] opensearch at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218) ~[?:?] opensearch at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:262) ~[?:?] opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] opensearch at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?] opensearch at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] opensearch at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?] opensearch at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?] opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.node.Node.(Node.java:420) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.node.Node.(Node.java:347) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.3.0.jar:2.3.0] opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.3.0.jar:2.3.0] opensearch ... 6 more opensearch java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin] opensearch Likely root cause: OpenSearchException[plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.] opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:255) opensearch at org.opensearch.security.ssl.DefaultSecurityKeyStore.(DefaultSecurityKeyStore.java:176) opensearch at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.(OpenSearchSecuritySSLPlugin.java:218) opensearch at org.opensearch.security.OpenSearchSecurityPlugin.(OpenSearchSecurityPlugin.java:262) opensearch at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) opensearch at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) opensearch at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) opensearch at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) opensearch at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) opensearch at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) opensearch at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730) opensearch at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532) opensearch at org.opensearch.plugins.PluginsService.(PluginsService.java:195) opensearch at org.opensearch.node.Node.(Node.java:420) opensearch at org.opensearch.node.Node.(Node.java:347) opensearch at org.opensearch.bootstrap.Bootstrap$5.(Bootstrap.java:242) opensearch at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) opensearch at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) opensearch at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) opensearch at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) opensearch at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) opensearch at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) opensearch at org.opensearch.cli.Command.main(Command.java:101) opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) opensearch at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) opensearch For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log opensearch Killing performance analyzer process 34 opensearch OpenSearch exited with code 1 opensearch Performance analyzer exited with code 143 Stream closed EOF for opensearch-cluster/opensearch-cluster-bootstrap-0 (opensearch)

How can one reproduce the bug?

Using the security setup above

What is the expected behavior?

The startup process should run successfully

Do you have any additional context?

I am using the opensearch-cluster helm chart to apply this configuration. The values above correspond to the values.yaml used.

According to the documentation, It supposed that the tls configuraton for transport and http could be configured in an independent way. However, I wonder either if once you disable the generation for any of them we need to use our own CA for both of them. or if there is a missing setup that is not documented.

On the other hand. If I set the http > generate parameter to true

http: generate: true

I don't get any error, and the cluster start up process is executed successfully.

I appreciate your help.

Cheers.

ericklife28 commented 9 months ago

Hi, looking at the opensearch cluster chart. I noticed that this line could be the reason of the problem

https://github.com/opensearch-project/opensearch-k8s-operator/blob/f88d40d8ae679cb400826bffc4d7939db42edff0/charts/opensearch-cluster/templates/opensearch-cluster-cr.yaml#L284

As you can see there the transport.generate value is dependen of the http.generate value and it shouldn't be. Is there someone working on this?

ericklife28 commented 9 months ago

Hi I made the change above on the chart locally and it seems to work. However, I get a new error on the opensearch-cluster-securityconfig-update pod:

This tool will be deprecated in the next major release of OpenSearch https://github.com/opensearch-project/security/issues/1755

Will connect to opensearch-cluster.opensearch-cluster.svc.cluster.local:9200 ... done ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetSecurity Admin v7 See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting. Trace: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting. at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947) at org.opensearch.client.RestClient.performRequest(RestClient.java:332) at org.opensearch.client.RestClient.performRequest(RestClient.java:320) at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462) at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159) Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:288) at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:356) at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:547) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ... 19 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 24 more

Is there anyone working on this?. Let me know if you need more info to continue.

Cheers.

cyxou commented 7 months ago

@ericklife28 have you managed to find a solution? Facing the same issue with Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target error ((

Siradjedd commented 7 months ago

@cyxou any updates ?

cyxou commented 7 months ago

@cyxou any updates ?

Nope((

opensearch-project / opensearch-k8s-operator