search

camunda / camunda-platform-helm

Camunda Platform 8 Self-Managed Helm charts
https://docs.camunda.io/docs/self-managed/overview/
Apache License 2.0
74 stars 137 forks source link

[ISSUE] With opensearch self-signed cert, there is no way to supply password to JKS trust store, operate and tasklist fail #2359

Closed jessesimpson36 closed 1 month ago

jessesimpson36 commented 1 month ago

Describe the issue:

Our documentation https://docs.camunda.io/docs/next/self-managed/setup/guides/using-existing-elasticsearch/#connecting-to-existing-elasticsearch-with-a-self-signed-certificate

provides some commands that allows you to create a JKS trust store from a CA certificate file using the following commands:

keytool -import -alias elasticsearch -keystore externaldb.jks -storetype jks -file elastic.crt -storepass changeit -noprompt
kubectl  create secret -n camunda generic elastic-jks --from-file=externaldb.jks

The relevant option here is -storepass changeit, which password-protects the trust store, meaning to get any of the data inside the JKS trust store, you have to supply this password. However, the camunda helm chart does not have any way to supply the password to the JKS file.

Furthermore, the keytool command line tool does not allow for creating a JKS trust store without password protection.

It may be relevant to note that I am using the opensearch operator with mostly default settings, so by default, it deploys opensearch with TLS terminated in the application-level.

Actual behavior: Operate and tasklist fail with the following error:

javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1351) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169) ~[?:?]
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
    at org.apache.hc.core5.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:339) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:410) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.ssl.SSLIOSession.access$100(SSLIOSession.java:74) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSession.java:201) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:142) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44) ~[httpcore5-5.2.4.jar!/:5.2.4]
    at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
    at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
    at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
    at java.base/sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
    ... 19 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
    at java.base/sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
    at java.base/sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
    at java.base/java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
    at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
    at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
    at java.base/sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
    at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329) ~[?:?]
    ... 19 more

2024-09-18 20:05:07.855  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #1 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:10.903  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #2 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:13.927  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #3 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:16.969  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #4 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:19.992  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #5 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:23.033  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #6 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:26.053  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #7 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:29.070  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #8 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:32.094  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #9 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:35.137  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #10 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:38.156  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #11 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:41.200  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #12 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
2024-09-18 20:05:44.217  INFO 7 --- [           main] i.c.t.o.OpenSearchConnector              : Retrying #13 connect to OpenSearch at https://camunda-opensearch.opensearch-operator:9200 due to javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Expected behavior:

How to reproduce:

Logs:

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

jessesimpson36 commented 1 month ago

I mark this as code-freeze-candidate because the issue is not a regression as result of 8.6-alpha, it is simply a problem that exists both in 8.5 and 8.6

jessesimpson36 commented 1 month ago

https://github.com/camunda/distribution/issues/306

jessesimpson36 commented 1 month ago

@hamza-m-masood informed me that the storepass is not relevant to the stacktrace I received.

hamza-m-masood commented 1 month ago

I will work on this issue

hamza-m-masood commented 1 month ago

I will test this issue by setting up external bitnami elasticsearch, with ingress and self-signed cert. Then, I will connect using the camunda helm chart and the self-signed cert.

hamza-m-masood commented 1 month ago

I can confirm that connecting to external elasticserach with self-signed certificates works with alpha chart

hamza-m-masood commented 1 month ago

The team has decided to leave this issue open just incase we end up supporting OpenSearch outside of just AWS.

hamza-m-masood commented 1 month ago

I will close this issue since this is more a feature request rather than an issue with our existing setup.