search

opensearch-project / dashboards-notifications

The Notifications plugin provides a central location for all of your notifications from OpenSearch plugins. You can use either OpenSearch Dashboards or the REST API to configure notifications. Dashboards offers a more organized way of selecting a channel type and selecting which OpenSearch plugin sources you want to use.
Apache License 2.0
9 stars 37 forks source link

[FEATURE] Webhook support custom SSL on the target #76

Open LHozzan opened 1 year ago

LHozzan commented 1 year ago

Is your feature request related to a problem? My current goal is provide our developers ability to watch over time some conditions and when reached matching criteria, they will be noticed via Prometheus AlertManager. Unfortunately, we would like to have more secure infrastructure, so internal AlertManager endpoint is secured via SSL certificate, which is signed by our internal CA. And here is a problem with supporting that scenario.

What solution would you like? Have some settings (in the OS Dashboard GUI preferable, but settings in config files are fine too) to specify custom CA, which is used to signed different endpoints for webhook targets.

What alternatives have you considered? A some way, how to set or specify custom CA certificate for webhook endpoint.

Do you have any additional context? If the endpoint isnt secured, everything working fine, but we need rise security. I asked on the forum and I was pointed to the old closed issue with workaround. Unfortunately, with OpenSearch version 2.7.0 the workaround not working. If I made trustore with keytool utility from the Docker image, JKS trustore looks good for the utility, but OpenSearch refuse start properly with bunch of warnings and errors. Looks like:

{"type": "logging", "timestamp": "2023-07-14T12:06:53,170Z", "level": "ERROR", "component": "o.o.b.OpenSearchUncaughtExceptionHandler", "cluster.name": "logging", "node.name": "ofd-client-77f4d94d7c-84nsr", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.opensearch.bootstrap.StartupException: org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.7.0.jar:2.7.0]",
"at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.7.0.jar:2.7.0]",
"Caused by: org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:83) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.security.KeyStoreException: problem accessing trust store",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:73) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
uncaught exception in thread [main]
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.io.IOException: keystore password was incorrect",
"at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]",
"at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242) ~[?:?]",
"at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]",
"at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more",
"Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.",
"at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159) ~[?:?]",
"at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242) ~[?:?]",
"at java.security.KeyStore.load(KeyStore.java:1473) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390) ~[?:?]",
"at sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336) ~[?:?]",
"at sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57) ~[?:?]",
"at sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49) ~[?:?]",
"at javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282) ~[?:?]",
"at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168) ~[?:?]",
"at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81) ~[?:?]",
"at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160) ~[?:?]",
"at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130) ~[?:?]",
"at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129) ~[?:?]",
"at org.opensearch.node.Node.lambda$new$16(Node.java:788) ~[opensearch-2.7.0.jar:2.7.0]",
"at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]",
"at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]",
"at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]",
"at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]",
"at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]",
"at org.opensearch.node.Node.<init>(Node.java:802) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.node.Node.<init>(Node.java:375) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.7.0.jar:2.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.7.0.jar:2.7.0]",
"... 6 more"] }
org.opensearch.common.ssl.SslConfigException: failed to initialize a TrustManager for the system keystore
Likely root cause: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2159)
        at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:242)
        at java.base/java.security.KeyStore.load(KeyStore.java:1473)
        at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.loadKeyStore(TrustStoreManager.java:390)
        at java.base/sun.security.ssl.TrustStoreManager$TrustAnchorManager.getTrustedCerts(TrustStoreManager.java:336)
        at java.base/sun.security.ssl.TrustStoreManager.getTrustedCerts(TrustStoreManager.java:57)
        at java.base/sun.security.ssl.TrustManagerFactoryImpl.engineInit(TrustManagerFactoryImpl.java:49)
        at java.base/javax.net.ssl.TrustManagerFactory.init(TrustManagerFactory.java:282)
        at org.opensearch.common.ssl.KeyStoreUtil.createTrustManager(KeyStoreUtil.java:168)
        at org.opensearch.common.ssl.DefaultJdkTrustConfig.createTrustManager(DefaultJdkTrustConfig.java:81)
        at org.opensearch.common.ssl.SslConfiguration.createSslContext(SslConfiguration.java:155)
        at org.opensearch.index.reindex.ReindexSslConfig.reload(ReindexSslConfig.java:160)
        at org.opensearch.index.reindex.ReindexSslConfig.<init>(ReindexSslConfig.java:130)
        at org.opensearch.index.reindex.ReindexPlugin.createComponents(ReindexPlugin.java:129)
        at org.opensearch.node.Node.lambda$new$16(Node.java:788)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at org.opensearch.node.Node.<init>(Node.java:802)
        at org.opensearch.node.Node.<init>(Node.java:375)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        <<<truncated>>>
For complete error details, refer to the log at /usr/share/opensearch/logs/logging.log

For generating truststore I used the command:

/usr/share/opensearch/jdk/bin/keytool \
  -importcert -trustcacerts -noprompt -file /tmp/devel-ca.pem \
  -alias company-ca -keystore /shared/truststore.jks \
  -storepass password123

Java settings for OpenSearch Client node looks like:

-Djavax.net.ssl.trustStore=/usr/share/opensearch/config/truststore.jks
-Djavax.net.ssl.trustStoreType=JKS
-Djavax.net.ssl.trustStorePassword="password123"

If I check truststore with keytool, everything looks good:

/usr/share/opensearch/jdk/bin/keytool -list -v -keystore /shared/truststore.jks -storepass password123

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: company-ca
Creation date: Jul 14, 2023
Entry type: trustedCertEntry

Owner: CN=cluster:devel
Issuer: CN=fake-ca
Serial number: 5a70d3abd4521488e4f702589377772cdcae0c2d
Valid from: Fri Jun 23 07:18:00 UTC 2023 until: Sat Jun 21 07:18:00 UTC 2031
Certificate fingerprints:
         SHA1: 9E:BA:EA:44:A5:D0:2D:B4:7C:3A:95:AD:D7:A1:9E:26:AA:CB:F3:87
         SHA256: D3:C8:E9:8B:3D:1E:53:1D:15:55:3D:26:A1:7D:F1:CE:01:F3:E2:01:55:2C:2E:94:D4:2C:EA:3E:EA:FC:BC:12
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 78 C4 D4 ED 99 07 BF D0   A3 18 11 1F 43 A8 18 BA  x...........C...
0010: 5F 9C 11 B8                                        _...
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:0
]

#3: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

#4: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
  Key_CertSign
  Crl_Sign
]

#5: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 70 91 5C 95 80 1F 77 14   DE 7C 19 3A 48 0B D6 FA  p.\...w....:H...
0010: 1E 67 69 DC                                        .gi.
]
]

*******************************************
*******************************************

If you need some additional information, please, let me know.

scubbx commented 11 months ago

I have the same problem. Providing a custom keystore did not solve the situation.

Hailong-am commented 7 months ago

@LHozzan can you try import the cert into java default cacerts file which under $JAVA_HOME/lib/security folder

/usr/share/opensearch/jdk/bin/keytool \
  -importcert -trustcacerts -noprompt -file /tmp/devel-ca.pem \
  -alias company-ca -keystore $JAVA_HOME/lib/security/cacerts \
  -storepass changeit
LHozzan commented 7 months ago

Hi @Hailong-am .

Thank you for your effort and interest. Meantime we tested another opportunity to get notifications from OS Dash (via M$ Teams) and this solution looks usable for us.

I will let the feature request opened for anybody else, who required this feature due to some limitations on his side.

BR, Lu