search

opensearch-project / security

🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
180 stars 263 forks source link

[BUG] Exception when reloading self signed certs via REST API #4480

Open reneradoi opened 1 week ago

reneradoi commented 1 week ago

What is the bug? As part of the TLS cert renewal process I want to use the REST API for hot reload of transport and http certificates. When reloading the already used certificates, everything works fine (response code 200). But when I want to apply newly generated, self signed certificates, I always run into the same exception telling me that new certs do not have valid Issuer DN, Subject DN or SAN.

I’ve checked my configuration and the validity of the certs, everything seems to be fine here. If the application is restarted, the new certs are also picked up without any issue. Only reloading via the API produces this error.

How can one reproduce the bug? Steps to reproduce the behavior:

What is the expected behavior?

What is your host/environment?

Do you have any screenshots? API response:

{"error":{"root_cause":[{"type":"i_o_exception","reason":"OpenSearchSecurityException[Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];"}],"type":"i_o_exception","reason":"OpenSearchSecurityException[Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];","caused_by":{"type":"security_exception","reason":"Error while initializing http SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.","caused_by":{"type":"exception","reason":"New Certs do not have valid Issuer DN, Subject DN or SAN."}}},"status":500}

Server logfile:

[2024-06-21T09:04:26,877][WARN ][r.suppressed             ] [opensearch-30] path: /_plugins/_security/api/ssl/transport/reloadcerts, params: {certType=transport}
java.io.IOException: OpenSearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.]; nested: Exception[New Certs do not have valid Issuer DN, Subject DN or SAN.];
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:203) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.lambda$securitySSLCertsRequestHandlers$3(SecuritySSLCertsApiAction.java:117) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.validation.ValidationResult.valid(ValidationResult.java:87) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.lambda$securitySSLCertsRequestHandlers$5(SecuritySSLCertsApiAction.java:105) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.RequestHandler$RequestHandlersBuilder.lambda$add$2(RequestHandler.java:97) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.AbstractApiAction.lambda$prepareRequest$23(AbstractApiAction.java:614) [opensearch-security-2.14.0.0.jar:2.14.0.0]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) [?:?]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [?:?]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) [opensearch-2.14.0.jar:2.14.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:484) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:190) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 11 more
Caused by: java.lang.Exception: New Certs do not have valid Issuer DN, Subject DN or SAN.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.validateNewCerts(DefaultSecurityKeyStore.java:644) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:462) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        at org.opensearch.security.dlic.rest.api.SecuritySSLCertsApiAction.reloadCertificates(SecuritySSLCertsApiAction.java:190) ~[opensearch-security-2.14.0.0.jar:2.14.0.0]
        ... 11 more

Do you have any additional context? Relevant part of the config file opensearch.yml:

plugins.security.authcz.admin_dn:
- O=opensearch-x7ga,CN=admin
plugins.security.ssl.transport.pemcert_filepath: certificates/unit-transport.cert
plugins.security.ssl.transport.pemkey_filepath: certificates/unit-transport.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/root-ca.cert
plugins.security.ssl.http.pemcert_filepath: certificates/unit-http.cert
plugins.security.ssl.http.pemkey_filepath: certificates/unit-http.key
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/root-ca.cert
plugins.security.ssl.http.clientauth_mode: OPTIONAL
cluster.name: opensearch-x7ga
node.name: opensearch-30
node.roles:
- data
- ingest
- ml
- coordinating_only
- cluster_manager
discovery.seed_providers: file
plugins.security.disabled: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.restapi.roles_enabled:
- all_access
- security_rest_api_access
plugins.security.unsupported.restapi.allow_securityconfig_modification: true
plugins.security.ssl_cert_reload_enabled: true

I did a lot of debugging on this, output (incl. the current and new cert) can be found here. The essential part is this: When comparing the SAN lists of the current and the new cert, the maps are being sorted. This sort function breaks:

opensearch[opensearch-31][generic][T#11][1] print currentCertDNList
 currentCertDNList = "[CN=CN_CA,C=US/O=opensearch-w4du,CN=10.27.170.243/[[2, juju-1e4384-35], [2, juju-1e4384-35.lxd], [2, opensearch-31], [7, 10.27.170.243], [8, 1.2.3.4.5.5]]]"
opensearch[opensearch-31][generic][T#11][1] print newCertDNList
 newCertDNList = "[CN=CN_CA,C=US/O=opensearch-w4du,CN=10.27.170.243/[[2, opensearch-31], [2, juju-1e4384-35], [2, juju-1e4384-35.lxd], [7, 10.27.170.243], [8, 1.2.3.4.5.5]]]"

It's happening an this position in the source code: https://github.com/opensearch-project/security/blob/3dae7a75692b058a460aa2a9dda1d0aa5faae0ef/src/main/java/org/opensearch/security/ssl/DefaultSecurityKeyStore.java#L675-L679

willyborankin commented 1 week ago

Thank you @reneradoi, it is known issue. We are planning to add https://github.com/opensearch-project/security/issues/4427 as a first step to fix hot reload problem.

reneradoi commented 6 days ago

Hi @willyborankin I'm not sure if it's the same issue. The feature request you linked is about the ssl reload only being performed on the corrdinator node instead of the whole cluster, but in this issue I'm encountering the certs cannot be reloaded at all, also not to a single node.

As far as I've debugged this is because of a sorting operation when validating the new cert against the current one (reg. issuer, subject and sans).

willyborankin commented 6 days ago

Hi @reneradoi, agree my comment is misleading sorry about that.

scrawfor99 commented 6 days ago

[Triage] Hi @reneradoi thank you for filing this issue. Looks like a real issue, and something that should be fixed. Going to mark as triaged.

willyborankin commented 6 days ago

I'm working on hot reload, so will try to fix this one as well.