search

opensearch-project / security

🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
199 stars 276 forks source link

[BUG] OpenID connection with certificate verification from keystore. #4060

Open GypsyJR777 opened 9 months ago

GypsyJR777 commented 9 months ago

What is the bug? An error occurs when using OpenID and PKCS12:

[2024-02-20T10:29:39,964][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [node-01] Error creating JWT authenticator. JWT authentication will not work
com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:337) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:195) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:116) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:130) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:40) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:80) [opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) [opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:406) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:310) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:87) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:281) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:406) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:395) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:379) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.lambda$new$0(ConfigurationRepository.java:221) [opensearch-security-1.3.0.jar:1.3.0]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: Keystore is closed
        at org.opensearch.common.settings.KeyStoreWrapper.ensureOpen(KeyStoreWrapper.java:672) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.KeyStoreWrapper.getString(KeyStoreWrapper.java:593) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:204) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:194) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting.get(SecureSetting.java:116) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:96) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:92) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:341) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 22 more
[2024-02-20T10:29:39,972][WARN ][o.o.s.s.ReflectionHelper ] [node-01] Unable to enable 'com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator' due to java.lang.reflect.InvocationTargetException
[2024-02-20T10:29:39,980][ERROR][o.o.s.s.DynamicConfigModelV7] [node-01] Unable to initialize auth domain openid_auth_domain=AuthcDomain [http_enabled=true, transport_enabled=true, order=0, http_authenticator=HttpAuthenticator [challenge=false, type=openid, config={openid_connect_url=my_url, openid_connect_idp={enable_ssl=true, verify_hostnames=false}, jwks_uri=my_uri, subject_key=preferred_username, roles_key=realm_access, roles_sub_key=roles}], authentication_backend=AuthcBackend [type=noop, config={}], description=Authenticate via proxy] due to OpenSearchException[java.lang.reflect.InvocationTargetException]; nested: InvocationTargetException; nested: RuntimeException[com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12]; nested: SSLConfigException[Error loading trust store from /etc/opensearch/certs/self/node.p12]; nested: IllegalStateException[Keystore is closed];
org.opensearch.OpenSearchException: java.lang.reflect.InvocationTargetException
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:73) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.newInstance(DynamicConfigModelV7.java:406) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.buildAAA(DynamicConfigModelV7.java:310) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigModelV7.<init>(DynamicConfigModelV7.java:87) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:281) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:406) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:395) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:379) [opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.configuration.ConfigurationRepository.lambda$new$0(ConfigurationRepository.java:221) [opensearch-security-1.3.0.jar:1.3.0]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more
Caused by: java.lang.RuntimeException: com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:85) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more
Caused by: com.amazon.dlic.util.SettingsBasedSSLConfigurator$SSLConfigException: Error loading trust store from /etc/opensearch/certs/self/node.p12
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:337) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:195) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:116) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:130) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:40) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:80) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more
Caused by: java.lang.IllegalStateException: Keystore is closed
        at org.opensearch.common.settings.KeyStoreWrapper.ensureOpen(KeyStoreWrapper.java:672) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.KeyStoreWrapper.getString(KeyStoreWrapper.java:593) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:204) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting$SecureStringSetting.getSecret(SecureSetting.java:194) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.common.settings.SecureSetting.get(SecureSetting.java:116) ~[opensearch-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:96) ~[opensearch-security-1.3.0.jar:1.3.0]
        at org.opensearch.security.ssl.SecureSSLSettings$SSLSetting.getSetting(SecureSSLSettings.java:92) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.initFromKeyStore(SettingsBasedSSLConfigurator.java:341) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.configureWithSettings(SettingsBasedSSLConfigurator.java:200) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLContext(SettingsBasedSSLConfigurator.java:120) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.util.SettingsBasedSSLConfigurator.buildSSLConfig(SettingsBasedSSLConfigurator.java:134) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.getSSLConfig(HTTPJwtKeyByOpenIdConnectAuthenticator.java:65) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.initKeyProvider(HTTPJwtKeyByOpenIdConnectAuthenticator.java:40) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.AbstractHTTPJwtAuthenticator.<init>(AbstractHTTPJwtAuthenticator.java:80) ~[opensearch-security-1.3.0.jar:1.3.0]
        at com.amazon.dlic.auth.http.jwt.keybyoidc.HTTPJwtKeyByOpenIdConnectAuthenticator.<init>(HTTPJwtKeyByOpenIdConnectAuthenticator.java:26) ~[opensearch-security-1.3.0.jar:1.3.0]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.security.support.ReflectionHelper.instantiateAAA(ReflectionHelper.java:62) ~[opensearch-security-1.3.0.jar:1.3.0]
        ... 9 more

How can one reproduce the bug? Steps to reproduce the behavior:

  1. Take some server
  2. Create security config like this: 
    
---
_meta:
config_version: 2
type: config

config: dynamic: http: anonymous_auth_enabled: false xff: enabled: false authc: openid_auth_domain: description: Authenticate via proxy http_enabled: true transport_enabled: true order: 0 http_authenticator: type: openid challenge: false config: openid_connect_url: my_url openid_connect_idp: enable_ssl: true verify_hostnames: false jwks_uri: my_uri subject_key: preferred_username roles_key: realm_access roles_sub_key: roles authentication_backend: type: noop basic_internal_auth_domain: description: "Authenticate via HTTP Basic against internal users database" http_enabled: true transport_enabled: true order: 1 http_authenticator: type: basic challenge: false authentication_backend: type: intern

3. Paste the excerpt into opensearch.yml:

plugins.security.ssl.transport: enabled: true keystore_type: PKCS12 truststore_type: PKCS12 keystore_filepath: "/etc/opensearch/certs/self/node.p12" truststore_filepath: "/etc/opensearch/certs/self/node.p12" enabled_protocols: ["TLSv1.2", "TLSv1.3"] enabled_ciphers: ["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"] enforce_hostname_verification: true


4. Start opensearch
5. See logs

**What is the expected behavior?**
Either do not allow work with keystore at all, or get the password and continue working without errors.

**What is your host/environment?**
 - OS: RHEL 8.1
 - Version opensearch 2.11.0.0
 - Plugins: default

**Do you have any screenshots?**
Nothing

**Do you have any additional context?**
No
derek-ho commented 8 months ago

[Triage] This seems like a bug and I am seeing OpenSearch 1.3 in the stacktrace. We would want to take a quick look at this and see if there is a bug that needs fixing.

GypsyJR777 commented 8 months ago

Version 1.3 is the versioning that I gave to my build. In fact, version 2.11 is used.