[BUG]Opensearch playbook issue with SSL configuration

[jnioche]

I followed the steps in the README.

The playbook fails on TASK [linux/opensearch : Wait for opensearch to startup]

When inspecting one of the servers, I find that Opensearch is not running. Its logs end in

Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
    at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422) ~[?:?]
    at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
    at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
    at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
    at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
    at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
    at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
    at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]

Looking at the documentation for TLS, I can't see any reference to _plugins.security.ssl.transport.client.pemcertfilepath nor _plugins.security.ssl.transport.server.pemcertfilepath.

The config dir contains a number of .key and .pem files.

-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 admin.key
-rw------- 1 opensearch opensearch  1537 Mar  3 10:12 admin.pem
-rw------- 1 opensearch opensearch  2204 Mar  3 13:42 jvm.options
drwxr-x--- 2 opensearch opensearch  4096 Jan 14 03:38 jvm.options.d
-rw-rw---- 1 opensearch opensearch 11646 Jan 14 03:42 log4j2.properties
-rw-rw---- 1 opensearch opensearch   196 Mar  3 10:28 opensearch.keystore
drwxr-x--- 2 opensearch opensearch  4096 Jan 14 03:59 opensearch-observability
drwxr-x--- 2 opensearch opensearch  4096 Jan 14 03:59 opensearch-reports-scheduler
-rw------- 1 opensearch opensearch   198 Mar  3 13:43 opensearch.yml
-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 os2_http.key
-rw------- 1 opensearch opensearch  1602 Mar  3 10:11 os2_http.pem
-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 os2.key
-rw------- 1 opensearch opensearch  1602 Mar  3 10:11 os2.pem
-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 root-ca.key
-rw------- 1 opensearch opensearch  1342 Mar  3 10:11 root-ca.pem

The openseach.yml file contains

cluster.name: "frontier"

node.name: "os2"

network.host: "xxx.xxx.xxx.xxx"   <- EDITED

http.port: 9200

bootstrap.memory_lock: true

discovery.seed_hosts: ["os1","os2","os3","os4"]

node.roles: [data,master]

If I add

plugins.security.ssl.transport.pemkey_filepath: os2.key 
plugins.security.ssl.transport.pemcert_filepath: os2.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

to the config file and restart Opensearch with systemctl restart opensearch, it goes past the error.

Shouldn't the configuration have been created correctly by the playbook?

Thanks

[gadgetmerc]

I hit the same issue. After some digging I noticed that the plugins.security.ssl. settings come from opensearch/tasks/security.yml. It only adds the snippet if the local /tmp/opensearch-nodecerts changes. But if there is a change to opensearch.yml in future runs it will strip out that snippet.

[peterzhuamazon]

@saravanan30erd anything we can do to help improve this? Thanks.

[saravanan30erd]

@peterzhuamazon we need to support the script for safe re-run, will work on this.

[jnioche]

Thanks. In the meantime, is there a way to circumvent the issue? e.g. something I can do on the target servers to fully reinstall from scratch?

[Slarker0k]

Hello, is there currently any workaround for this?

[saravanan30erd]

I hit the same issue. After some digging I noticed that the plugins.security.ssl. settings come from opensearch/tasks/security.yml. It only adds the snippet if the local /tmp/opensearch-nodecerts changes. But if there is a change to opensearch.yml in future runs it will strip out that snippet.

@gadgetmerc @jnioche Actually when we created this script, its focused only on first time installation because RPM/deb packages are not available that time so upgrade process is not straight forward. We will work on proper upgrade process soon.

For now, I created a quick workaround for this issue. opensearch.yml config file is completely overwrited on re-run which causing this issue. Instead of whole file copy, now I am verifying and just copying only the content (lines) so it will not overwrite the whole file on next run.

[demanuPL]

I'm still getting the same error @jnioche, problem is not solved.

Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:487) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:298) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:204) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:256) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:308) ~[?:?]
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:794) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:743) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:544) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:196) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.node.Node.<init>(Node.java:493) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.node.Node.<init>(Node.java:420) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.14.0.jar:2.14.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.14.0.jar:2.14.0]```

[DarkHostDH]

I'm still getting the same error @jnioche, problem is not solved.

@demanuPL Just delete /tmp/opensearch-nodecerts folder and retry again.