Role fails when pointed to es_ssl_keystore / es_ssl_truststore using CA and Cert created with bin/elasticsearch-certutil

[brsolomon-deloitte]

• Elasticsearch version 7.15.0
• Role version: 7.15.0
• JVM version (java -version): 11.0.11
• OS version (uname -a if on a Unix-like system): Ubuntu Focal 20.04 Server x86_64

Description of the problem including expected versus actual behaviour:

The elasticsearch role fails when pointed to es_ssl_keystore / es_ssl_truststore using CA and Cert created with bin/elasticsearch-certutil.

The role bombs out with

org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration

Playbook:

Install Ansible for ubuntu user on control node:

sudo apt-get update -y
sudo apt-get install -y python3.8 python3.8-venv

sudo -i
cd /opt
curl -fsSL -O https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
rm get-pip.py
exit
python3 -m pip install --user -U pip wheel setuptools
python3 -m pip install --user 'ansible==2.9.*'

echo 'export PATH=${HOME}/.local/bin:$PATH' >> ~/.bashrc

Download elasticsearch binaries on control node so that we can run bin/elasticsearch-certutil:

sudo -i
cd /opt
curl -fsSL -O \
    https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.0-linux-x86_64.tar.gz
curl -fsSL -O \
    https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.0-linux-x86_64.tar.gz.sha512
shasum -a 512 -c elasticsearch-7.15.0-linux-x86_64.tar.gz.sha512 
tar -xzf elasticsearch-7.15.0-linux-x86_64.tar.gz
rm *.tar.gz
rm *.sha512
cd elasticsearch-7.15.0/

# generates a new certificate authority (CA)
# produces a single PKCS#12 output file, which holds the CA certificate and the private key for the CA
bin/elasticsearch-certutil ca --out /opt/my-ca.p12 --pass ""

cat << EOF > /opt/certutil-input.yaml
---
instances:
  - name: "es-1" 
    ip: 
      - "XXX.YY.ZZ.58"
    dns: 
      - "es-1.XXXXXX.lab"
  - name: "es-2" 
    ip: 
      - "XXX.YY.ZZ.57"
    dns: 
      - "es-2.XXXXXX.lab"
  - name: "es-3" 
    ip: 
      - "XXX.YY.ZZ.56"
    dns: 
      - "es-3.XXXXXX.lab"
EOF

# generates X.509 certificates and private keys
bin/elasticsearch-certutil cert --multiple \
    --ca /opt/my-ca.p12 --out /opt/my-keystore.p12 --pass "" \
    --in /opt/certutil-input.yaml

cd /opt
sudo chown ubuntu:ubuntu /opt/*.p12
exit

Create variables file:

cat << EOF > es-vars.yaml
---
es_version: "7.15.0"
oss_version: false

es_heap_size: 16g
es_api_basic_auth_username: "elastic"
es_api_basic_auth_password: "changeme"
es_enable_http_ssl: true
es_ssl_keystore: /opt/my-keystore.p12
es_ssl_truststore: /opt/my-ca.p12
es_validate_certs: false
EOF

Create playbook for master node:

cat << EOF > es-pb-1.yaml 
- hosts: master_node
  roles:
    - role: elastic.elasticsearch
  vars:
    es_config:
      cluster.name: "test-cluster"
      cluster.initial_master_nodes: "es-1"
      discovery.seed_hosts: "es-1:9300"
      http.host: 0.0.0.0
      http.port: 9200
      node.data: false
      node.master: true
      transport.host: 0.0.0.0
      transport.port: 9300
      bootstrap.memory_lock: false
EOF

Run:

ansible-playbook -kK -v -e '@es-vars.yaml' es-pb-1.yaml

Provide logs from Ansible:

Tail of /var/tmp/ansible.log:

in/elasticsearch-keystore remove 'xpack.security.http.ssl.keystore.secure_password'", "delta": "0:00:00.670811", "end": "2021-10-04 15:15:21.656795", "item": "http", "msg": "non-zero return code", "rc": 78, "start": "2021-10-04 15:15:20.985984", "stderr": "\nERROR: Setting [xpack.security.http.ssl.keystore.secure_password] does not exist in the keystore.", "stderr_lines": ["", "ERROR: Setting [xpack.security.http.ssl.keystore.secure_password] does not exist in the keystore."], "stdout": "", "stdout_lines": []}
2021-10-04 15:15:22,571 p=109151 u=ubuntu n=ansible | failed: [es-1.XXXXXX.lab] (item=transport) => {"ansible_loop_var": "item", "changed": true, "cmd": "/usr/share/elasticsearch/bin/elasticsearch-keystore remove 'xpack.security.transport.ssl.keystore.secure_password'", "delta": "0:00:00.699546", "end": "2021-10-04 15:15:22.543436", "item": "transport", "msg": "non-zero return code", "rc": 78, "start": "2021-10-04 15:15:21.843890", "stderr": "\nERROR: Setting [xpack.security.transport.ssl.keystore.secure_password] does not exist in the keystore.", "stderr_lines": ["", "ERROR: Setting [xpack.security.transport.ssl.keystore.secure_password] does not exist in the keystore."], "stdout": "", "stdout_lines": []}
2021-10-04 15:15:22,572 p=109151 u=ubuntu n=ansible | ...ignoring
2021-10-04 15:15:22,578 p=109151 u=ubuntu n=ansible | TASK [elastic.elasticsearch : Remove truststore password] ******************************************
2021-10-04 15:15:23,488 p=109151 u=ubuntu n=ansible | failed: [es-1.XXXXXX.lab] (item=http) => {"ansible_loop_var": "item", "changed": true, "cmd": "/usr/share/elasticsearch/bin/elasticsearch-keystore remove 'xpack.security.http.ssl.truststore.secure_password'", "delta": "0:00:00.680539", "end": "2021-10-04 15:15:23.458126", "item": "http", "msg": "non-zero return code", "rc": 78, "start": "2021-10-04 15:15:22.777587", "stderr": "\nERROR: Setting [xpack.security.http.ssl.truststore.secure_password] does not exist in the keystore.", "stderr_lines": ["", "ERROR: Setting [xpack.security.http.ssl.truststore.secure_password] does not exist in the keystore."], "stdout": "", "stdout_lines": []}
2021-10-04 15:15:24,364 p=109151 u=ubuntu n=ansible | failed: [es-1.XXXXXX.lab] (item=transport) => {"ansible_loop_var": "item", "changed": true, "cmd": "/usr/share/elasticsearch/bin/elasticsearch-keystore remove 'xpack.security.transport.ssl.truststore.secure_password'", "delta": "0:00:00.692642", "end": "2021-10-04 15:15:24.336956", "item": "transport", "msg": "non-zero return code", "rc": 78, "start": "2021-10-04 15:15:23.644314", "stderr": "\nERROR: Setting [xpack.security.transport.ssl.truststore.secure_password] does not exist in the keystore.", "stderr_lines": ["", "ERROR: Setting [xpack.security.transport.ssl.truststore.secure_password] does not exist in the keystore."], "stdout": "", "stdout_lines": []}
2021-10-04 15:15:24,365 p=109151 u=ubuntu n=ansible | ...ignoring
2021-10-04 15:15:24,371 p=109151 u=ubuntu n=ansible | TASK [elastic.elasticsearch : Set key password] ****************************************************
2021-10-04 15:15:24,411 p=109151 u=ubuntu n=ansible | skipping: [es-1.XXXXXX.lab] => (item=None)  => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2021-10-04 15:15:24,414 p=109151 u=ubuntu n=ansible | skipping: [es-1.XXXXXX.lab] => (item=None)  => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2021-10-04 15:15:24,417 p=109151 u=ubuntu n=ansible | skipping: [es-1.XXXXXX.lab] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2021-10-04 15:15:24,422 p=109151 u=ubuntu n=ansible | TASK [elastic.elasticsearch : Remove key password] *************************************************
2021-10-04 15:15:24,463 p=109151 u=ubuntu n=ansible | skipping: [es-1.XXXXXX.lab] => (item=http)  => {"ansible_loop_var": "item", "changed": false, "item": "http", "skip_reason": "Conditional result was False"}
2021-10-04 15:15:24,465 p=109151 u=ubuntu n=ansible | skipping: [es-1.XXXXXX.lab] => (item=transport)  => {"ansible_loop_var": "item", "changed": false, "item": "transport", "skip_reason": "Conditional result was False"}
2021-10-04 15:15:24,467 p=109151 u=ubuntu n=ansible | RUNNING HANDLER [elastic.elasticsearch : restart elasticsearch] ************************************
2021-10-04 15:15:37,369 p=109151 u=ubuntu n=ansible | fatal: [es-1.XXXXXX.lab]: FAILED! => {"changed": false, "msg": "Unable to start service elasticsearch: Job for elasticsearch.service failed because the control process exited with error code.\nSee \"systemctl status elasticsearch.service\" and \"journalctl -xe\" for details.\n"}
2021-10-04 15:15:37,371 p=109151 u=ubuntu n=ansible | NO MORE HOSTS LEFT *********************************************************************************
2021-10-04 15:15:37,373 p=109151 u=ubuntu n=ansible | PLAY RECAP *****************************************************************************************
2021-10-04 15:15:37,374 p=109151 u=ubuntu n=ansible | es-1.XXXXXX.lab               : ok=34   changed=4    unreachable=0    failed=1    skipped=91   rescued=0    ignored=2 

ES Logs if relevant:

Tail of /var/log/elasticsearch/test-cluster.log:

[2021-10-04T15:15:33,537][INFO ][o.e.e.NodeEnvironment    ] [es-1.XXXXXX.lab] using [1] data paths, mounts [[/ (/dev/mapper/ubuntu--vg-ubuntu--lv)]], net usable_space [106.8gb], net total_space [124.9gb], types [ext4]
[2021-10-04T15:15:33,537][INFO ][o.e.e.NodeEnvironment    ] [es-1.XXXXXX.lab] heap size [16gb], compressed ordinary object pointers [true]
[2021-10-04T15:15:33,552][INFO ][o.e.n.Node               ] [es-1.XXXXXX.lab] node name [es-1.XXXXXX.lab], node ID [4r4cFWfxQO6xlE7CvGULGg], cluster name [test-cluster], roles [master, remote_cluster_client, ml, ingest]
[2021-10-04T15:15:36,781][ERROR][o.e.b.Bootstrap          ] [es-1.XXXXXX.lab] Exception
org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl]
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
    at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
    at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.0.jar:7.15.0]
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) [elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) [elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) [elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) [elasticsearch-cli-7.15.0.jar:7.15.0]
    at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) [elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.15.0.jar:7.15.0]
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL KeyManager
    at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:87) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:440) ~[?:?]
    at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
    ... 26 more
Caused by: java.io.IOException: toDerInputStream rejects tag type 80
    at sun.security.util.DerValue.toDerInputStream(DerValue.java:1081) ~[?:?]
    at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1944) ~[?:?]
    at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:220) ~[?:?]
    at java.security.KeyStore.load(KeyStore.java:1472) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:98) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:77) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:440) ~[?:?]
    at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
    ... 26 more
[2021-10-04T15:15:36,786][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-1.XXXXXX.lab] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.security.http.ssl]]; nested: ElasticsearchException[failed to initialize SSL KeyManager]; nested: IOException[toDerInputStream rejects tag type 80];
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:171) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114) ~[elasticsearch-cli-7.15.0.jar:7.15.0]
    at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.15.0.jar:7.15.0]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl]
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:530) ~[?:?]
    at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
    at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.0.jar:7.15.0]
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-7.15.0.jar:7.15.0]
    ... 6 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL KeyManager
    at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:87) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:440) ~[?:?]
    at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
    at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
    at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.0.jar:7.15.0]
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-7.15.0.jar:7.15.0]
    ... 6 more
Caused by: java.io.IOException: toDerInputStream rejects tag type 80
    at sun.security.util.DerValue.toDerInputStream(DerValue.java:1081) ~[?:?]
    at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1944) ~[?:?]
    at sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:220) ~[?:?]
    at java.security.KeyStore.load(KeyStore.java:1472) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.TrustConfig.getStore(TrustConfig.java:98) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.StoreKeyConfig.createKeyManager(StoreKeyConfig.java:77) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:440) ~[?:?]
    at java.util.HashMap.computeIfAbsent(HashMap.java:1224) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:528) ~[?:?]
    at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
    at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:526) ~[?:?]
    at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:144) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:459) ~[?:?]
    at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:301) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$18(Node.java:622) ~[elasticsearch-7.15.0.jar:7.15.0]
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273) ~[?:?]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
    at org.elasticsearch.node.Node.<init>(Node.java:626) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.node.Node.<init>(Node.java:288) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399) ~[elasticsearch-7.15.0.jar:7.15.0]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167) ~[elasticsearch-7.15.0.jar:7.15.0]
    ... 6 more

Output of systemctl status elasticsearch on the managed node:

● elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/elasticsearch.service.d
             └─override.conf
     Active: failed (Result: exit-code) since Mon 2021-10-04 15:15:37 UTC; 12min ago
       Docs: https://www.elastic.co
    Process: 128266 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 128266 (code=exited, status=1/FAILURE)

Oct 04 15:15:36 es-1.XXXXXX.lab systemd-entrypoint[128266]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:219)
Oct 04 15:15:36 es-1.XXXXXX.lab systemd-entrypoint[128266]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:219)
Oct 04 15:15:36 es-1.XXXXXX.lab systemd-entrypoint[128266]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:399)
Oct 04 15:15:36 es-1.XXXXXX.lab systemd-entrypoint[128266]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167)
Oct 04 15:15:36 es-1.XXXXXX.lab systemd-entrypoint[128266]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158)
Oct 04 15:15:36 es-1.XXXXXX.lab systemd-entrypoint[128266]:         <<<truncated>>>
Oct 04 15:15:36 es-1.XXXXXX.lab systemd-entrypoint[128266]: For complete error details, refer to the log at /var/log/elasticsearch/test-cluster.log
Oct 04 15:15:37 es-1.XXXXXX.lab systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Oct 04 15:15:37 es-1.XXXXXX.lab systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Oct 04 15:15:37 es-1.XXXXXX.lab systemd[1]: Failed to start Elasticsearch.

[brsolomon-deloitte]

Regenerated the ca and cert with a password for each, added es_ssl_keystore_password and es_ssl_truststore_password to es-vars.yaml - same failure.

[bndabbs]

It looks like you are passing the --multiple flag to elasticsearch-certutil, which outputs a zip archive of all the requested certs.

https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html#certutil-cert

If you specify the --keep-ca-key, --multiple or --in parameters, the command produces a zip file containing the generated certificates and keys.

I think you just need to extract the archive and move the contents to the expected locations to remediate this issue.

[brsolomon-deloitte]

@bndabbs thanks for responding, I confirmed this is the case. The resulting my-keystore.p12 is a ZIP archive of individual p12s that, when unzip'd, produces es-{1..3}/*.p12 for each node.

I'll be re-deploying a new cluster today pointing to the correct keystore for each node, and will close this issue if that resolves the exception.

[brsolomon-deloitte]

Confirmed this was the case. By passing an individual es_ssl_keystore to each host block the cluster now comes up green and is accessible via https (with a --cacert needed).