Bootstrapping a new/additional instance with activated security fails

Elasticsearch version: 7.14.0

Role version: tag 7.14.0 / 0d22243e8e6477518b3266ff6aa7849c400a7e83

JVM version (java -version): openjdk version "11.0.12" 2021-07-20 OpenJDK Runtime Environment Temurin-11.0.12+7 (build 11.0.12+7) OpenJDK 64-Bit Server VM Temurin-11.0.12+7 (build 11.0.12+7, mixed mode)

OS version: Debian Buster 10.10 - Linux es-test-2 4.19.0-17-cloud-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/Linux

Description of the problem including expected versus actual behaviour:

A Elasticsearch cluster consisting of three nodes was created successfully. After successful configuration and cluster-forming, "security" was enabled. Valid certificates are in place, TLS is activated on transport. All is fine.

My expectation is, that I can add additional VM-instances to a ES-cluster, even if "security" is enabled. With "configured properly" I think of: configured in the same was as the other two machines of this cluster.

The expectation was not met when I tried this:

I decided to stop Elasticsearch on one instance and purge the Debian-packages including all other configuration and data from this machine. I then ran the playbook to reinstall/reconfigure Elasticsearch on the cluster.

The playbook fails.

If we needed to either grow the cluster horizontally or replace virtual-machines, we want to install new Elasticsearch-instances with enabled security.

Playbook: Please specify the full playbook used to reproduce this issue.

Elasticsearch variables:

es_heap_size: 1g

es_version: "7.14.0"

es_api_basic_auth_username: "elastic"
es_api_basic_auth_password: "{{ es_api_elastic_password }}"

es_enable_transport_ssl: true
es_enable_http_ssl: false

es_config:
  cluster.name: "es-team"
  cluster.initial_master_nodes: "{{ es_cluster_initial_master_nodes }}"
  http.port: "{{ es_http_port|int }}"
  transport.tcp.port: "{{ es_transport_tcp_port|int }}"
  discovery.zen.ping.unicast.hosts: "{{ es_cluster_discovery_unicast_hosts }}"
  discovery.zen.minimum_master_nodes: 2
  node.data: true
  node.master: true
  cluster.routing.allocation.awareness.attributes: zone_id
  bootstrap.memory_lock: True
  network.host: "0"
  indices.recovery.max_bytes_per_sec: "1000m"
  xpack.security.authc.realms.file.file1.order: 0
  xpack.security.authc.realms.native.native1.order: 1
  xpack.security.authc.anonymous:
      username: _es_anonymous_user
      roles:
        - healthcheck
        - user
        - kibana_admin
      authz_exception: true

Provide logs from Ansible:

TASK [elastic.elasticsearch : Check if bootstrap password is set] ************************************************************************************************************************************************************
fatal: [es-test-2]: FAILED! => {"changed": false, "cmd": "/usr/share/elasticsearch/bin/elasticsearch-keystore list", "msg": "[Errno 2] No such file or directory: b'/usr/share/elasticsearch/bin/elasticsearch-keystore': b'/usr/share/elasticsearch/bin/elasticsearch-keystore'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

ES Logs if relevant: not relevant

elastic / ansible-elasticsearch

Bootstrapping a new/additional instance with activated security fails #816