Unable to obtain auth details from logs when detaching (in terminal or ansible)

[millerthegorilla]

Elasticsearch Version

8.5.1

Installed Plugins

n/a

Java Version

docker.io/library/ container

OS Version

6.0.8-300.fc37.x86_64

Problem Description

I am unable to find any way of obtaining the output of the elasticsearch command that is run in docker_entrypoint.sh, when running the command in detached mode using podman. This occurs even when I send a tty via the -t switch to the podman run command. It also occurs when I fork the command or use a bash subprocess. In all cases, the following error is generated in the logs:

Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the bin/elasticsearch-reset-password tool to set the password for the elastic user."

This means that I am unable to use ansible or the shell to automatically start the container as part of a test suite.

I have opened an issue at podman-ansible, and I have asked a question on the elasticsearch forum here and I have opened a stack overflow regarding obtaining output here

Steps to Reproduce

in the shell:

podman pull docker.io/library/elasticsearch:8.5.1
podman run -dt --name "elastic_cont" --env "discovery.type=single-node" --env "ES_JAVA_OPTS=-Xms512m -Xmx512m" --pod django_forum_test_pod docker.io/library/elasticsearch:8.5.1
podman logs elastic_cont

ansible playbook

  - name: Create Pod
    containers.podman.podman_pod:
      name: django_forum_test_pod
      publish: 8000:8000
- name: Create ElasticSearch Container
   # shell: echo $( podman run -t --name "elastic_cont" --env "discovery.type=single-node" --env "ES_JAVA_OPTS=-Xms512m -Xmx512m" --pod django_forum_test_pod docker.io/library/elasticsearch:8.5.1 2>&1) &    # containers.podman.podman_container:
    containers.podman.podman_container:
      name: elastic_cont
      image: docker.io/library/elasticsearch:8.5.1 
      state: started
      recreate: true
      detach: true
      env:
        'discovery.type': 'single-node'
        'ES_JAVA_OPTS': '-Xms512m -Xmx512m'
      pod: django_forum_test_pod
      log_level: debug
      tty: true
    register: elastic_output
  - debug: var=elastic_output
  - name: output image info
    shell: echo {{ elastic_output }}

Logs (if relevant)

Even when using the tty parameter of the ansible-podman-container, the logs report:

Auto-configuration will not generate a password for the elastic built-in superuser, as we cannot determine if there is a terminal attached to the elasticsearch process. You can use the bin/elasticsearch-reset-password tool to set the password for the elastic user."

There is no elastic user created, and when I exec into the container, the bin/elasticsearch-reset-password tool fails with:

ROR: Failed to reset password for the [elasticsearch] user

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[millerthegorilla]

I have opened an issue with podman, in case it is an issue with the way podman works, at

https://github.com/containers/ansible-podman-collections/issues/523

[millerthegorilla]

Podman have closed the issue, stating that it is an issue with elasticsearch detaching. https://github.com/containers/ansible-podman-collections/issues/523#issuecomment-1329672305

[millerthegorilla]

In the meantime, I have had to use a bash command in a script that also runs the rest of my ansible-playbook. In order to get the podman run command to bring up an elasticsearch container and fork and to make sure that elasticsearch detects a tty (and thus generate a password) I have had to put the podman run command in a separate process and fork it:

( podman run .....  ) &

I then use the following in the bash script to obtain the password:

while [ -z "${ELASTIC_PASSWORD}" ];
do
    ELASTIC_PASSWORD=$(podman logs elastic_cont | grep reset-password -A1 \
                       |sed -n 2p | tr -d [:space:] \
                       | sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2};?)?)?[mGK]//g")
    sleep 5s
    if [[ $(podman logs elastic_cont | grep "will not generate") ]]; then
        echo "password not generated";
        exit 1
    fi
done

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[tvernum]

I think this is approaching the problem from the wrong angle.

The password that is written to stdout is intended purely for interactive setup - it is intention that that password isn't provided under automated setups.

If you want to automate cluster setup then we recommend that you generate a password as part of your automation and provide it to Elasticsearch. The options there are:

1. Set the bootstrap password
2. Configure your cluster with a file realm that includes the users you want to use in your testsuite (this is what elasticsearch's test suite does)
3. Run elastcisearch-reset-password (which should work in a container, and we can try and debug why it wasn't working for you).