[BUG] OPENSEARCH_INITIAL_ADMIN_PASSWORD doesn't work with opensearch:2.17.0 docker image

[Tlecomte13]

Describe the bug

In version 2.17 of the official OpenSearch Docker image, the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD is not functioning as expected. When launching the container, OpenSearch prompts for an admin password despite the variable being set in the docker-compose.yml file.

The admin password is correctly set in the container and can be verified within the environment variables.

  opensearch-node:
    image: opensearchproject/opensearch:2.17.0
    environment:
      - cluster.name=${CLUSTER_NAME}
      - node.name=${NODE_NAME}
      - discovery.seed_hosts=${NODE_NAME}
      - cluster.initial_master_nodes=${NODE_NAME}
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data:/usr/share/opensearch/data
    ports:
      - "9200:9200"
      - "9600:9600"
    networks:
      - opensearch-net

curl curl -k -u admin:<password> https://localhost:9200 Unauthorized

Using version 2.16 of the OpenSearch Docker image works perfectly fine, with the OPENSEARCH_INITIAL_ADMIN_PASSWORD variable being recognized correctly and allowing the container to start without issues.

curl -k -u admin:<password> https://localhost:9200
{
  "name" : "opensearch-node",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "dGE6WpovS7u-C8Y_qwhL1Q",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.16.0",
    "build_type" : "tar",
    "build_hash" : "f84a26e76807ea67a69822c37b1a1d89e7177d9b",
    "build_date" : "2024-08-06T20:32:34.547531562Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Related component

Build

To Reproduce

1. Use the official OpenSearch Docker image version 2.17.
2. Set the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD in the docker-compose.yml.
3. Start the container using docker-compose up.
4. try to curl https://localhost:9200

Expected behavior

The container should start successfully without prompting for an admin password. It should utilize the password provided through the OPENSEARCH_INITIAL_ADMIN_PASSWORD environment variable, allowing access to the OpenSearch instance without any authentication issues.

Additional Details

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: macos M3
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[hainenber]

hi @Tlecomte13, I think you missed out the right-side env var. It should be similar following diff here, no? 👀

- - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_ADMIN_PASSWORD}`
+ - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}`

[Tlecomte13]

Hi @hainenber, my environment variable in the .env file is indeed OPENSEARCH_ADMIN_PASSWORD, I haven’t changed anything except the image version, where I went from 2.17 to 2.16, and with version 2.16 it works perfectly.

I tried directly without using an env file, like this: OPENSEARCH_INITIAL_ADMIN_PASSWORD=Yd76.... Of course, taking into account the uppercase, lowercase, special characters, etc., as mentioned in the documentation.

[peterzhuamazon]

Hi @hainenber @Tlecomte13 ,

OPENSEARCH_INITIAL_ADMIN_PASSWORD wont take effect if you havent delete the volume. If you already have 2.16 started and set password to 123, then update to 2.17 with password 456 wont take effect. The password will still be 123.

Would you provide more log about the error to know exactly what is happening.

Thanks

[hainenber]

Oh im a bystander in this case 😄

I assume you are trying to resolve @Tlecomte13 case

[peterzhuamazon]

Also I just checked that build code did not make any related changes for months. I do see some changes related to the cert and user setup here directly in https://github.com/opensearch-project/security/compare/2.16...2.17.

cc: @cwperks @DarshitChanpura @derek-ho to take a look here.

We have tested the docker container in 2.17.0 here and seems it works fine: https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-validation/detail/distribution-validation/1250/pipeline/51

[Tlecomte13]

Hi @peterzhuamazon, @hainenber

So, after a night’s sleep, I wanted to try what @peterzhuamazon suggested, which was to upgrade from version 2.16 to 2.17 while keeping my volumes. It worked fine. I thought it must be something during the project’s initiation, so I deleted all the volumes using docker system prune -a to make sure I had nothing left.

And it works... even though the day before I did the same steps and only changed the image version.

The only thing I can recall is that before setting up OpenSearch, in the meantime, I updated my Docker version to 4.34.2. I don't think it’s related, but how can I explain how crazy this is driving me? I can’t understand why it didn’t work yesterday.

I can now upgrade to version 2.17 and stay up to date, but I don’t know what happened or how to reproduce it now.

Thanks anyway for your help, I'm going to close the ticket.