Pelias doesn't use ES auth credentials

[mabilinab]

Describe the bug When ES cluster needs authentication credentials, Schema doesn't seem to be using the admin:password credentials of the ES cluster, despite having an "auth" section in pelias.json. Here's the error obtained when running ./bin/create_index

2024-08-21T10:16:02.785Z - debug: [schema-synonyms] [line] way,wy
2024-08-21T10:16:02.785Z - debug: [schema-synonyms] [line] wl,well
2024-08-21T10:16:02.785Z - debug: [schema-synonyms] [line] wls,wells

--------------
 create index 
--------------

StatusCodeError: [security_exception] missing authentication credentials for REST request [/], with { header={ WWW-Authenticate={ 0="Basic realm=\"security\", charset=\"UTF-8\"" & 1="Bearer realm=\"security\"" & 2="ApiKey" } } }
    at respond (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/code/pelias/schema/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/code/pelias/schema/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/code/pelias/schema/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  status: 401,
  displayName: 'AuthenticationException',
  path: '/',
  query: {},
  body: undefined,
  statusCode: 401,
  response: '{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\\"security\\", charset=\\"UTF-8\\"","Bearer realm=\\"security\\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\\"security\\", charset=\\"UTF-8\\"","Bearer realm=\\"security\\"","ApiKey"]}},"status":401}',
  wwwAuthenticateDirective: 'Basic realm="security", charset="UTF-8", Bearer realm="security", ApiKey',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}
unsupported elasticsearch version. try: >=7.4.2

This means the schema pod is not sending requests to the ES cluster using auth credentials. I am using the "auth" section in the pelias.json to feed the auth credentials, but not sure if this section is taken into account by the code.

Steps to Reproduce

1. Install a ECK ES cluster (in k8) (it will automatically create a user 'elasticsearch' and password)

2. Create a pelias.json file that has the following section:

   apiVersion: v1
   kind: ConfigMap
   metadata:
   name: pelias-test-config
   data:
   pelias.json: |
   {
     "logger": {
       "level": "debug",
       "timestamp": true
     },
     "esclient": {
       "apiVersion": "7.5",
       "hosts": [
         {
           "host": "quickstart-es-http",
           "port": 9200,
           "protocol": "https"
         }
       ],
       "auth": {
         "username": "elastic",
         "password": "XXXX ES Password Here XXXX"
       }
     },
     "elasticsearch": {
       "settings": {
         "index": {
           "refresh_interval": "10s",
           "number_of_replicas": "1",
           "number_of_shards": "2"
         }
       }
     },

3. Create a schema pod mounting the pelias.json as configmap. Use a sleep command so that you can ssh into the pod and run the command manually to check the error (otherwise, you'll get crashloop)

4. SSH into the pod and run ./bin/create_index. You'll get the error stated above

Additional context If you try to manually send a request inside the pod to the ES cluster using the credentials, it works fine, which means that the pod can contact the ES.

pelias@schema:/code/pelias/schema$ curl -u "elastic:$PASSWORD" -k "https://quickstart-es-http:9200"
{
  "name" : "quickstart-es-default-0",
  "cluster_name" : "quickstart",
  "cluster_uuid" : "_edAcVGlQya2-k7kZ5aVHQ",
  "version" : {
    "number" : "8.15.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "1a77947f34deddb41af25e6f0ddb8e830159c179",
    "build_date" : "2024-08-05T10:05:34.233336849Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

And if you remove the auth part from the curl, you get exactly the same error as the one returned by "./bin/create_index":

pelias@schema:/code/pelias/schema$ curl  -k "https://quickstart-es-http:9200"
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

Another weird behavior How come the pelias returning this error at the end of the message if it's not able to connect to the ES cluster.

unsupported elasticsearch version. try: >=7.4.2

You normally only get the ES API version if you can establish a connection with the cluster. Also, the error doesn't make any sense since my ES cluster is running on API version "8.15.0"

[missinglink]

Possible dupe of https://github.com/pelias/api/issues/1668

[mabilinab]

You're right. Thank you for the fast reply! The syntax should be:

      "esclient": {
        "apiVersion": "7.5",
        "hosts": [
          {
            "host": "quickstart-es-http",
            "port": 9200,
            "protocol": "https",
            "auth": "USER:PASSWORD"
          }
        ]
      },

[mabilinab]

But don't you think that this behavior is indicator of error in the code logic?

Another weird behavior
How come the pelias returning this error at the end of the message if it's not able to connect to the ES cluster.

unsupported elasticsearch version. try: >=7.4.2

You normally only get the ES API version if you can establish a connection with the cluster. Also, the error doesn't make any sense since my ES cluster is running on API version "8.15.0"

[missinglink]

The elasticsearch client library we use is old and only supports up to v7, fortunately each version is backwards compatible one version so a v8 server will still work.

There is a ticket to upgrade the client library and (I believe) a PR but it's a fairly hairy change set so we've been holding off until it's strictly required by v9.

[missinglink]

Nice clear bug report BTW :+1: I'll close this issue since it seems resolved, if you feel the docs can be updated then we'd be happy to accept a PR.