Cannot list indices on AWS Elasticsearch

[curry684]

Running on Docker with latest (3.3.0), I'm getting all information, including clusters, nodes, diagnostics and everything, but no indices.

JS console notes:

GET http://localhost:5000/api/indices/785857882783:censored/_summary 500 (INTERNAL SERVER ERROR)
cluster-details.controller.js:24 ---- err:  {data: {…}, status: 500, headers: ƒ, config: {…}, statusText: "INTERNAL SERVER ERROR", …}
angular.js:14791 Possibly unhandled rejection: {"data":{"message":"Oops! Something bad happened.","status_code":500},"status":500,"config":{"method":"GET","transformRequest":[null],"transformResponse":[null],"jsonpCallbackParam":"callback","url":"api/indices/785857882783:censored/_summary","timeout":{},"headers":{"Accept":"application/json, text/plain, */*"}},"statusText":"INTERNAL SERVER ERROR","xhrStatus":"complete"}

Regrettably, no further information. The Docker logs only contain startup information (they should likely forward error logs). Inside the container I cannot find any relevant logfiles.

[royrusso]

In the container, the logs should be under src/application.log (http://docs.elastichq.org/installation.html#logging)

Also, what version of ES and HQ are you using, and do all of the other functions work fine?

[curry684]

ES is standard Amazon AWS Elasticsearch 5.1.1, Docker image for HQ is latest which is 3.3.0.

HQ correctly detects ES cluster to be on 5.1.1.

In the application log I get:

2018-04-16 20:43:42,316      ERROR   elastichq   exceptions._request_wrapper:37      Oops! Something bad happened.
Traceback (most recent call last):
  File "/src/elastichq/common/exceptions.py", line 29, in _request_wrapper
    return functor(*args, **kwargs)
  File "/src/elastichq/api/indices.py", line 28, in get
    response = IndicesService().get_indices_summary(cluster_name, index_names)
  File "/src/elastichq/service/IndicesService.py", line 115, in get_indices_summary
    'number_of_shards': int(jmespath.search("settings.index.number_of_shards", index_state)),
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Like I said, all other functions work fine, it's just fetching indices.

The call is to GET https://<instance>.es.amazonaws.com:443/_cat/indices?format=json [status:200 request:0.116s], and indeed it does not have a number_of_shards property in its response.

Same error btw on every page that lists indices anywhere, like for example the Alias detail page.

[royrusso]

The call to ES under IndicesService is the summary call: http://localhost:5000/api/indices/<CLUSTER_NAME>/_summary which in turn is calling your server with http://localhost:8200/_cluster/state/metadata

The response from the above metadata call should look like this: metadata": { "cluster_uuid": "xxqmR397RkG-w_2wY8Vvag", "templates": {}, "indices": { "cars": { "state": "close", "settings": { "index": { "creation_date": "1522093848913", "number_of_shards": "5", "number_of_replicas": "1",

Can you show me what that outputs. Running on 5.x seems to work, so this may be an issue with the AWS instances. I can add a hot patch, but am concerned also with what else is missing in that call.

[royrusso]

The fix was marked as closed automatically, but I'll reopen and wait for your feedback if the latest hot patch / build solves the problem. I'm merely zeroing out the shard numbers, so it's not an ideal longterm solution.

[curry684]

There's no settings key in the index details on that call:

     "test-for-elastichq": {
        "state": "open",
        "mappings": {

        },
        "aliases": Array[0][

        ],
        "primary_terms": {
          "0": 1,
          "1": 1,
          "2": 1,
          "3": 1,
          "4": 1
        },
        "in_sync_allocations": {
          "0": Array[2][
            "Ry_itGl7SjK-8uRJ_zbJNA",
            "rph2QOhCSvGm4FQX58Cc5A"
          ],
          "1": Array[2][
            "r_73Waj4QFaBvbz7isb19w",
            "6G1TOTtOTqqSl5bQ1Quqiw"
          ],
          "2": Array[2][
            "sdh5-TPqRgSGcG1diSMn2g",
            "eREai22pT5eWlJTLekx4Ng"
          ],
          "3": Array[2][
            "kGgyQq5SS7SmvqFs8HhcxA",
            "PG9F9S0nREOV2uTNXPQkQg"
          ],
          "4": Array[2][
            "1E7AkGFPQry1PYNl8h4cDw",
            "wIR2sUCsT5WC3frIu4NnPA"
          ]
        }
      },

[curry684]

I can confirm docker image elastichq/elasticsearch-hq:develop is working fine, no errors anywhere in the entire app 👍

[royrusso]

Note that the push should've made it to the master docker image as well. I wouldn't update from the develop branch going forward unless you're feeling really brave.

[curry684]

In that case your documentation at http://docs.elastichq.org/installation.html?highlight=docker#docker-images has an issue since it implies latest is the current stable 3.3.0 😉

[royrusso]

That's fair. I patched both develop and master. Master is currently 3.3.0. I also renamed the docker image, as it was labeled "latest", when it should be "master".

[curry684]

Best practice for Docker is to alias latest to the "recommend for general audience" image, which would be 3.3.0 in this case, as it's the one that gets downloaded when no tag is specified.