ERROR: failed to wait for yellow status

[scheiblr]

ZomboDB version: v3000.0.0-beta1 Postgres version: 13.2 Elasticsearch version: 7.12

Problem Description: I expected the new version to treat as the old one. So basically I just updated a running software setup (es, pg & zdb). Zdb creates an index on the side of ES which remains in status red and as far as I see, does not contain any data.

Error Message (if any):

│2021-04-29 11:53:12.883 UTC [83] ERROR:  failed to wait for yellow status: ElasticsearchError(Some(408), "{\n  \"cluster_name\": \"es-docker-cluster\",\n  \"status\": \"red\",\n  \"timed_out\": true,\n  \"number_of_nodes\":     │
│3,\n  \"number_of_data_nodes\": 3,\n  \"active_primary_shards\": 0,\n  \"active_shards\": 0,\n  \"relocating_shards\": 0,\n  \"initializing_shards\": 0,\n  \"unassigned_shards\": 5,\n  \"delayed_unassigned_shards\": 0,\n        │
│\"number_of_pending_tasks\": 0,\n  \"number_of_in_flight_fetch\": 0,\n  \"task_max_waiting_in_queue_millis\": 0,\n  \"active_shards_percent_as_number\": 0.0\n}")                                                                   │
│2021-04-29 11:53:12.883 UTC [83] CONTEXT:  src/elasticsearch/create_index.rs:37:14                                                                                                                                                  │
│2021-04-29 11:53:12.883 UTC [83] STATEMENT:  CREATE INDEX idx_es_search                                                                                                                                                             │
│      ON data.search_container                                                                                                                                                                                                      │
│      USING zombodb ((                                                                                                                                                                                                              │
│        ROW(                                                                                                                                                                                                                        │
│          data.search_container.id,                                                                                                                                                                                                 │
│                                                                                                                                                                                                                                    │
│          data.search_container.search_text,                                                                                                                                                                                        │
│          data.search_container.lang,                                                                                                                                                                                               │
│          data.search_container.entry_type) :: idx_es_search_type                                                                                                                                                                   │
│        )) WITH (url='http://es01:9200/');                                                                                                                                                                                          │
│psql:/docker-entrypoint-initdb.d/data/search.sql:82: ERROR:  failed to wait for yellow status: ElasticsearchError(Some(408), "{\n  \"cluster_name\": \"es-docker-cluster\",\n  \"status\": \"red\",\n  \"timed_out\": true,\n  │
│ \"number_of_nodes\": 3,\n  \"number_of_data_nodes\": 3,\n  \"active_primary_shards\": 0,\n  \"active_shards\": 0,\n  \"relocating_shards\": 0,\n  \"initializing_shards\": 0,\n  \"unassigned_shards\": 5,\n  \"delayed_unassigned │
│_shards\": 0,\n  \"number_of_pending_tasks\": 0,\n  \"number_of_in_flight_fetch\": 0,\n  \"task_max_waiting_in_queue_millis\": 0,\n  \"active_shards_percent_as_number\": 0.0\n}")                                                  │
│CONTEXT:  src/elasticsearch/create_index.rs:37:14

Index Definition:

CREATE TYPE idx_es_search_type AS (
  id bigint,
  search_text text,
  lang text,
  entry_type text
);

--
-- worked with former zdb version
--
DROP INDEX IF EXISTS idx_es_search;
CREATE INDEX idx_es_search 
  ON data.search_container 
  USING zombodb ((
    ROW(
      data.search_container.id,
      data.search_container.search_text,
      data.search_container.lang,
      data.search_container.entry_type) :: idx_es_search_type
    )) WITH (url='http://es01:9200/');

The ES cluster was setup according to the ES documentation.

[eeeebbbbrrrr]

Thanks for the report. Off the top of my head I don't know what might cause this. I'll do some investigations with my local cluster over the weekend.

[eeeebbbbrrrr]

I'm unable to re-create this, but it looks like you're getting a 408 back, which is a "request timeout", so I've added a retry for this, up to 10 times.

I guess what's happening is it's taking your cluster longer to turn (at least) yellow (which means all primary shards are online) than whatever the default timeout for waiting is.

I'll a fix pushed in a bit and then I think I'll be able to make a final release!

[eeeebbbbrrrr]

slight change of plans. I'm just going to bump the request timeout for the cluster health request from 30s to 5m. That's the same thing as retrying up to 10 times.

[eeeebbbbrrrr]

This will be released in v3000.0.0 -- which is happening now.