search

elastic / cloud-on-k8s

Elastic Cloud on Kubernetes
Other
2.58k stars 702 forks source link

ECK Operator 2.6.1 failing to upgrade ES from 7.17.10 to 8.9.1 due to invalid setting which is actually valid #7173

Open gustavosci opened 1 year ago

gustavosci commented 1 year ago

Bug Report

What did you do? Tried to upgrade an ES cluster from v7.17.10 to v8.9.1 using eck-operator v2.6.1 with setting thread_pool.write.size defined in elasticsearch.yaml.

What did you expect to see? Successful ES upgrade and write thread pool size with the correct size.

What did you see instead? Under which circumstances? Error to upgrade the cluster (please see logs). The operator does not allow proceeding with the upgrade with the setting thread_pool.write.size defined in the elasticsearch.yaml.

Environment

barkbay commented 1 year ago

How do know that the problem comes from thread_pool.write.size? I'm asking because other issues are known to generate the same error: https://github.com/elastic/elasticsearch/pull/96138 I want to double check that it is a new one.

gustavosci commented 1 year ago

Hi @barkbay Once we remove this setting from the yaml, the cluster is created without issues. That's why we suspect of this setting. Please let me know if you want more information about our configs.

pebrc commented 1 year ago

I tried to reproduce this error with the following manifest on ECK 2.9 and ES upgraded without problems:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.17.10 # then changed to 8.9.1
  nodeSets:
  - name: default
    config:
      thread_pool.write.size: 2
      node.roles: ["master", "data", "ingest", "ml"]
      # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            limits:
              memory: 4Gi
              cpu: 1
    count: 3
gustavosci commented 1 year ago

@pebrc here is our complete config:

{
  "action.auto_create_index": false,
  "action.destructive_requires_name": true,
  "cluster.routing.allocation.awareness.attributes": "zone",
  "indices.queries.cache.size": 0,
  "indices.query.bool.max_clause_count": 2048,
  "network.tcp.keep_alive": true,
  "network.tcp.keep_idle": 300,
  "node.attr.zone": "us-east-1c",
  "node.roles": [
    "data",
    "ingest"
  ],
  "thread_pool.write.queue_size": 200,
  "thread_pool.write.size": 6,
  "xpack.graph.enabled": false,
  "xpack.ml.enabled": false,
  "xpack.monitoring.collection.enabled": true,
  "xpack.monitoring.exporters": {
    "data-3": {
      "host": [
        "http://es-xpk-es-http:9200/"
      ],
      "type": "http"
    }
  },
  "xpack.security.authc": {
    "anonymous": {
      "authz_exception": false,
      "roles": "superuser",
      "username": "anonymous"
    }
  },
  "xpack.security.authc.realms": {
    "native": {
      "native1": {
        "order": 1
      }
    }
  },
  "xpack.security.enabled": false,
  "xpack.watcher.enabled": false
}

Removing "thread_pool.write.size" was enough for eck-operator 2.6.1 to complete the upgrade successfully.

gustavosci commented 11 months ago

@pebrc @barkbay we've found the reason for the issue to happen.

Master nodes config:

          resources:
            limits:
              cpu: "4"
              memory: 16Gi
            requests:
              cpu: "4"
              memory: 16Gi

Data nodes config:

          resources:
            limits:
              cpu: "13"
              memory: 57Gi
            requests:
              cpu: "13"
              memory: 57Gi

The issue only happens when we try to apply thread_pool.write.size=6 for our data nodes. As per ES documentation, the write thread pool is fixed with a size of # of allocated processors. The maximum size for this pool is 1 + # of allocated processors. It means that, in our case, as we are setting tp size to 6 only for data nodes, it should work, since data nodes have 13 CPU cores allocated.

However, it looks like that eck-operator is considering the nodes with less CPU allocated in the cluster, regardless of the node type, to make this validation, which is not correct. We did some tests and, for example, thread_pool.write.size=5 already works. Increasing the CPUs for master nodes from 4 to 5 also makes thread_pool.write.size=6 work.

So, to summarize, it seems that the node type should be taken into consideration for this validation. Can you please take a look at that?

barkbay commented 11 months ago

However, it looks like that eck-operator is considering the nodes with less CPU allocated in the cluster [...]

The configuration is rejected by Elasticsearch. The operator submits the configuration, but the validation is ultimately done by Elasticsearch. I'm afraid an Elasticsearch issue is likely to be needed to make some progress.

gustavosci commented 11 months ago

@barkbay thanks, I've just created an issue for ES: https://github.com/elastic/elasticsearch/issues/101206.