Non-Graceful Cluster Rollout on `Version` Change on ECK 2.10

[SiorMeir]

Non-Graceful Cluster Rollout on Version Change on ECK 2.10

What did you do? Upgraded the version of Elasticsearch (& Kibana) from 8.12.2 to 8.14.1

What did you expect to see? Graceful rollout of cluster, where nodes are replaced one at a time.

What did you see instead? Under which circumstances? All nodes went down at the same time, Operator changed status to UNKNOWN, resulting in downtime.

However, Manually changing the image of the different node types, adding environment variables or settings caused a graceful rollout of the cluster.

We have ruled out the possibility of non-HA cluster behavior, as the cluster is around 30 nodes. In Addition, the cluster has designated master (around 3) to create orchestrating redundancy.

Environment

• ECK version: 2.10

• Kubernetes information:

  • On premise : NO
  • Cloud: EKS, version 1.24.17-eks-3af4770
  • Kubernetes distribution: EKS

• Logs:

  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-data-srv-2"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-data-srv-1"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-data-srv-0"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-data-wrk-2"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-data-wrk-1"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-data-wrk-0"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-master-0"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-srv-coord-2"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-srv-coord-1"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-srv-coord-0"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-wrk-coord-2"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-wrk-coord-1"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Node shutdown complete, can start node deletion","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","type":"RESTART","node":"elasticsearch-recs-es-wrk-coord-0"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.311Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-data-srv-2","pod_uid":"066d7ec8-5f54-4814-9e64-50105816e1d6"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.322Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-data-srv-1","pod_uid":"1f075407-0548-4b6d-82ee-b63a0396050f"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.333Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-data-srv-0","pod_uid":"7fc38ad3-423c-43bb-a01b-c89d5ee2ffd3"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.348Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-data-wrk-2","pod_uid":"c1f32fb1-63b9-4442-9dc7-b274076f9f68"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.367Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-data-wrk-1","pod_uid":"c9c719ea-57f4-4243-9e38-8187ea1da56b"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.383Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-data-wrk-0","pod_uid":"70012d1b-a7aa-4bd6-8ce2-08ea9bdfb9d1"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.402Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-master-0","pod_uid":"4fa2d1e0-0342-481b-b021-d3fe7ae98497"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.419Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-srv-coord-2","pod_uid":"5a4c357b-6625-408f-9a39-97af5589da41"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.433Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-srv-coord-1","pod_uid":"d84f2727-4ac3-4152-ba1c-84bcce4c9750"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.449Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-srv-coord-0","pod_uid":"1c131e6d-8b94-4bb6-893e-2f6bc2aae5e4"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.468Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-wrk-coord-2","pod_uid":"ff421fdc-5f48-4bd4-aa42-c69589f8fab9"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.482Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-wrk-coord-1","pod_uid":"e80d528d-3362-46d4-87d0-e60bdae09110"}
  {"log.level":"info","@timestamp":"2024-07-29T12:29:34.500Z","log.logger":"elasticsearch-controller","message":"Deleting Pod for full cluster upgrade","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"822","namespace":"elasticsearch-recs","es_name":"elasticsearch-recs","es_name":"elasticsearch-recs","namespace":"elasticsearch-recs","pod_name":"elasticsearch-recs-es-wrk-coord-0","pod_uid":"a2a7501c-ac17-4fec-8067-03d69ebc354d"}

[SiorMeir]

@pebrc As you requested, I re-opened the issue as the cluster is more than 3 nodes.

This behavior was observed on multiple clusters & multiple k8s clusters (representing dev, staging etc..)

[pebrc]

You opened a new issue instead of re-opening the existing one.