OpenSearch K8s Operator not restarting the bootstarp, coordinators, nodes and master pods after upgrade the version

rameshar16 commented 3 months ago

What is the bug?

OpenSearch K8s Operator not restarting the bootstarp, coordinators, nodes and master pods after upgrade the version.

How can one reproduce the bug?

Change the version from 2.4.0 to 2.6.0 and run the helm upgrade to deploy the new version.

What is the expected behavior?

OpenSearch K8s Operator should restart the bootstarp, coordinators, nodes and master pods after upgrade the version.

What is your host/environment?

AWS EKS cluster and deployed OpenSearch K8s Operator. Deployed OpenSearch cluster using helm chart.

Do you have any screenshots?

No

Do you have any additional context?

No

rameshar16 commented 3 months ago

`>helm diff upgrade opensearch-cr ./charts/opensearch-cluster/ --values ./charts/opensearch-cluster/values.yaml -n opensearch apiVersion: opensearch.opster.io/v1 kind: OpenSearchCluster metadata: name: opensearch-cr namespace: opensearch spec: general: **- version: 2.4.0

version: 2.6.0** httpPort: 9200 vendor: opensearch serviceName: opensearch-svc setVMMaxMapCount: true drainDataNodes: true dashboards: service: type: NodePort tolerations:
- effect: NoSchedule key: dedicated operator: Equal value: opensearch version: 2.3.0 enable: true replicas: 1 resources: requests: memory: 1Gi cpu: 500m limits: memory: 1Gi cpu: 500m tls: enable: true # Configure TLS secret: name: ssl-secret nodePools:
- component: masters tolerations:
  - effect: NoSchedule key: dedicated operator: Equal value: opensearch replicas: 3 pdb: # Add pdb configuration enable: true minAvailable: 3 diskSize: "30Gi" nodeSelector: eks.amazonaws.com/nodegroup: opensearch-cluster resources: requests: memory: 2Gi cpu: 500m limits: memory: 2Gi cpu: 500m roles:
  - cluster_manager persistence: pvc: storageClass: opensearch accessModes:
    - ReadWriteOnce
- component: nodes tolerations:
  - effect: NoSchedule key: dedicated operator: Equal value: opensearch replicas: 3 pdb: # Add pdb configuration enable: true minAvailable: 2 diskSize: "30Gi" nodeSelector: eks.amazonaws.com/nodegroup: opensearch-cluster resources: requests: memory: 2Gi cpu: 500m limits: memory: 2Gi cpu: 500m roles:
  - data persistence: pvc: storageClass: opensearch accessModes:
    - ReadWriteOnce
- component: coordinators tolerations:
  - effect: NoSchedule key: dedicated operator: Equal value: opensearch replicas: 3 pdb: # Add pdb configuration enable: true minAvailable: 2 diskSize: "30Gi" nodeSelector: eks.amazonaws.com/nodegroup: opensearch-cluster resources: requests: memory: 2Gi cpu: 500m limits: memory: 2Gi cpu: 500m roles:
  - ingest persistence: pvc: storageClass: opensearch accessModes:
    - ReadWriteOnce security: tls: transport: http:
      
      helm upgrade opensearch-cr ./charts/opensearch-cluster/ --values ./charts/opensearch-cluster/values.yaml -n opensearch Release "opensearch-cr" has been upgraded. Happy Helming! NAME: opensearch-cr LAST DEPLOYED: Sat Jun 22 11:45:33 2024 NAMESPACE: opensearch STATUS: deployed REVISION: 10 TEST SUITE: None`

rameshar16 commented 3 months ago

>k get po -n opensearch -w
NAME                                                      READY   STATUS    RESTARTS   AGE
opensearch-cr-coordinators-0                              1/1     Running   0          17h
opensearch-cr-coordinators-1                              1/1     Running   0          16h
opensearch-cr-coordinators-2                              1/1     Running   0          16h
opensearch-cr-dashboards-54d6ccb67c-nw9sp                 1/1     Running   0          17h
opensearch-cr-masters-0                                   1/1     Running   0          15m
opensearch-cr-masters-1                                   1/1     Running   0          16h
opensearch-cr-masters-2                                   1/1     Running   0          16h
opensearch-cr-nodes-0                                     1/1     Running   0          26m
opensearch-cr-nodes-1                                     1/1     Running   0          16h
opensearch-cr-nodes-2                                     1/1     Running   0          16h
opensearch-operator-controller-manager-68f76ffd94-knsw4   2/2     Running   0          13m

prudhvigodithi commented 3 months ago

[Triage] Adding @swoehrl-mw to please verify this, I assume we have seen similar issue in past and was fixed with PR https://github.com/opensearch-project/opensearch-k8s-operator/pull/789 right? Thank you

prudhvigodithi commented 3 months ago

Also @rameshar16 I assume you are using the latest version of the operator, can you please confirm ? Thank you

swoehrl-mw commented 2 months ago

@prudhvigodithi This does not look like the parallel recovery bug.

@rameshar16 Please check the OpenSearchCluster CR status (kubectl describe opensearchcluster opensearch-cr) what the operator is reporting. Also please verify if your opensearch cluster is healthy (green status). Because it looks like the first pod of each pool was restarted but the operator is not continuing. And that points towards the cluster not being green.

opensearch-project / opensearch-k8s-operator