[BUG] OpenSearchISMPolicy runtime error during cluster install

[f3l1x98]

What is the bug?

I tried to install an OpenSearch cluster with the following OpenSearchISMPolicy but the installation failed with an error.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchISMPolicy
metadata:
   name: default-ism-policy
spec:
  opensearchCluster:
    name: {{ .Values.opensearch.clusterName }}
  description: Default ISM policy for indices 
  ismTemplate:
    priority: 1
    indexPatterns:
      {{- toYaml .Values.opensearch.ism.defaultPolicy.indexPatterns | nindent 6 }}
  defaultState: hot
  states:
    - name: hot
      actions:
        - replicaCount:
            numberOfReplicas: {{ .Values.opensearch.ism.defaultPolicy.numberOfHotReplicas }}
      transitions:
        - stateName: delete
          conditions:
            minIndexAge: {{ .Values.opensearch.ism.defaultPolicy.deleteAgeCondition }}
    - name: delete
      actions:
        - delete: {}

{"level":"info","ts":"2024-06-03T07:42:17.025Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":{"name":"default-ism-policy","namespace":"monitoring"},"namespace":"monitoring","name":"default-ism-policy","reconcileID":"cbecd117-c438-4d0c-837b-7f2149d6eff5"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x16fbcfe]

goroutine 376 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:115 +0x1e5
panic({0x192a120?, 0x2c41390?})
    /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/Opster/opensearch-k8s-operator/opensearch-operator/opensearch-gateway/services.(*OsClusterClient).GetISMConfig(0x0, {0x1e0c7d8, 0xc000b41d70}, {0xc000cc25a0?, 0xc000041008?})
    /workspace/opensearch-gateway/services/os_client.go:314 +0x3e
github.com/Opster/opensearch-k8s-operator/opensearch-operator/opensearch-gateway/services.PolicyExists({0x1e0c7d8?, 0xc000b41d70?}, 0xc000d9e700?, {0xc000cc25a0?, 0x7?})
    /workspace/opensearch-gateway/services/os_ism_service.go:31 +0x55
github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*IsmPolicyReconciler).Reconcile(0xc000dabd00)
    /workspace/pkg/reconcilers/ismpolicy.go:159 +0x84b
github.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpensearchISMPolicyReconciler).Reconcile(0xc0004659a0, {0x1e0c7d8, 0xc000b41d70}, {{{0xc00079eaa0, 0xa}, {0xc00014c960, 0x12}}})
    /workspace/controllers/opensearchism_controller.go:53 +0x385
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1e0c7d8?, {0x1e0c7d8?, 0xc000b41d70?}, {{{0xc00079eaa0?, 0x18550a0?}, {0xc00014c960?, 0x10?}}})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0002097c0, {0x1e0c810, 0xc0004654a0}, {0x19a7860, 0xc0000ae020})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314 +0x345
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0002097c0, {0x1e0c810, 0xc0004654a0})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 107
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:222 +0x50c

It works if I remove the ISM Policy from the initial install and update the chart later with the policy.

How can one reproduce the bug?

1. Install operator version 2.6.0 (or 2.5.1)
2. Install new OpenSearch cluster chart containing an OpenSearchISMPolicy

What is the expected behavior?

Installation succeeds and ISM Policy is created

[prudhvigodithi]

[Triage] There are few fixes went in for ISM https://github.com/opensearch-project/opensearch-k8s-operator/pull/788, @cthtrifork can you please confirm of this issue reported by @f3l1x98 would be solved? Thank you