[BUG] ISM Policies Support can not handle "ism_template" (array)

[cthtrifork]

What is the bug?

A clear and concise description of the bug. Example ISM from OpenSearch Rest API:

{
    "id": "test-policy",
    "seqNo": 249493,
    "primaryTerm": 6,
    "policy": {
        "policy_id": "test-policy",
        "description": "hot delete workflow",
        "last_updated_time": 1708690376161,
        "schema_version": 19,
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "3d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "transformandstore*"
                ],
                "priority": 100,
                "last_updated_time": 1708690376161
            }
        ]
    }
}

Where you can see ism_template is an array.

The causes the following issue:

{"level":"error","ts":"2024-02-23T13:58:02.186Z","msg":"failed to get policy from Opensearch API","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io" ,"controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":\{"name":"test","namespace":"opensearch"}
,"namespace":"opensearch","name":"test","reconcileID":"59b5f7e6-2767-4c0d-a11d-555ff758956e","reconciler":"ismpolicy","error":"json: cannot unmarshal array into Go struct field ISMPolicy.policy.ism_template of type requests.ISMTemplate","stacktrace":"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*IsmPolicyReconciler).Reconcile\n\t/workspace/pkg/reconcilers/ismpolicy.go:200\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpensearchISMPolicyReconciler).Reconcile\n\t/workspace/controllers/opensearchism_controller.go:53\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

See code here, where it reties to get the ISM: https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/opensearch-operator/pkg/reconcilers/ismpolicy.go#L200

The fix should be applied here: https://github.com/opensearch-project/opensearch-k8s-operator/blob/c3d2a79f932b9a6e02e742318911e93b5d511b11/opensearch-operator/opensearch-gateway/requests/IsmPolicy.go#L18 (should be array)

and (possibly!) here: https://github.com/opensearch-project/opensearch-k8s-operator/blob/c3d2a79f932b9a6e02e742318911e93b5d511b11/opensearch-operator/config/crd/bases/opensearch.opster.io_opensearchismpolicies.yaml#L97 (should be array)

How can one reproduce the bug?

What is the expected behavior?

A clear and concise description of what you expected to happen.

What is your host/environment?

Running opensearch:2.11.1

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem. Original feature: https://github.com/opensearch-project/opensearch-k8s-operator/issues/435

[cthtrifork]

Surprisingly, applying the ISM actually works, so OpenSearch REST API can handle ism_template as an array or as a single object. The issue is only when unmarshalling in GO.

[cthtrifork]

FYI @IshaGirdhar

[cthtrifork]

As far as I can see, the "ism_template" has been returned as an array the past 3 years: https://github.com/opensearch-project/index-management/blame/042f72841ff85032beecc20c3f9338161af2ab42/src/main/kotlin/org/opensearch/indexmanagement/opensearchapi/OpenSearchExtensions.kt#L117 https://github.com/opensearch-project/documentation-website/blame/363abb5870ad76d007e8865f926d15cbc8c607e3/_im-plugin/ism/policies.md#L27 https://github.com/opensearch-project/index-management/blob/042f72841ff85032beecc20c3f9338161af2ab42/src/test/kotlin/org/opensearch/indexmanagement/indexstatemanagement/IndexStateManagementRestTestCase.kt#L1071

However, when creating a new ISM, it can accept both a single object or an array.

I can not find anything in release notes, so it is hard for me to confirm this.

[mrvdcsg]

I can confirm that we are seeing the same issue: "failed to get policy from Opensearch API" showing as a warning in the logs; however, the policy is successfully created in the cluster. This appears to be an issue confirming it exists to not try to recreate it. We are using FLUX which checks regularly if there are changes to be deployed so this error might be happening on FLUX deploy.