Handle array for ismTemplate in ISMPolicy opensearch API

[cthtrifork]

Description

Change from

ISMTemplate *ISMTemplate json:"ismTemplate,omitempty"

to

ISMTemplate []ISMTemplate json:"ismTemplate,omitempty"

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

https://github.com/opensearch-project/opensearch-k8s-operator/issues/739

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cgroschupp]

operator-controller-manager {"level":"error","ts":"2024-03-07T08:00:47.969Z","msg":"failed to get policy from Opensearch API","controller":"opensearchismpolicy","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchISMPolicy","OpenSearchISMPolicy":{"name":"myismpolicy","namespace":"default"},"namespace":"default","name":"myismpolicy","reconcileID":"642d2228-afef-4ca3-9d6e-7c4b171d9e3a","reconciler":"ismpolicy","error":"json: cannot unmarshal array into Go struct field ISMPolicy.policy.ism_template of type requests.ISMTemplate","stacktrace":"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers.(*IsmPolicyReconciler).Reconcile\n\t/workspace/pkg/reconcilers/ismpolicy.go:200\ngithub.com/Opster/opensearch-k8s-operator/opensearch-operator/controllers.(*OpensearchISMPolicyReconciler).Reconcile\n\t/workspace/controllers/opensearchism_controller.go:53\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

I have a custom build of the operator based on the latest commit of the main branch, I get this error when updating this OpenSearchISMPolicy.

apiVersion: opensearch.opster.io/v1
kind: OpenSearchISMPolicy
metadata:
  name: myismpolicy
spec:
  opensearchCluster:
    name: testcluster
  policyId: example
  description: "Example Policy"
  ismTemplate:
    indexPatterns:
      - "test-*"
    priority: 100
  defaultState: hot
  states:
    - name: hot
      actions:
        - rollover:
            minIndexAge: 1h
            minPrimaryShardSize: 10mb
      transitions:
        - stateName: delete
          conditions:
            minIndexAge: 3h
    - name: delete
      actions:
        - delete: {}

[cthtrifork]

What version of opensearch are you running? It seems we need to support the object both as an array or object. I am not sure how, since the schema is autogenerated from the code.

[cgroschupp]

I am using version 2.11.1.

[cthtrifork]

I tried recreating your ISM manually and got the following structure when i choose "view json" in OpenSearch dashboard:

And I get the same from the rest API:

curl -k https://opensearch.opensearch.svc.cluster.local:9200/_plugins/_ism/policies/myismpolicy

{"_id":"myismpolicy","_version":1,"_seq_no":475148,"_primary_term":9,"policy":{"policy_id":"myismpolicy","description":"A sample description of the policy","last_updated_time":1709883236528,"schema_version":19,"error_notification":null,"default_state":"hot","states":[{"name":"hot","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"rollover":{"min_doc_count":5,"min_index_age":"1h","min_primary_shard_size":"10mb","copy_alias":false}}],"transitions":[{"state_name":"delete","conditions":{"min_index_age":"3h"}}]},{"name":"delete","actions":[{"retry":{"count":3,"backoff":"exponential","delay":"1m"},"delete":{}}],"transitions":[]}],"ism_template":[{"index_patterns":["test-*"],"priority":1,"last_updated_time":1709883236528}]}}

I am running docker.io/opensearchproject/opensearch:2.11.1

[cgroschupp]

@swoehrl-mw any ideas to solve this? My ideas:

• Introduce a new apiVersion for OpenSearchISMPolicy and only support OpenSearch versions above x.x.x there.
• Define ISMTemplate as json.RawMessage and support both cases in code.

[cgroschupp]

Ok, found out that only the transition is not working. When I create an ISMPolicy with the dev build, the creation and also the update works.

But when I create an ISMPolicy with 2.5.1 and then update to the dev build, I get the already mentioned error message.

When is a new release planned?

[swoehrl-mw]

@cgroschupp

When is a new release planned?

No concrete plans on a new release yet. Probably when more new features have accumulated. Although this one is something that could be part of a bugfix patch release @prudhvigodithi .

Introduce a new apiVersion for OpenSearchISMPolicy and only support OpenSearch versions above x.x.x there.

Probably the best solution long-term, but for the short-term I'd like to avoid that as it has quite the overhead. For the short term it's best to hide this in operator logic (there are already other places in the code where logic is done differently based on version so its nothing new) and accept a less than optimal interface.

Define ISMTemplate as json.RawMessage and support both cases in code.

I'd like to avoid that as it removes alot of the validation that the kubernetes api does on its own.