[BUG] Opensearch ISM Policy resource

dirnikk commented 9 months ago

Terraform version: 1.4.6
Opensearch Terraform provider: 2.0.0
Opensearch cluster hosted in AWS: OpenSearch_2_7_R20230928

It seems that there are 2 issues with the resource opensearch_ism_policy:

When I run the following terraform code:
```
resource "opensearch_ism_policy" "studio-retention-policy" {
policy_id = "studio-retention-policy"
body      = file("../index-policies/studio-retention-policy.json")
}
```
If the resource is not created beforehand it gives the following error:
```
Error: error getting policy: /_plugins/_ism/policies/retention-policy : elastic: Error 404 (Not Found): Policy not found [type=status_exception]

with opensearch_ism_policy.retention-policy,
on main.tf line 162, in resource "opensearch_ism_policy" "retention-policy":
162: resource "opensearch_ism_policy" "retention-policy" {
```
The correct behaviour should be if there is no such policy let Terraform create it. This is the logic for a resource in Terraform in general. If I want to manager resources created outside of Terraform I should import them. The resource works like semi-apply-import thing. I noticed similar behaviour with role_mapping resource the difference is that once the changes from Terraform are applied it does not tries to reapply the same changes on next Terraform run.

And if I manually create the resource in Opensearch dashboard it tries to apply some changes:

Terraform will perform the following actions:

# opensearch_ism_policy.retention-policy will be updated in-place
~ resource "opensearch_ism_policy" "retention-policy" {
  ~ body         = jsonencode(
      ~ {
          ~ policy = {
              + error_notification = null
              + policy_id          = "retention-policy"
              + schema_version     = 17
                # (4 unchanged attributes hidden)
            }
        }
    )
    id           = "retention-policy"
    # (3 unchanged attributes hidden)
}

The code completes successfully and the modifications are done. However, if I run the same code again it tries to apply the same changes again and I don't understand why. The expected behaviour is once the changes are applied to not re-apply the same thing again. I compared line by line with what I have in the JSON Policy in Opensearch dashboard and what I have in the policy_document.json file. They are the same.

{
"policy": {
    "policy_id": "retention-policy",
    "description": "Demonstrate a retention-policy.",
    "schema_version": 17,
    "error_notification": null,
    "default_state": "hot_state",
    "states": [
        {
            "name": "hot_state",
            "actions": [],
            "transitions": [
                {
                    "state_name": "warm_state",
                    "conditions": {
                        "min_index_age": "2d"
                    }
                },
                {
                    "state_name": "warm_state",
                    "conditions": {
                        "min_size": "30gb"
                    }
                }
            ]
        },
        {
            "name": "warm_state",
            "actions": [
                {
                    "timeout": "24h",
                    "retry": {
                        "count": 5,
                        "backoff": "exponential",
                        "delay": "1h"
                    },
                    "warm_migration": {}
                }
            ],
            "transitions": [
                {
                    "state_name": "cold_state",
                    "conditions": {
                        "min_index_age": "28d"
                    }
                }
            ]
        },
        {
            "name": "cold_state",
            "actions": [
                {
                    "retry": {
                        "count": 3,
                        "backoff": "exponential",
                        "delay": "1m"
                    },
                    "cold_migration": {
                        "start_time": null,
                        "end_time": null,
                        "timestamp_field": "@timestamp",
                        "ignore": "none"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "30d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "retry": {
                        "count": 3,
                        "backoff": "exponential",
                        "delay": "1m"
                    },
                    "cold_delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "eks_*",
                "logstash-*",
                "cwl-*"
            ],
            "priority": 1
        }
    ]
}
}

Also there is a configuration in the json policy file for all 3 attributes that Terraform wants to update.

Any information will be much appreciated, thank you :)

prudhvigodithi commented 9 months ago

[Untriage] Adding @rblcoder, can you please take a look and your thoughts? Thank you

rblcoder commented 9 months ago

The following example works with the attached json. I am able to create the policy and re-apply does not show changes.

OpenSearch v2.11.0 Terraform v1.6.0 OpenSearch Terraform provider v2.0.0

resource "opensearch_ism_policy" "rollover_index_policy" {
  policy_id = "rollover_policy"
  body      = jsonencode(jsondecode(file("./policy.json")))

}

policy.json

The equivalent code with inline json


resource "opensearch_ism_policy" "rollover_index_policy" {
  policy_id = "rollover_policy"
  body = jsonencode(
  {
  "policy": {
    "description": "Example rollover policy.",
    "default_state": "rollover",
    "states": [
      {
        "name": "rollover",
        "actions": [
          {
            "rollover": {
              "min_doc_count": 1
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": ["log*"],
      "priority": 100
    }
  }
}

  )
}

dirnikk commented 9 months ago

Hey @rblcoder, Thanks for checking! What could be the reason of not creating the resource if does not exists?

rblcoder commented 9 months ago

@dirnikk I am able to create the policy through this code without manual creating the policy through OpenSearch UI or API.

dirnikk commented 9 months ago

@rblcoder thanks again, It's so confusing why it interact like that. I can see there are some yellow line in my ide not sure why:

Could you please paste the exact code that you are using to create the resource?

provider config:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.12.0"
    }
    opensearch = {
      source  = "opensearch-project/opensearch"
      version = "2.0.0"
    }
  }
}

provider "opensearch" {
  url         = "https://${var.cluster_name}.${data.aws_route53_zone.opensearch.name}"
  aws_region  = data.aws_region.current.name
  healthcheck = false
}

Thanks!

rblcoder commented 9 months ago

@dirnikk Here is the exact code

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.0.0"
    }
  }
}

provider "opensearch" {
  url = "http://127.0.0.1:9200"
  username = "admin"
  password = "admin"
}

resource "opensearch_ism_policy" "rollover_index_policy" {
  policy_id = "rollover_policy"
  body      = jsonencode(jsondecode(file("./policy.json")))

}

prudhvigodithi commented 9 months ago

Thanks @rblcoder, Closing this issue @dirnikk please re-open if required.

adelsz commented 1 month ago

Running into both of these issues with latest provider version 2.3.0. Can this be reopened? @prudhvigodithi

opensearch-project / terraform-provider-opensearch

[BUG] Opensearch ISM Policy resource #121