[BUG] Opensearch index template creation fails via terraform

pastibog commented 1 year ago

What is the bug?

Attempting to create an index template via terraform results in an [type=x_content_parse_exception]. The resource definition is the exact one taken from here: https://registry.terraform.io/providers/opensearch-project/opensearch/latest/docs/resources/index_template

How can one reproduce the bug?

Terraform file:

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.0.0-beta.1"
    }
  }
}
resource "opensearch_index_template" "template_1" {
  name = "template_1"
  body = <<EOF
{
  "template": "te*",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z YYYY"
        }
      }
    }
  }
}
EOF
}

Nothing has been changed in the resource definition, it is the exact one from the provider page.

What is the expected behavior?

The index template is created.

What is your host/environment?

AWS OpenSearch 2.7 domain Terraform v1.4.5 opensearch-project/opensearch 2.0.0-beta.1

jordarlu commented 1 year ago

cc : @phillbaker , and @prudhvigodithi ... if you guys can help take a look . thanks!!

huziahmetovsv commented 1 year ago

Same for me: Onprem OpenSearch 2.9.0 Terraform v1.5.7 opensearch-project/opensearch 2.0.0-beta.1

prudhvigodithi commented 1 year ago

Hey @pastibog @huziahmetovsv works for me, please check the following main.tf file. I have tested with index_template and composable_index_template.

`main.tf`

## Import the provider

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.0.0-beta.1"
    }
  }
}

## Configure the provider
provider "opensearch" {
  url = "http://127.0.0.1:9200"
  username = "admin"
  password = "admin"
  insecure = true
}

## Create the resource 
resource "opensearch_composable_index_template" "template_1" {
  name = "template_1"
  body = <<EOF
{
  "index_patterns": ["te*", "bar*", "logs*"],
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1
      }
    },
    "mappings": {
      "properties": {
        "host_name": {
          "type": "keyword"
        },
        "created_at": {
          "type": "date",
          "format": "EEE MMM dd HH:mm:ss Z yyyy"
        }
      }
    },
    "aliases": {
      "mydata": { }
    }
  },
  "priority": 200,
  "version": 3
}
EOF
}

resource "opensearch_index_template" "template_2" {
  name = "template_2"
  body = <<EOF
  {
    "index_patterns": [
      "logs-2020-01-*"
    ],
    "template": {
      "aliases": {
        "my_logs": {}
      },
      "mappings": {
        "properties": {
          "timestamp": {
            "type": "date",
            "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
          },
          "value": {
            "type": "double"
          }
        }
      }
    }
  }
EOF
}

API test `index_template`

curl http://127.0.0.1:9200/_index_template/template_2 -u admin:admin | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   292  100   292    0     0    566      0 --:--:-- --:--:-- --:--:--   570
{
  "index_templates": [
    {
      "name": "template_2",
      "index_template": {
        "index_patterns": [
          "logs-2020-01-*"
        ],
        "template": {
          "mappings": {
            "properties": {
              "value": {
                "type": "double"
              },
              "timestamp": {
                "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis",
                "type": "date"
              }
            }
          },
          "aliases": {
            "my_logs": {}
          }
        },
        "composed_of": []
      }
    }
  ]
}

API test `composable_index_template`

curl http://127.0.0.1:9200/_index_template/template_1 -u admin:admin | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   355  100   355    0     0    689      0 --:--:-- --:--:-- --:--:--   692
{
  "index_templates": [
    {
      "name": "template_1",
      "index_template": {
        "index_patterns": [
          "te*",
          "bar*",
          "logs*"
        ],
        "template": {
          "settings": {
            "index": {
              "number_of_shards": "1"
            }
          },
          "mappings": {
            "properties": {
              "created_at": {
                "format": "EEE MMM dd HH:mm:ss Z yyyy",
                "type": "date"
              },
              "host_name": {
                "type": "keyword"
              }
            }
          },
          "aliases": {
            "mydata": {}
          }
        },
        "composed_of": [],
        "priority": 200,
        "version": 3
      }
    }
  ]
}

Some fields have changed with 2.x for index_templates, please refer https://opensearch.org/docs/latest/im-plugin/index-templates/#composable-index-templates.

Adding @phillbaker @bbarani @peterzhuamazon

Thank you

huziahmetovsv commented 1 year ago

Thanx, works for me

pastibog commented 1 year ago

Thanks, will try this.

prudhvigodithi commented 1 year ago

Thanks for the confirmation @huziahmetovsv, closing this issue, @pastibog @huziahmetovsv please feel free re-open if required. Thank you

opensearch-project / terraform-provider-opensearch