Support for multiple OpenSearch domains/workspaces/endpoints

[barthel]

What/Why

What are you proposing?

I would like to propose the possibility to configure more than one static OpenSearch domain at the same time with a terraform apply.

What users have asked for this feature?

There are a lot of users who, for example, want to further process a set of results from a terraform module where the static configuration of a provider prevents them from doing so. However, this will not be possible in terraform for the foreseeable future.

What problems are you trying to solve?

We are building a complete platform with everthing-as-code and terraform in AWS. This includes a lot of OpenSearch domains (in AWS managed OpenSearch). The AWS terraform provider does not offer OpenSearch specific configurations. Based on the created OpenSearch domains (list), we now face the challenge of having to perform user mapping for each individual domain, for example. This does not work with statically initialized providers, because each domain has its own endpoint.

What is the developer experience going to be?

Are there any security considerations?

AWS Request Signing must be supported (already supported).

Are there any breaking changes to the API

Resources must be customized to support an optional endpoint (e.g., found via a DataSource).

What is the user experience going to be?

For example, as a user I want to be able to iterate over my list of OpenSearch domains with for_each in a resource and pass the endpoint in the resource.

Are there breaking changes to the User Experience?

This feature can be implemented as an optional feature in my opinion, so there should be no break in usage.

Why should it be built? Any reason not to?

The feature would significantly expand the provider's purpose and make it more flexible. It saves (in this particular focus) the use of terragrunt or similar external tools.

What will it take to execute?

Any remaining open questions?

How can the AWS OpenSearch Provider be harmonized with the OpenSearch Provider so that there is no break in processing?

[barthel]

Interesting concept, used by https://registry.terraform.io/providers/elastic/elasticstack/latest/docs

[barthel]

@phillbaker

[phillbaker]

Hi @barthel, this is built into terraform itself, there's no need for changing this provider specifically. Please see the terraform docs here: https://developer.hashicorp.com/terraform/language/providers/configuration

[barthel]

Thanks for the hint @phillbaker but I know this concept and it doesn't fit my needs.

As long as the list of domains is known and static before plan/apply the alias concept could be used. But in my case, the list is dynamic and it is not possible to iterate over this list and configure e.g. the user mapping for each domain because of the static configured endpoint.

Another example is a fresh created domain. It's not possible to use the provider right after the domain creation because the endpoint is not predictable and the provider must be initialized with the endpoint before the plan/apply phase of terraform.

[phillbaker]

because the endpoint is not predictable and the provider must be initialized with the endpoint before the plan/apply phase of terraform

This provider supports interpolation of variables in the provider configuration. Although not clearly documented, some details are in https://github.com/phillbaker/terraform-provider-elasticsearch/pull/119. So for example, Elasticsearch clusters can be created on Elasticstack/AWS and then configured using this provider dynamically, e.g.:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "..."
    }
    elasticsearch = {
      source  = "phillbaker/elasticsearch"
      version = "..."
    }
  }
}

provider "aws" {
 ...
}

resource "aws_elasticsearch_domain" "foo" {
  domain_name           = "foo"
  elasticsearch_version = "7.10"

  cluster_config {
...
  }
...
}

provider "elasticsearch" {
  url         = "https://${aws_elasticsearch_domain.foo.endpoint}"
}

resource "elasticsearch_opensearch_ism_policy" "ism-policy" {
  policy_id = "hot-warm"
  body      = ...
}

This terraform module has a similar example of using this provider in this manner: https://github.com/idealo/terraform-aws-opensearch

[phillbaker]

Hi @barthel did you get a chance to take a look at my last comment? I believe it addresses the aspects described.