OpenSearch connection issues

[rasatabular]

Hi,

I have an AWS VPC with Opensearch version 1.2 which is not publicly accessible. I am trying to create I have tried different configurations for the provider but I have failed to make things work and I am trying to find out what is wrong in the configuration.

The following configuration:

terraform {
  required_providers {
    elasticsearch = {
      source  = "phillbaker/elasticsearch"
      version = "2.0.2"
    }
  }
}

provider "elasticsearch" {
  url                   = "https://${aws_opensearch_domain.main.endpoint}"
  insecure              = true
  healthcheck           = false
  elasticsearch_version = "OpenSearch_1.2"
  sign_aws_requests     = true
  aws_profile           = "my_aws_profile"
}

resource "elasticsearch_index_template" "test_template" {
  name = "test-index-template"
  body = <<EOF
{
  "template": "test_template.*",
  "settings": {
    "number_of_shards": 4
  }
}
EOF
}

Gives the following output:

module.opensearch.elasticsearch_index_template.test_template: Creating...
module.opensearch.elasticsearch_index_template.test_template: Still creating... [10s elapsed]
module.opensearch.elasticsearch_index_template.test_template: Still creating... [20s elapsed]
module.opensearch.elasticsearch_index_template.test_template: Still creating... [30s elapsed]
module.opensearch.elasticsearch_index_template.test_template: Still creating... [40s elapsed]
module.opensearch.elasticsearch_index_template.test_template: Still creating... [50s elapsed]
module.opensearch.elasticsearch_index_template.test_template: Still creating... [1m0s elapsed]
module.opensearch.elasticsearch_index_template.test_template: Still creating... [1m10s elapsed]

│ Error: Put "https://url.amazonaws.com/_template/test-index-template?create=true": dial tcp 10.0.0.22:443: connect: operation timed out
│ 

If I remove the insecure = true I get:

module.opensearch.elasticsearch_index_template.test_template: Creating...
module.opensearch.elasticsearch_index_template.test_template: Still creating... [10s elapsed]

│ Error: Put "https://url.amazonaws.com/_template/test-index-template?create=true": context deadline exceeded

The error is slightly different than before, so it might give some hint of what is wrong in my configuration.

---

The aws_profile my_aws_profile I am using is configured locally and I can use the terraform AWS provider with it without any issues.

Additionally, I tried providing the aws_region or alternatively setting the aws_access_key and aws_secret_key at the provider directly but the result was the same. I tried removing the elasticsearch_version form the provider configuration and I got:

module.opensearch.elasticsearch_index_template.test_template: Creating...

│ Error: timeout after 5 seconds while pinging 'https://url.amazonaws.com' to determine server version, please consider setting 'elasticsearch_version' to avoid this lookup

I have also verified that the 'https://url.amazonaws.com' is the correct URL showing in my AWS console under Domain endpoint (VPC).

[moritzzimmer]

We're using a serverless API proxy (basically an API Gateway using proxy integration to a python Lambda deployed in the VPC) in order to be able to connect to a VPC based OpenSearch domain API via public internet/in terraform.

provider "elasticsearch" {
  aws_region            = data.aws_region.current.name
  aws_signature_service = "execute-api"
  elasticsearch_version = "1.1.0"
  healthcheck           = false
  url                   = "https://${your.api_proxy_domain_name}"
}

[phillbaker]

Hi @rasatabular @moritzzimmer are the two of you referring to the same cluster? The provider configuration you provided is very different.

@rasatabular the error message is descriptive in this case: dial tcp 10.0.0.22:443: connect: operation timed out. The provider cannot reach the configured elasticsearch cluster.

Please include the following information:

• from where are you running the provider?
• where is the elasticsearch cluster located?
• what is the networking configuration for where the elasticsearch cluster runs and all the networks connecting the location where the provider is running to the cluster?

Note: In general, issues on this repository are for reporting bugs and feature requests for this provider, not providing support for unique environments.

[moritzzimmer]

@phillbaker just wanted to give an example/provide help how to use this provider to connect to an OpenSearch domain inside a VPC

[rasatabular]

Hi @phillbaker I am not sure whether the error I am getting is due to some issue with the code or me not using the repo in the correct way. I assumed that I would be using it in the same way that I use the AWS provider to deploy AWS resources.

To be more explicit:

from where are you running the provider?

I am running the TerraForm code locally. The code connect to my AWS profile and deploys the infrastructure (loadbalancer, EC2, OpenSearch, etc).

where is the elasticsearch cluster located?

The OpenSearch cluster is a VPC-accessible cluster on AWS.

what is the networking configuration for where the elasticsearch cluster runs and all the networks connecting the location where the provider is running to the cluster?

I have a VPC with two subnets and one Internet gateway. I have a few EC2 instances in the subnets and a loadbalancer that forwards traffic to the EC2 instances. I also have an RDS instance where the EC2 instances connect and the OpenSearch cluster is in the same VPC.

I also have an IAM user that I assign to the EC2 instances to be able to access OpenSearch.

What I am interested in is creating index templates from TerraForm and be able to deploy them locally.

[phillbaker]

@rasatabular this is not an issue with the provider.

If your OpenSearch cluster is a VPC-accessible cluster on AWS, then it will only be network accessible from within the VPC. You either need to run terraform from within the VPC or, if you want to run it outside the VPC (e.g. locally), set up a proxy to get into the VPC. Please see @moritzzimmer's comment for one example, but there are many ways to do this, SSH tunnels, bastion hosts, etc.