rs-garrick commented 1 year ago

Terraform Core Version

v1.3.2

AWS Provider Version

v4.35.0

Affected Resource(s)

aws_elasticsearch_domain

Expected Behavior

I expect 'override_main_response_version' = "false" to be accepted by an OpenSearch 1.3 domain.

Actual Behavior

Error: error updating Elasticsearch Domain (arn:aws:es:us-east-1:3XX1:domain/foo) config: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.

This is with an OpenSearch 1.3 domain.

Relevant Error/Panic Output Snippet

~ resource "aws_elasticsearch_domain" "es" {
      ~ advanced_options      = {
          ~ "override_main_response_version"         = "true" -> "false"
            # (1 unchanged element hidden)
        }
...
Error: error updating Elasticsearch Domain (arn:aws:es:us-east-1:3XX1:domain/foo) config: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.

Terraform Configuration Files

resource "aws_elasticsearch_domain" "es" { domain_name = var.es_domain_name elasticsearch_version = var.es_version

advanced_options = { "override_main_response_version" = substr(var.es_version, 0, 10) == "OpenSearch" ? "false" : null "rest.action.multi.allow_explicit_index" = "true" }

Steps to Reproduce

I have another OpenSearch 1.3 domain that works correctly. I don't know why this particular one errors like this.

Debug Output

No response

Panic Output

No response

Important Factoids

Both of my OpenSearch 1.3 domains were upgraded from previous elasticsearch domains.

References

No response

Would you like to implement a fix?

No response

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

rs-garrick commented 1 year ago

The override option is already in the state:

    {
      "module": "module.xxx",
      "mode": "managed",
      "type": "aws_elasticsearch_domain",
      "name": "es",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "access_policies": ...,
            "advanced_options": {
              "override_main_response_version": "false",
              "rest.action.multi.allow_explicit_index": "true"
            },
            "advanced_security_options": [
              {
                "enabled": false,
                "internal_user_database_enabled": false,
                "master_user_options": []
              }
            ],
...
          "elasticsearch_version": "OpenSearch_1.3",

danroestorf commented 1 year ago

I have a similar issue when running the Elasticsearch_7.10 engine version. It looks like existing clusters were defaulted to have compatibility mode on. When importing an existing cluster and running an apply I see that it is being switched to null, and I'm unable to set it to true..

advanced_options = {
  + "override_main_response_version" = "true"
  - "indices.fielddata.cache.size"                 = "20" -> null
  - "indices.query.bool.max_clause_count"          = "1024" -> null
  - "override_main_response_version"               = "true" -> null
    # (1 unchanged element hidden)
}

│ Error: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.
│ 
│   with aws_opensearch_domain.default,
│   on elasticsearch.tf line 1, in resource "aws_opensearch_domain" "default":
│    1: resource "aws_opensearch_domain" "default" {

After removing it and successfully applying some changes to the imported cluster, I now always see this diff:

# aws_opensearch_domain.meez_api will be updated in-place
  ~ resource "aws_opensearch_domain" "default" {
      ~ advanced_options = {
          - "indices.fielddata.cache.size"           = "20" -> null
          - "indices.query.bool.max_clause_count"    = "1024" -> null
          - "override_main_response_version"         = "true" -> null
            # (1 unchanged element hidden)
        }
        id               = "arn:aws:es:us-east-1:xxx:domain/default-elasticsearch"
        tags             = {}
        # (8 unchanged attributes hidden)

        # (10 unchanged blocks hidden)
    }

rockydonohue commented 1 year ago

`resource "aws_elasticsearch_domain" "opensearch" { domain_name = var.cluster_name elasticsearchversion = "OpenSearch${var.cluster_version}" access_policies = data.aws_iam_policy_document.access_policy.json

cluster_config { dedicated_master_enabled = var.master_instance_enabled dedicated_master_count = var.master_instance_enabled ? var.master_instance_count : null dedicated_master_type = var.master_instance_enabled ? var.master_instance_type : null

instance_count = var.hot_instance_count
instance_type  = var.hot_instance_type

warm_enabled = var.warm_instance_enabled
warm_count   = var.warm_instance_enabled ? var.warm_instance_count : null
warm_type    = var.warm_instance_enabled ? var.warm_instance_type : null

zone_awareness_enabled = (var.availability_zones > 1) ? true : false

dynamic "zone_awareness_config" {
  for_each = (var.availability_zones > 1) ? [var.availability_zones] : []
  content {
    availability_zone_count = zone_awareness_config.value
  }
}

}

ebs_options { ebs_enabled = var.ebs_enabled volume_size = var.ebs_volume_size volume_type = var.ebs_volume_type throughput = var.ebs_throughput
iops = var.ebs_iops }

advanced_options = { "override_main_response_version" = "true" }

advanced_security_options { enabled = true internal_user_database_enabled = false

master_user_options {
  master_user_arn = (var.master_user_arn != "") ? var.master_user_arn : data.aws_caller_identity.current.arn
}

}

domain_endpoint_options { enforce_https = true tls_security_policy = "Policy-Min-TLS-1-2-2019-07"

custom_endpoint_enabled         = true
custom_endpoint                 = "${var.cluster_name}.${data.aws_route53_zone.opensearch.name}"
custom_endpoint_certificate_arn = module.acm.acm_certificate_arn

}

node_to_node_encryption { enabled = true }

encrypt_at_rest { enabled = true kms_key_id = var.encrypt_kms_key_id }

tags = var.tags

depends_on = [aws_iam_service_linked_role.es] }`

xposix commented 1 year ago

I'm having the same issue, it appeared when migrating Opensearch 1.0 to 1.2 via Infrastructure as Code

rishabhToshniwal commented 1 year ago

Same Issue I am facing:

aws_provider_version: 4.55.0 terraform_version: 1.3.6 phillbaker/elasticsearch : 2.0.7

Error: updating Elasticsearch Domain config: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.

DagW commented 1 year ago

Seeing this too, kind of fresh install - but with the override version set. In my case, the clusters were created fine, but adding one more advanced option produced this error

  ~ resource "aws_opensearch_domain" "opensearch" {
      ~ advanced_options   = {
          + "action.auto_create_index"       = "false"
            # (1 unchanged element hidden)
        }

helene-henrichsen-cko commented 1 year ago

aws_provider_version: 4.66.1 terraform_version: 0.12.31 elasticsearch : 7.10

Similar issue that it always wants to update this on every run despite no changes:

  ~ resource "aws_elasticsearch_domain" "es" {
      ~ advanced_options      = {
          - "override_main_response_version"         = "true" -> null
            "rest.action.multi.allow_explicit_index" = "false"
        }

doukha commented 1 year ago

I'm unable to explain it but when i set

 advanced_options = {
    "override_main_response_version"         = "true"
  }

it don't shows me the changes anymore.

gromnsk commented 1 year ago

same issue, I can't update Opensearch via terraform because of this error Terraform v1.0.7 aws provider v4.67.0

module.elastic.aws_elasticsearch_domain.logs: Modifying... [id=arn:aws:es:eu-central-1:513829378710:domain/es]
╷
│ Error: updating Elasticsearch Domain (arn:aws:es:eu-central-1:513829378710:domain/es) config: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.

tried different version of aws provider, but same problem. if I remove override_main_response_version, then I have update when it should not be

rishabhToshniwal commented 1 year ago

same issue, I can't update Opensearch via terraform because of this error Terraform v1.0.7 aws provider v4.67.0
module.elastic.aws_elasticsearch_domain.logs: Modifying... [id=arn:aws:es:eu-central-1:513829378710:domain/es]
╷
│ Error: updating Elasticsearch Domain (arn:aws:es:eu-central-1:513829378710:domain/es) config: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions.
tried different version of aws provider, but same problem. if I remove override_main_response_version, then I have update when it should not be

You can try removing "override_main_response_version" and set it using the Post API as mentioned in AWS documentation (https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html). The terraform plan will show drift for the same, but it should not impact anything.

albertoCrego commented 1 year ago

Any alternative to modifying this parameter with the terraform-provider?, in my case we use the SAML role to access the cluster and changing the same config in every cluster could be really long. Provider options are the best option

dbaggott commented 11 months ago

Since it's currently impossible to update the override_main_response_version property via terraform, you can make it so terraform doesn't manage this property via:

lifecycle {
  ignore_changes = [
    # Workaround for https://github.com/hashicorp/terraform-provider-aws/issues/27371
    advanced_options["override_main_response_version"]
  ]
}

This is not ideal but it at least avoids "perpetual diff hell"

zsnyder commented 10 months ago

The ignore_changes lifecycle meta-argument will prevent the plan from nagging, but it won't help if you're trying to initiate an upgrade via Terraform.

Ignoring the override_main_response_version parameter will prevent you from being able to initiate an upgrade by modifying the engine_version resource argument. This seems to be because override_main_response_version can only be passed to the AWS API as part of an UpgradeDomain request. This bug causes the parameter to be included in the UpdateDomainConfig request--resulting in the validation error.

My workaround is to explicitly set the override_main_response_version = null in the advanced_options. This allows the upgrade to succeed if engine_version changes. When I'm not doing an upgrade, I add the ignore_changes lifecycle meta-argument to prevent the nagging.

leetn00b commented 6 months ago

I have also been having the same issue but with the aws_opensearch_domain resource, using latest terraform aws provider version 5.38.0.

Example included below:

resource "aws_opensearch_domain" "example" {
  domain_name    = "example"
  engine_version = "OpenSearch_2.11"

  cluster_config {
    instance_type = "r4.large.search"
  }
  advanced_options = {
    "override_main_response_version" = "false"
  }
}

fire2 commented 6 months ago

Same issue here with aws_opensearch_domain...

fposch commented 4 months ago

We also discovered the same issue with _action.auto_createindex - is there any intent of fixing this behavior?

In my opinion, rolling out adjusted configuration to an AWS cluster isn't exactly an edge case (for example, AWS OpenSearch has auto-creation disabled which can be quite a pain if your service writes unreflected to an non-existing index).

shalomm-gloat commented 2 months ago

encountered this as well, I guess this is still waiting.

venkata-plutotv commented 10 hours ago

encountered this as well, Is there any ETA or new solution provided for this BUG

hashicorp / terraform-provider-aws

[Bug]: ValidationException: Unrecognized advanced option 'override_main_response_version' passed in advancedOptions #27371

Terraform Core Version

AWS Provider Version

Affected Resource(s)

Expected Behavior

Actual Behavior

Relevant Error/Panic Output Snippet

Terraform Configuration Files

Steps to Reproduce

Debug Output

Panic Output

Important Factoids

References

Would you like to implement a fix?

Community Note