aws_elasticsearch_domain shows persistent diff for cognito options after disabling

[ericksoen]

For the aws_elasticsearch_domain resource defined below, there is a persistent diff that appears for cognito_options when enabled=false and the other configuration options are valued. This issue only appears after at least one terraform apply cycle has been executed where enabled=true. After that cycle, subsequent executions always show a diff for cognito_options and the values are never persisted to terraform.tfstate.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

$ terraform -v
> Terraform v0.14.7
> + provider registry.terraform.io/hashicorp/aws v3.36.0

Affected Resource(s)

• aws_elasticsearch_domain

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

variable "vpc" { }

variable "domain" { }

variable "identity_pool_id" { }

variable "user_pool_id" { }

data "aws_vpc" "selected" {
  tags = {
    Name = var.vpc
  }
}

data "aws_subnet_ids" "selected" {
  vpc_id = data.aws_vpc.selected.id

  tags = {
    Tier = "private"
  }
}

data "aws_region" "current" {}

data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "assume" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type = "Service"
      identifiers = ["es.amazonaws.com"]
    }
  }
}

resource "aws_iam_role_policy" "test_policy" {
  name = "test_policy_inline"
  role = aws_iam_role.es.id

  policy = jsonencode({
      Version = "2012-10-17"
      Statement = [
        {
          Effect = "Allow"
          Action = ["cognito-idp:*", "cognito-identity:*"]
          Resource = "*"
        },
        {
          Effect = "Allow"
          Action = "iam:PassRole"
          Resource = "*"
          Condition = {
            "StringEqualsIfExists": {
              "iam:PassedToService": "cognito-identity.amazonaws.com"
            }
          }
        }
      ]
    })
}

resource "aws_iam_role" "es" {
  name = "${var.domain}-es-role"

  assume_role_policy = data.aws_iam_policy_document.assume.json
}
resource "aws_security_group" "es" {
  name        = "${var.vpc}-elasticsearch-${var.domain}"
  description = "Managed by Terraform"
  vpc_id      = data.aws_vpc.selected.id

  ingress {
    from_port = 443
    to_port   = 443
    protocol  = "tcp"

    cidr_blocks = [
      data.aws_vpc.selected.cidr_block,
    ]
  }
}

resource "aws_elasticsearch_domain" "es" {
  domain_name           = "${var.domain}-v2"
  elasticsearch_version = "6.3"

  cluster_config {
    instance_type = "t3.small.elasticsearch"
  }

  vpc_options {
    subnet_ids = [
      tolist(data.aws_subnet_ids.selected.ids)[0],
    ]

    security_group_ids = [aws_security_group.es.id]
  }

  advanced_options = {
    "rest.action.multi.allow_explicit_index" = "true"
  }

  access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Resource": "arn:aws:es:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:domain/${var.domain}/*"
        }
    ]
}
CONFIG

  snapshot_options {
    automated_snapshot_start_hour = 23
  }

  ebs_options {
    ebs_enabled = true
    volume_size = 100
  }

  cognito_options {
    enabled = false
    identity_pool_id = var.identity_pool_id
    role_arn = aws_iam_role.es.arn
    user_pool_id = var.user_pool_id
  }

}

Debug Output

Panic Output

Expected Behavior

Terraform does not show a diff for any cognito options if enabled = false and there are no changes to other cognito properties like identity_pool_id, role_arn or user_pool_id.

Actual Behavior

Terraform shows diff for every apply to identity_pool_id, role_arn or user_pool_id when enabled = false.

Steps to Reproduce

1. terraform apply
2. Update enabled = true
3. terraform apply
4. Update enabled = false
5. terraform apply

Important Factoids

1. AWS API returns cognito_options with simple object when enabled = false for initial apply

   $ terraform output domain | xargs -I {} aws es describe-elasticsearch-domain --domain-name {} --query DomainStatus.CognitoOptions
   > { "Enabled": false }

2. Terraform state file does not show any cognito_options when enabled = false for initial apply

   $ cat terraform.tfstate | jq -r '.resources[]|select(.type | startswith("aws_elasticsearch_domain"))' | jq .instances[0].attributes.cognito_options
   > [ {"enabled": false, "identity_pool_id": "", "role_arn": "", "user_pool_id": "" } ]

3. Terraform debug logs show same values for CognitoOptions as returnd by the AWS CLI

   CognitoOptions: {
   Enabled: false
   },

4. After updating enabled=true and applying, the AWS API and Terraform state file all show the provided values for identity_pool_id, role_arn, and user_pool_id:

{
    "Enabled": true,
    "UserPoolId": "region_id",
    "IdentityPoolId": "region:guid",
    "RoleArn": "arn:aws:iam::account_id:role/tf-test-es-role"
}

1. After toggling enabled = false again and re-applying, the AWS API and Terraform state file now show conflicting representations:
   • AWS CLI

     {
     "Enabled": false,
     "UserPoolId": "region_id",
     "IdentityPoolId": "region:guid",
     "RoleArn": "arn:aws:iam::account_id:role/tf-test-es-role"
     }

   • TF State

     [ {"enabled": false, "identity_pool_id": "", "role_arn": "", "user_pool_id": "" } ]

References

• 0000

[YakDriver]

@ericksoen Ugh! Thanks for reporting this and the very detailed information. That will definitely make it easier to track down the problem. I'm not sure when we'll have a chance to tackle this. Hopefully, you or someone from the community will be able to address this.

It looks like there may already be an issue for this at #5557. (For future travelers, this does not look terribly similar to #14628.)

[ericksoen]

I'm hoping to take a look at this one at some point. For my immediate needs, I needed a GitHub issue id to add to my project source code to explain why the issue persists 😊.

Thanks for the cross-reference on #5557 (at first glance it didn't seem similar to me, but I'll review with a closer eye this time).

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.