[Bug]: Changing tags on aws_secret_manager_secret resource can break usage of the secret via aws_secret_manager_secret_version datasource

Terraform Core Version

0.15.5

AWS Provider Version

4.54.0

Affected Resource(s)

resource aws_secretsmanager_secret
data aws_secretsmanager_secret_version

Expected Behavior

Making changes to tags on a aws_secretsmanager_secret resource should not cause a problem with using the secret with a aws_secretsmanager_secret_version datasource.

A diff of

resource "aws_secretsmanager_secret" "audit_logs_elasticsearch_terraform_credentials" {
  name = "${local.elastic_service_name}-${local.elastic_service_env}-elasticsearch_terraform-CREDENTIALS"

   tags = {
     Application = local.elastic_service_name
     Environment = local.elastic_service_env
-    team        = local.elastic_team
+    Squad       = local.elastic_team
+    ServiceEnv  = local.elastic_service_env
   }
 }
}

Should have no effect on the usage of a secret

data "aws_secretsmanager_secret_version" "audit_logs_elasticsearch_terraform_credentials" {
  secret_id = aws_secretsmanager_secret.audit_logs_elasticsearch_terraform_credentials.id
}

provider "elasticsearch" {
  url                   = "https://HOSTNAME:9243"
  username              = jsondecode(data.aws_secretsmanager_secret_version.audit_logs_elasticsearch_terraform_credentials.secret_string)["username"]
  password              = jsondecode(data.aws_secretsmanager_secret_version.audit_logs_elasticsearch_terraform_credentials.secret_string)["password"]
  elasticsearch_version = "7.15.2"
}

Actual Behavior

Making a change to the tags of a aws_secretsmanager_secret resource is causing a problem when a aws_secretsmanager_secret_version datasource is used to access the secrets.

running "/home/atlantis/.atlantis/bin/terraform0.15.5 plan -input=false -refresh -no-color -out \"/home/atlantis/.atlantis/repos/vestwell/docker-registry/4163/default/terraform/dev.ue1/services/audit_log/dev.ue1::services::audit_log-default.tfplan\"" in "/home/atlantis/.atlantis/repos/vestwell/docker-registry/4163/default/terraform/dev.ue1/services/audit_log": exit status 1
aws_secretsmanager_secret.audit_logs_elasticsearch_terraform_credentials: Refreshing state... [id=arn:aws:secretsmanager:us-east-1:NUM:secret:audit-log-dev-elasticsearch_terraform-CREDENTIALS-EtuAUC]
...
Error: elastic: Error 401 (Unauthorized)

  with elasticsearch_xpack_index_lifecycle_policy.ilm_audit_log_policy,
  on elastic.tf line 109, in resource "elasticsearch_xpack_index_lifecycle_policy" "ilm_audit_log_policy":
 109: resource "elasticsearch_xpack_index_lifecycle_policy" "ilm_audit_log_policy" {

This unexpected behavior does not occur in situations where a aws_secretsmanager_secret resource that does not have any tags is changed to have some tags. It occurs, so far, only if the resource has existing tags and there are changes.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

  required_version = "0.15.5"

  required_providers {
    elasticsearch = {
      source  = "phillbaker/elasticsearch"
      version = "2.0.0-beta.2"
    }
    aws = {
      version = "4.54.0"
    }
  }
}

locals {
  elastic_service_name      = "audit-log"
  elastic_name              = "elastic"
  elastic_service_env       = "dev"
  elastic_team              = "RED"
}

resource "aws_secretsmanager_secret" "audit_logs_elasticsearch_terraform_credentials" {
  name = "${local.elastic_service_name}-${local.elastic_service_env}-elasticsearch_terraform-CREDENTIALS"

  tags = {
    Application = local.elastic_service_name
    Environment = local.elastic_service_env
    team        = local.elastic_team
  }
}

data "aws_secretsmanager_secret_version" "audit_logs_elasticsearch_terraform_credentials" {
  secret_id = aws_secretsmanager_secret.audit_logs_elasticsearch_terraform_credentials.id
}

provider "elasticsearch" {
  url                   = "https://HOSTNAME:9243"
  username              = jsondecode(data.aws_secretsmanager_secret_version.audit_logs_elasticsearch_terraform_credentials.secret_string)["username"]
  password              = jsondecode(data.aws_secretsmanager_secret_version.audit_logs_elasticsearch_terraform_credentials.secret_string)["password"]
  elasticsearch_version = "7.15.2"
}

resource "elasticsearch_component_template" "idx_comp_templ_ecs_agent" {
  name = "ecs_agent@mappings"
  body = file("${path.module}/data/index_templates/component/ecs/agent.json")
}

/data/index_templates/component/ecs/agent.json

{
  "_meta": {
    "documentation": "https://www.elastic.co/guide/en/ecs/current/ecs-agent.html",
    "ecs_version": "8.2.1"
  },
  "template": {
    "mappings": {
      "properties": {
        "agent": {
          "properties": {
            "build": {
              "properties": {
                "original": {
                  "ignore_above": 1024,
                  "type": "keyword"
                }
              }
            },
            "ephemeral_id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "name": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "type": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "version": {
              "ignore_above": 1024,
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

Steps to Reproduce

You can start an Elastic stack

docker-compose.yml

version: '3.9'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.5.2
    container_name: es01-test
    environment:
      - node.name=es01-test
      - discovery.type=single-node
      - xpack.security.enabled=false
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - poc

  kibana:
    image: docker.elastic.co/kibana/kibana:8.5.2
    container_name: kib01-test
    environment:
      - ELASTICSEARCH_HOSTS=http://es01-test:9200
    ports:
      - 5601:5601
    networks:
      - poc
    depends_on:
      - elasticsearch

Navigate into the local Kibana interface. Choose Stack Management. Create a user for Terraform to use with just all permissions to keep this simple.

Get the credential values into AWS secret manager

Execute Terraform. Confirm the Elastic Index Template was created.

Change the tags on the aws_secretsmanager_secret resource

Execute Terraform again. See the error.

Debug Output

No response

Panic Output

No response

Important Factoids

We've used a secret creation process for secrets in AWS which is sort of chicken-and-egg problem but keeps secrets out of the repo and is more-convenient for multiple teams to work with. This probably isn't a recommended way yet it isn't clear what this would have to do with tag changes.

Define the secret in Terraform as a resource
Execute Terraform to have the secret created in AWS
Update the secret in AWS with the actual values
Execute Terraform to use the secret and do the rest of the work

References

No response

hashicorp / terraform-provider-aws