aws_route53_record records getting destroyed but should not be

[scott-doyland-burrows]

Terraform Version

Terraform v1.5.1
on linux_amd64
+ provider registry.terraform.io/cloudposse/awsutils v0.16.0
+ provider registry.terraform.io/hashicorp/aws v4.67.0
+ provider registry.terraform.io/hashicorp/local v2.4.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/time v0.9.1
+ provider registry.terraform.io/hashicorp/tls v3.4.0

Terraform Configuration Files

locals {
  alternative_domains = [
    "theandpartnership.dk",
    "*.theandpartnership.dk",
    "theandpartnership.com",
    "*.theandpartnership.com",
    "muster.co.uk",
    "*.muster.co.uk",
  ]
}

resource "aws_acm_certificate" "primary_domain" {

  provider          = aws.ou
  domain_name       = "*.${var.environment}.${data.aws_route53_zone.root.name}"
  validation_method = "DNS"

  subject_alternative_names = local.alternative_domains
}

# ALB certificate records
resource "aws_route53_record" "cert_validation" {
  provider = aws.ou
  for_each = {
    for dvo in aws_acm_certificate.primary_domain.domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }

  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = data.aws_route53_zone.root.zone_id
}

# ALB certificate validation
resource "aws_acm_certificate_validation" "primary_domain" {
  provider = aws.ou

  certificate_arn         = aws_acm_certificate.primary_domain.arn
  validation_record_fqdns = [for record in aws_route53_record.cert_validation : record.name]
}

# Cloudfront certificate
resource "aws_acm_certificate" "primary_domain_cdn" {
  lifecycle {
    create_before_destroy = true
  }

  count    = var.with_cdn == true ? 1 : 0
  provider = aws.cdn

  domain_name       = "*.${var.environment}.${data.aws_route53_zone.root.name}"
  validation_method = "DNS"

  subject_alternative_names = local.alternative_domains
}

# cloudfront certificate validation
resource "aws_acm_certificate_validation" "primary_domain_cdn" {
  count    = var.with_cdn == true ? 1 : 0
  provider = aws.cdn

  certificate_arn         = aws_acm_certificate.primary_domain_cdn[0].arn
  validation_record_fqdns = [for record in aws_route53_record.cert_validation : record.name]
}

Debug Output

Here is an apply when I am adding in a new local.alternative_domains to the list, as can be see, terraform destroys the route53 records, and there are mentions of deposed object.

It is a lot of output, but essentially at the end you can see it is destroying all the route53 records, even though they should be replaced (destroyed/created) according to the output from the plan.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
-/+ destroy and then create replacement
+/- create replacement and then destroy

Terraform will perform the following actions:

  # module.terraform-module-environment.aws_acm_certificate.primary_domain must be replaced
+/- resource "aws_acm_certificate" "primary_domain" {
      ~ arn                       = "arn:aws:acm:eu-west-1:[redacted]:certificate/f14a6dcc-518c-468d-a6a0-7a7264e3639f" -> (known after apply)
      ~ domain_validation_options = [
          - {
              - domain_name           = "*.muster.co.uk"
              - resource_record_name  = "[redacted].muster.co.uk."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "*.prod.tandp.[redacted].digital"
              - resource_record_name  = "[redacted].prod.tandp.[redacted].digital."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zsqxqtfsbl.acm-validations.aws."
            },
          - {
              - domain_name           = "*.theandpartnership.com"
              - resource_record_name  = "[redacted].theandpartnership.com."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "*.theandpartnership.dk"
              - resource_record_name  = "[redacted].theandpartnership.dk."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "muster.co.uk"
              - resource_record_name  = "[redacted].muster.co.uk."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "prod-corp.tandp.[redacted].digital"
              - resource_record_name  = "[redacted].prod-corp.tandp.[redacted].digital."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zsqxqtfsbl.acm-validations.aws."
            },
          - {
              - domain_name           = "prod.tandp.[redacted].digital"
              - resource_record_name  = "[redacted].prod.tandp.[redacted].digital."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zsqxqtfsbl.acm-validations.aws."
            },
          - {
              - domain_name           = "theandpartnership.com"
              - resource_record_name  = "[redacted].theandpartnership.com."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          + {
              + domain_name           = "*.muster.co.uk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "*.prod.tandp.[redacted].digital"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "*.theandpartnership.com"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "*.theandpartnership.dk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "muster.co.uk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "prod-corp.tandp.[redacted].digital"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "prod.tandp.[redacted].digital"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "theandpartnership.com"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "theandpartnership.dk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
        ]
      ~ id                        = "arn:aws:acm:eu-west-1:[redacted]:certificate/f14a6dcc-518c-468d-a6a0-7a7264e3639f" -> (known after apply)
      ~ key_algorithm             = "RSA_2048" -> (known after apply)
      ~ not_after                 = "2024-08-08T23:59:59Z" -> (known after apply)
      ~ not_before                = "2023-07-11T00:00:00Z" -> (known after apply)
      ~ pending_renewal           = false -> (known after apply)
      ~ renewal_eligibility       = "ELIGIBLE" -> (known after apply)
      ~ renewal_summary           = [] -> (known after apply)
      ~ status                    = "ISSUED" -> (known after apply)
      ~ subject_alternative_names = [ # forces replacement
          + "theandpartnership.dk",
            # (8 unchanged elements hidden)
        ]
      - tags                      = {} -> null
      ~ type                      = "AMAZON_ISSUED" -> (known after apply)
      ~ validation_emails         = [] -> (known after apply)
        # (3 unchanged attributes hidden)

      - options {
          - certificate_transparency_logging_preference = "ENABLED" -> null
        }
    }

  # module.terraform-module-environment.aws_acm_certificate.primary_domain_cdn[0] must be replaced
+/- resource "aws_acm_certificate" "primary_domain_cdn" {
      ~ arn                       = "arn:aws:acm:us-east-1:[redacted]:certificate/76097512-6bbe-489f-b615-b3e3267fb535" -> (known after apply)
      ~ domain_validation_options = [
          - {
              - domain_name           = "*.muster.co.uk"
              - resource_record_name  = "[redacted].muster.co.uk."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "*.prod.tandp.[redacted].digital"
              - resource_record_name  = "[redacted].prod.tandp.[redacted].digital."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zsqxqtfsbl.acm-validations.aws."
            },
          - {
              - domain_name           = "*.theandpartnership.com"
              - resource_record_name  = "[redacted].theandpartnership.com."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "*.theandpartnership.dk"
              - resource_record_name  = "[redacted].theandpartnership.dk."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "muster.co.uk"
              - resource_record_name  = "[redacted].muster.co.uk."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          - {
              - domain_name           = "prod-corp.tandp.[redacted].digital"
              - resource_record_name  = "[redacted].prod-corp.tandp.[redacted].digital."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zsqxqtfsbl.acm-validations.aws."
            },
          - {
              - domain_name           = "prod.tandp.[redacted].digital"
              - resource_record_name  = "[redacted].prod.tandp.[redacted].digital."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zsqxqtfsbl.acm-validations.aws."
            },
          - {
              - domain_name           = "theandpartnership.com"
              - resource_record_name  = "[redacted].theandpartnership.com."
              - resource_record_type  = "CNAME"
              - resource_record_value = "[redacted].zykwrxlgwv.acm-validations.aws."
            },
          + {
              + domain_name           = "*.muster.co.uk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "*.prod.tandp.[redacted].digital"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "*.theandpartnership.com"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "*.theandpartnership.dk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "muster.co.uk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "prod-corp.tandp.[redacted].digital"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "prod.tandp.[redacted].digital"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "theandpartnership.com"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
          + {
              + domain_name           = "theandpartnership.dk"
              + resource_record_name  = (known after apply)
              + resource_record_type  = (known after apply)
              + resource_record_value = (known after apply)
            },
        ]
      ~ id                        = "arn:aws:acm:us-east-1:[redacted]:certificate/76097512-6bbe-489f-b615-b3e3267fb535" -> (known after apply)
      ~ key_algorithm             = "RSA_2048" -> (known after apply)
      ~ not_after                 = "2024-08-08T23:59:59Z" -> (known after apply)
      ~ not_before                = "2023-07-11T00:00:00Z" -> (known after apply)
      ~ pending_renewal           = false -> (known after apply)
      ~ renewal_eligibility       = "ELIGIBLE" -> (known after apply)
      ~ renewal_summary           = [] -> (known after apply)
      ~ status                    = "ISSUED" -> (known after apply)
      ~ subject_alternative_names = [ # forces replacement
          + "theandpartnership.dk",
            # (8 unchanged elements hidden)
        ]
      - tags                      = {} -> null
      ~ type                      = "AMAZON_ISSUED" -> (known after apply)
      ~ validation_emails         = [] -> (known after apply)
        # (3 unchanged attributes hidden)

      - options {
          - certificate_transparency_logging_preference = "ENABLED" -> null
        }
    }

  # module.terraform-module-environment.aws_acm_certificate_validation.primary_domain must be replaced
+/- resource "aws_acm_certificate_validation" "primary_domain" {
      ~ certificate_arn         = "arn:aws:acm:eu-west-1:[redacted]:certificate/f14a6dcc-518c-468d-a6a0-7a7264e3639f" # forces replacement -> (known after apply) # forces replacement
      ~ id                      = "2023-07-11 12:04:59.22 +0000 UTC" -> (known after apply)
      ~ validation_record_fqdns = [ # forces replacement
          - "[redacted].prod-corp.tandp.[redacted].digital",
          - "[redacted].prod.tandp.[redacted].digital",
          - "[redacted].theandpartnership.com",
          - "[redacted].muster.co.uk",
          - "[redacted].theandpartnership.dk",
        ] -> (known after apply) # forces replacement
    }

  # module.terraform-module-environment.aws_acm_certificate_validation.primary_domain_cdn[0] must be replaced
-/+ resource "aws_acm_certificate_validation" "primary_domain_cdn" {
      ~ certificate_arn         = "arn:aws:acm:us-east-1:[redacted]:certificate/76097512-6bbe-489f-b615-b3e3267fb535" # forces replacement -> (known after apply) # forces replacement
      ~ id                      = "2023-07-11 12:04:57.414 +0000 UTC" -> (known after apply)
      ~ validation_record_fqdns = [ # forces replacement
          - "[redacted].prod-corp.tandp.[redacted].digital",
          - "[redacted].prod.tandp.[redacted].digital",
          - "[redacted].theandpartnership.com",
          - "[redacted].muster.co.uk",
          - "[redacted].theandpartnership.dk",
        ] -> (known after apply) # forces replacement
    }

  # module.terraform-module-environment.aws_lb_listener.https will be updated in-place
  ~ resource "aws_lb_listener" "https" {
      ~ certificate_arn   = "arn:aws:acm:eu-west-1:[redacted]:certificate/f14a6dcc-518c-468d-a6a0-7a7264e3639f" -> (known after apply)
        id                = "arn:aws:elasticloadbalancing:eu-west-1:[redacted]:listener/app/tandp-prod-alb/79b686aff10238e0/2c76a0220386e54f"
        tags              = {}
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].muster.co.uk.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].muster.co.uk" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zykwrxlgwv.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].prod.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].prod.tandp.[redacted].digital._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].prod.tandp.[redacted].digital" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zsqxqtfsbl.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].theandpartnership.com.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].theandpartnership.com._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].theandpartnership.com" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zykwrxlgwv.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].theandpartnership.dk.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].theandpartnership.dk._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].theandpartnership.dk" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zykwrxlgwv.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].muster.co.uk.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].muster.co.uk" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zykwrxlgwv.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].prod-corp.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].prod-corp.tandp.[redacted].digital._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].prod-corp.tandp.[redacted].digital" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zsqxqtfsbl.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].prod.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].prod.tandp.[redacted].digital._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].prod.tandp.[redacted].digital" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zsqxqtfsbl.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"] must be replaced
+/- resource "aws_route53_record" "cert_validation" {
      ~ fqdn                             = "[redacted].theandpartnership.com.tandp.[redacted].digital" -> (known after apply)
      ~ id                               = "Z1015136EEMLEKYDPSO8[redacted].theandpartnership.com._CNAME" -> (known after apply)
      - multivalue_answer_routing_policy = false -> null
      ~ name                             = "[redacted].theandpartnership.com" # forces replacement -> (known after apply) # forces replacement
      ~ records                          = [
          - "[redacted].zykwrxlgwv.acm-validations.aws.",
        ] -> (known after apply)
      ~ type                             = "CNAME" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.dk"] will be created
  + resource "aws_route53_record" "cert_validation" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = (known after apply)
      + records         = (known after apply)
      + ttl             = 60
      + type            = (known after apply)
      + zone_id         = "Z1015136EEMLEKYDPSO8"
    }

  # module.terraform-module-environment.module.cdn[0].aws_cloudfront_distribution.default[0] will be updated in-place
  ~ resource "aws_cloudfront_distribution" "default" {
      ~ aliases                        = [
          + "theandpartnership.dk",
            # (7 unchanged elements hidden)
        ]
        id                             = "E27HVDH4Y62S08"
        tags                           = {
            "Attributes" = "default"
            "Name"       = "tandp-prod-cfd-default"
        }
        # (18 unchanged attributes hidden)

      ~ viewer_certificate {
          ~ acm_certificate_arn            = "arn:aws:acm:us-east-1:[redacted]:certificate/76097512-6bbe-489f-b615-b3e3267fb535" -> (known after apply)
          ~ cloudfront_default_certificate = false -> (known after apply)
          ~ ssl_support_method             = "sni-only" -> (known after apply)
            # (1 unchanged attribute hidden)
        }

        # (9 unchanged blocks hidden)
    }

Plan: 13 to add, 2 to change, 12 to destroy.

Do you want to perform these actions in workspace "di-devops-terraform-dynamic-sites__prod-tandp"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.terraform-module-environment.aws_acm_certificate_validation.primary_domain_cdn[0]: Destroying... [id=2023-07-11 12:04:57.414 +0000 UTC]
module.terraform-module-environment.aws_acm_certificate_validation.primary_domain_cdn[0]: Destruction complete after 0s
module.terraform-module-environment.aws_acm_certificate.primary_domain_cdn[0]: Creating...
module.terraform-module-environment.aws_acm_certificate.primary_domain: Creating...
module.terraform-module-environment.aws_acm_certificate.primary_domain: Still creating... [10s elapsed]
module.terraform-module-environment.aws_acm_certificate.primary_domain_cdn[0]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_acm_certificate.primary_domain: Creation complete after 10s [id=arn:aws:acm:eu-west-1:[redacted]:certificate/f604c396-b31e-41ee-9c28-1962f0efab9e]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.dk"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Creating...
module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"]: Creating...
module.terraform-module-environment.aws_acm_certificate.primary_domain_cdn[0]: Creation complete after 11s [id=arn:aws:acm:us-east-1:[redacted]:certificate/a5bcaab8-7f5b-4868-96b4-7f74f99bce07]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.dk"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Still creating... [10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.dk"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Still creating... [20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.dk"]: Creation complete after 26s [id=Z1015136EEMLEKYDPSO8[redacted].theandpartnership.dk._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Still creating... [30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"]: Creation complete after 32s [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Creation complete after 36s [id=Z1015136EEMLEKYDPSO8[redacted].theandpartnership.dk._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"]: Creation complete after 39s [id=Z1015136EEMLEKYDPSO8[redacted].prod.tandp.[redacted].digital._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"]: Creation complete after 40s [id=Z1015136EEMLEKYDPSO8[redacted].theandpartnership.com._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still creating... [40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still creating... [40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still creating... [40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still creating... [40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Creation complete after 41s [id=Z1015136EEMLEKYDPSO8[redacted].prod-corp.tandp.[redacted].digital._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Creation complete after 41s [id=Z1015136EEMLEKYDPSO8[redacted].prod.tandp.[redacted].digital._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Creation complete after 42s [id=Z1015136EEMLEKYDPSO8[redacted].theandpartnership.com._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Creation complete after 42s [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME]
module.terraform-module-environment.aws_acm_certificate_validation.primary_domain: Creating...
module.terraform-module-environment.aws_acm_certificate_validation.primary_domain_cdn[0]: Creating...
module.terraform-module-environment.aws_acm_certificate_validation.primary_domain: Creation complete after 1s [id=2023-07-11 12:34:28.203 +0000 UTC]
module.terraform-module-environment.aws_lb_listener.https: Modifying... [id=arn:aws:elasticloadbalancing:eu-west-1:[redacted]:listener/app/tandp-prod-alb/79b686aff10238e0/2c76a0220386e54f]
module.terraform-module-environment.aws_acm_certificate_validation.primary_domain_cdn[0]: Creation complete after 1s [id=2023-07-11 12:34:32.087 +0000 UTC]
module.terraform-module-environment.module.cdn[0].aws_cloudfront_distribution.default[0]: Modifying... [id=E27HVDH4Y62S08]
module.terraform-module-environment.aws_lb_listener.https: Modifications complete after 1s [id=arn:aws:elasticloadbalancing:eu-west-1:[redacted]:listener/app/tandp-prod-alb/79b686aff10238e0/2c76a0220386e54f]module.terraform-module-environment.aws_acm_certificate_validation.primary_domain (deposed object 63dcefba): Destroying... [id=2023-07-11 12:04:59.22 +0000 UTC]
module.terraform-module-environment.aws_acm_certificate_validation.primary_domain: Destruction complete after 0s
module.terraform-module-environment.module.cdn[0].aws_cloudfront_distribution.default[0]: Still modifying... [id=E27HVDH4Y62S08, 4m40s elapsed]
module.terraform-module-environment.module.cdn[0].aws_cloudfront_distribution.default[0]: Modifications complete after 4m50s [id=E27HVDH4Y62S08]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"] (deposed object 0662b899): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].theandpartnership.dk._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"] (deposed object 1f57cd4f): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].theandpartnership.com._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"] (deposed object ad81a1c0): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].prod.tandp.[redacted].digital._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"] (deposed object 52d95a06): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].prod-corp.tandp.[redacted].digital._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"] (deposed object b5d23e9d): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].theandpartnership.com._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"] (deposed object 2f854307): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME]module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"] (deposed object ab1ae551): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].prod.tandp.[redacted].digital._CNAME]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"] (deposed object 496c469a): Destroying... [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME]
module.terraform-module-environment.aws_acm_certificate.primary_domain_cdn[0] (deposed object d71c7ae8): Destroying... [id=arn:aws:acm:us-east-1:[redacted]:certificate/76097512-6bbe-489f-b615-b3e3267fb535]
module.terraform-module-environment.aws_acm_certificate.primary_domain_cdn[0]: Destruction complete after 0s
module.terraform-module-environment.aws_route53_record.cert_validation["*.muster.co.uk"]: Destruction complete after 1s
module.terraform-module-environment.aws_route53_record.cert_validation["*.prod.tandp.[redacted].digital"]: Destruction complete after 1s
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.com"]: Destruction complete after 1s
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__fea3f925e07efb413...9a0d57f7c7.theandpartnership.dk._CNAME, 10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__ad3acdfbd055ca93b...437296f70.theandpartnership.com._CNAME, 10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__387dd76a4927c98d9...bd.prod.tandp.[redacted].digital._CNAME, 10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME, 10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__2e15d358b917628f4...od-corp.tandp.[redacted].digital._CNAME, 10s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__387dd76a4927c98d9...bd.prod.tandp.[redacted].digital._CNAME, 20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__fea3f925e07efb413...9a0d57f7c7.theandpartnership.dk._CNAME, 20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__ad3acdfbd055ca93b...437296f70.theandpartnership.com._CNAME, 20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME, 20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__2e15d358b917628f4...od-corp.tandp.[redacted].digital._CNAME, 20s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__2e15d358b917628f4...od-corp.tandp.[redacted].digital._CNAME, 30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME, 30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__ad3acdfbd055ca93b...437296f70.theandpartnership.com._CNAME, 30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__387dd76a4927c98d9...bd.prod.tandp.[redacted].digital._CNAME, 30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__fea3f925e07efb413...9a0d57f7c7.theandpartnership.dk._CNAME, 30s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__2e15d358b917628f4...od-corp.tandp.[redacted].digital._CNAME, 40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__ad3acdfbd055ca93b...437296f70.theandpartnership.com._CNAME, 40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__387dd76a4927c98d9...bd.prod.tandp.[redacted].digital._CNAME, 40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8__fea3f925e07efb413...9a0d57f7c7.theandpartnership.dk._CNAME, 40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Still destroying... [id=Z1015136EEMLEKYDPSO8[redacted].muster.co.uk._CNAME, 40s elapsed]
module.terraform-module-environment.aws_route53_record.cert_validation["prod.tandp.[redacted].digital"]: Destruction complete after 44s
module.terraform-module-environment.aws_route53_record.cert_validation["*.theandpartnership.dk"]: Destruction complete after 46s
module.terraform-module-environment.aws_route53_record.cert_validation["prod-corp.tandp.[redacted].digital"]: Destruction complete after 46s
module.terraform-module-environment.aws_route53_record.cert_validation["theandpartnership.com"]: Destruction complete after 47s
module.terraform-module-environment.aws_route53_record.cert_validation["muster.co.uk"]: Destruction complete after 47s
module.terraform-module-environment.aws_acm_certificate.primary_domain (deposed object bf0adcb1): Destroying... [id=arn:aws:acm:eu-west-1:[redacted]:certificate/f14a6dcc-518c-468d-a6a0-7a7264e3639f]
module.terraform-module-environment.aws_acm_certificate.primary_domain: Destruction complete after 1s

Apply complete! Resources: 13 added, 2 changed, 12 destroyed.

Expected Behavior

Route53 records do not get destroyed at the end of the apply.

Actual Behavior

Route53 records are getting destroyed.

Steps to Reproduce

terraform apply

Additional Context

The code generates a certificate, route53 certificate validation records, and lastly a certificate validation.

There are a few other things that happen in other .tf files, essentially various AWS infrastructure uses the certifcates.

If I update the local.alternative_domains by adding/deleting a domain, and then apply, it all works OK, except the route53 records are destroyed at the end of the apply, and I need a second apply to add them back again.

I cannot work out why the route53 records are destroyed. The plan does not say this will happen.

Note the mention of various deposed object as well.

This happens every time I make a local.alternative_domains update - and of course I have made sure I start from clean environment to check there is nothing broken in the statefile.

References

N/A

[apparentlymart]

Hi @scott-doyland-burrows,

From what's described in your plan it seems like the route53 record resource is in "create before destroy" mode, but the configuration of that resource is not compatible with that mode:

The underlying hashicorp/aws provider doesn't seem to be able to detect that it's being asked to create a record that already exists -- probably a limitation of the underlying API only supporting a single combined "create or replace" operation -- and so the provider thinks it's creating a new record but it's actually overwriting the one that already exists. It then tries to delete the old one, but the old one was already overwritten by the new one and so the new one gets deleted.

This is technically an AWS provider bug that could be reported in the provider's own repository, but the design of the underlying API might mean that it isn't fixable. I'll leave the provider development team to decide that.

With the provider as it is currently implemented, I think you will need to make sure that "create before destroy" is never enabled for this resource, and instead accept that the record will briefly be unavailable during an update of this kind.

Part of the problem here is that aws_acm_certificate seems to be calculating the validation record names and values only during the apply phase, so they appear as unknown during planning and thus Terraform must assume that they are going to change. If the AWS provider could instead fix the values of those attributes during planning this would not be such a problem, because it would be clear from the values that the hostnames are not changing. Again, whether that is possible in practice depends on how the underlying ACM API is designed, so I would suggest taking this to the AWS provider repository so that the provider team can evaluate it based on their knowledge of these APIs.

From Terraform Core's perspective unfortunately things behaved as designed here given this particular set of proposed changes from the provider. There are two potential improvements to the AWS provider that I'll re-summarize here:

• Make aws_route53_record fail if it's asked to create a record that already exists in the specified zone, rather than just silently overwriting the existing record. Correct Terraform behavior relies on providers not treating a create action as an update/replace action.
• Make aws_acm_certificate set at least the name and type of each validation record request to known values during planning, so that the downstream record resource instances can see that those values are not changing and thus not plan to replace the record in the first place.

The feasibility of each of these will depend on the design of the underlying AWS APIs and so I suggest opening an issue in the AWS provider repository so that the provider team can evaluate whether each of these is viable, or if they can find other solutions that would avoid this situation.

I don't think there's anything we could change in Terraform Core to improve this situation, and so I'm going to close this. Please open an issue about it in the AWS provider repository.

[scott-doyland-burrows]

The route53 record resource isn't in create-before-destroy mode but it does have this setting:

allow_overwrite = true which allows records to be overwritten. This is due to the domain CNAME cert validation records being the same for "*.example.com" and for "example.com", so route53 effectively has to create two records the same (or in other words create one, and then overwrite with the second).

Potentially I may be able to prevent this, but I don't think it will actually make any difference overall.

Thanks for the detailed reply. I will raise an issue with the AWS provider.

[apparentlymart]

  # module.terraform-module-environment.aws_acm_certificate_validation.primary_domain must be replaced
+/- resource "aws_acm_certificate_validation" "primary_domain" {

The +/- at the start of the second line above is the shorthand icon for "create and then destroy", so this resource does seem to be in create-before-destroy mode. If you didn't explicitly enable it for this resource then this must be a situation where this resource is in a dependency chain with another resource that has it enabled, and so Terraform must propagate the setting along the chain so that there is a viable order of creating and destroying.

Thanks for pointing out the allow_overwrite = true setting. I wasn't aware of that feature and it's good to know that silently overwriting is not the default behavior. I think this is the root cause of the problem in your case, though: the behavior of allow_overwrite = true is incompatible with "create before destroy" because it prevents Terraform (really: the AWS provider) from noticing that the new object it's creating conflicts with the one that hasn't been destroyed yet. If you disable that setting then you will presumably see the AWS provider notice that it would need to overwrite the existing object to apply this change, and so the create step would fail rather than it succeeding and then having its result deleted by the subsequent destroy step.

Of course, disabling allow_override will just cause it to "fail better" -- it will return an error describing the conflict rather than just leaving the remote system with no record at all -- so that second point about the aws_acm_certificate resource being able to produce known values for the validation settings during planning would be required to fully resolve this. That would presumably avoid the presumed need to replace the Route53 record, since Terraform will be able to see that the hostname and record type are not changing. Hopefully the AWS provider team can find a way to make that work!

[scott-doyland-burrows]

https://github.com/hashicorp/terraform-provider-aws/issues/32460

[scott-doyland-burrows]

One last comment - we use Cloudposse modules and they have create_before_destroy set in multiple locations.

I ran a terraform init and then commented out all the create_before_destroy from the imported modules, and then ran a plan.

I can then see that the route53 records will be destoyed/created rather than created/destroyed.

So, as you say, the issue is the create_before_destroy.

I think the only sensible way to fix this, is for us to move away from Cloudposse modules (even then we may find reasons to use create_before_destroy in our own code).

I don't think it would be possible to fix this any other way, eg even if a plan could calculate the certificate validation values that will be passed to route53 records, this still will cause issues if we are creating/deleting domains from the certificate, as then route53 will still need to add/delete these records (regardless of whether they are known in the plan or not).

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.