Can't iterate over certificate_validation_records attributes of aws_apprunner_custom_domain_association resource

[vmignot]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v4.3.0

Affected Resource(s)

• aws_apprunner_custom_domain_association

Terraform Configuration Files

resource "aws_apprunner_custom_domain_association" "main" {
  domain_name          = "${local.domain_name}.${data.aws_route53_zone.main.name}"
  service_arn          = aws_apprunner_service.main.arn
  enable_www_subdomain = true
}

resource "aws_route53_record" "main-www" {
  name           = local.domain_name
  set_identifier = local.domain_name

  type    = "CNAME"
  zone_id = data.aws_route53_zone.main.zone_id
  ttl     = 300
  records = [aws_apprunner_service.main.service_url]

  weighted_routing_policy {
    weight = 90
  }
}

resource "aws_route53_record" "main-cert" {
  for_each = {
    for entry in aws_apprunner_custom_domain_association.main.certificate_validation_records : entry.name => {
      name   = entry.name
      record = entry.value
      type   = entry.type
    }
  }

  allow_overwrite = true
  zone_id         = data.aws_route53_zone.main.zone_id
  type            = each.value.type
  ttl             = 300
  name            = each.key
  records         = [each.value.record]
}

Expected Behavior

The resource aws_route53_record.main-cert should be created properly. We should be able to iterate through aws_apprunner_custom_domain_association.main.certificate_validation_records dynamically.

Actual Behavior

│ Error: Invalid for_each argument
│ 
│   on modules/app_runner/main.tf line 94, in resource "aws_route53_record" "main-cert":
│   94:   for_each = {
│   95:     for entry in aws_apprunner_custom_domain_association.main.certificate_validation_records : entry.name => {
│   96:       name   = entry.name
│   97:       record = entry.value
│   98:       type   = entry.type
│   99:     }
│  100:   }
│     ├────────────────
│     │ aws_apprunner_custom_domain_association.main.certificate_validation_records is a set of object, known only after apply
│ 
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the
│ for_each depends on.

Steps to Reproduce

1. terraform apply

[justinretzolk]

Hey @vmignot 👋 Thank you for taking the time to raise this. Based on the error, a targeted apply may be needed in order to work around this due to the values not being known at plan time. This is mentioned as a limitation in the for_each documentation as well, with a note about the possible need for a targeted apply.

After a targeted apply, I suspect this would probably work (though I haven't set up a reproduction, and am not entirely familiar with the structure of this resource, so don't hold me to that). In case you need it, the documentation around targeting resources may be found here.

[vmignot]

Hi, thanks for your reply.

To be maybe more precise, I can't figure why this resource can't work the way acm_certificate_validation works, as it is described here: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/acm_certificate_validation#dns-validation-with-route-53

Regards,

[justinretzolk]

Hey @vmignot 👋 Thank you for following up, and for pointing that out. Looking at the schema for these two resources, I'd think that a similar for_each approach could be used. Perhaps this is due to the aws_acm_certificate resource having Set: defined. I'm going to mark this as a bug so that we can look into it and determine whether this should be possible.

[jritsema]

I was able to reproduce this bug. In case it helps, I noticed that if you remove the aws_route53_record.main-cert resource then run apply, then re-add it, it works as expected.

[jritsema]

Perhaps this is due to the aws_acm_certificate resource having Set: defined.

I tried implementing Set locally and it still didn't work.

I also compared aws_acm_certificate.domain_validation_options with aws_apprunner_custom_domain_association.certificate_validation_records and the only other notable difference is that the one that works implements CustomizeDiffFunc, however, this implementation doesn't seem to do anything that would affect this behavior.

[jritsema]

We may have incorrectly assumed that the aws_acm_certificate.domain_validation_options works in a single apply. I believe I have gotten this to work in the past, however, it appears that it no longer works? see https://github.com/hashicorp/terraform-provider-aws/issues/14447

[anilmujagic]

Using the learnings from here I'm using this workaround:

resource "aws_route53_record" "api_certificate_validation" {
    count = length(aws_apprunner_custom_domain_association.api.certificate_validation_records)

    name = aws_apprunner_custom_domain_association.api.certificate_validation_records.*.name[count.index]
    type = aws_apprunner_custom_domain_association.api.certificate_validation_records.*.type[count.index]
    ttl = 300
    zone_id = data.aws_route53_zone.api.id

    records = [aws_apprunner_custom_domain_association.api.certificate_validation_records.*.value[count.index]]
}

[mwaaas]

@anilmujagic

That will still result on the same error

The "count" value depends on resource attributes that cannot be determined │ until apply, so Terraform cannot predict how many instances will be │ created. To work around this, use the -target argument to first apply only │ the resources that the count depends on

[hostmit]

Hey! Is there a solution to it?

[mwaaas]

I ended up running the service with the target module to make sure dependent resources are being created first.

[kevincloud]

@justinretzolk The documentation does address this. However, I think most of us are boggled about why Terraform has trouble with this. Terraform does its own dependency mapping, and therefore should be able to run the aws_route53_record resource after the aws_apprunner_custom_domain_association resource is created. After that resource is created, the certificate_validation_records values are available for the aws_route53_record resource to use.

This is really not expected behavior. Due to dependency mapping, Terraform infers a promise that these values can be used in the same script. Just like how the endpoint for an aws_db_instance is available for use in the same configuration, so should the certificate_validation_records map be available for use within a script.

As for solutions, for larger configurations, the -target workaround is not even an option. For the other workaround:

When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.

There are no examples of how to do this. It's a convoluted message and I personally can't make sense of it (especially since the values are available at the time the script runs the resource block...again, due to dependency mapping). It would be great if in the documentation, there would be an example of how to define the map keys statically for use in the same config.

Is that something HashiCorp can provide?

[holograph]

"I think most of us are boggled about why Terraform has trouble with this" pretty much nails it. Unfortunately this is still a problem as of 1.3.4 with provider 4.26, and it took a lot of attempts to find a reasonable workaround (via the tolist(...)[0] hack). I'm obviously missing the deeper technical context, but... this does seem far simpler than everything else Terraform already does, what am I missing?

[TechplexEngineer]

@holograph can you elaborate on the tolist hack? How does one use that to avoid needing to use -target?

[holograph]

I used the following code successfully @TechplexEngineer. As I understand it, the tolist(...) (instead of direct array access) has Terraform create a runtime dependency as opposed to a statically known one - which is the default for array access, and obviously won't work because the DNS values aren't known ahead of time - and allows validation to function correctly. This obviously won't work in case of multiple validation options, but honestly who has those ;-)

resource "aws_acm_certificate" "example" {
  domain_name       = var.domain_name
  validation_method = "DNS"

}

resource "aws_acm_certificate_validation" "example" {
  certificate_arn = aws_acm_certificate.example.arn
}

resource "aws_route53_record" "example" {
  zone_id = var.route53_public_zone_id
  ttl     = 60
  type    = "CNAME"
  name    = tolist(aws_acm_certificate.example.domain_validation_options)[0].resource_record_name
  records = [tolist(aws_acm_certificate.example.domain_validation_options)[0].resource_record_value]
}

[fracampit]

Full working example:

resource "aws_apprunner_service" "this" {
  . . .
}

resource "aws_apprunner_custom_domain_association" "this" {
  domain_name = "custom-sub-domain.my-domain.com"
  service_arn = aws_apprunner_service.this.arn
}

data "aws_route53_zone" "this" {
  name = "my-domain.com"
}

resource "aws_route53_record" "validation_records_linglinger_1" {
  name     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[0].name
  type     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[0].type
  records  = [tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[0].value]
  ttl      = 300
  zone_id  = data.aws_route53_zone.this.id
}

resource "aws_route53_record" "validation_records_linglinger_2" {
  name     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[1].name
  type     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[1].type
  records  = [tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[1].value]
  ttl      = 300
  zone_id  = data.aws_route53_zone.this.id
}

resource "aws_route53_record" "validation_records_linglinger_3" {
  name     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[2].name
  type     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[2].type
  records  = [tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[2].value]
  ttl      = 300
  zone_id  = data.aws_route53_zone.this.id
}

resource "aws_route53_record" "custom_domain" {
  name    = aws_apprunner_custom_domain_association.this.domain_name
  type    = "CNAME"
  records = [aws_apprunner_service.this.service_url]
  ttl     = 300
  zone_id = data.aws_route53_zone.this.id
}

At the time of writing, AWS provides 3 certificate validation records. The above exmple works with 3 certificate validation records. It also creates the record to send traffic to the app via the custom domain.

The above example enables the following custom domain for the app: custom-sub-domain.my-domain.com

[holograph]

I'm not sure why you'd get three separate validation records - I always get just the one? - but this is, as far as I can tell, the only way to automate this...

[nemosupremo]

I tried @fracampit's workaround with something like

resource "cloudflare_record" "byteman-verify" {
  count      = 3 //length(aws_apprunner_custom_domain_association.byteman.certificate_validation_records)
  zone_id    = var.CLOUDFLARE_ZONE
  name       = element(aws_apprunner_custom_domain_association.byteman.certificate_validation_records.*.name, count.index)
  value      = element(aws_apprunner_custom_domain_association.byteman.certificate_validation_records.*.value, count.index)
  type       = "CNAME"
  proxied    = false
  depends_on = [aws_apprunner_custom_domain_association.byteman]
}

but this broke when one of my app runners needed only 2 domains

[Habikki]

I wanted to chime in on this as I encountered this same issue today running 1.5.4 with aws provider 5.8.0.

Here's my full main.tf:

# Retrieve our data
data "aws_route53_zone" "dns_zone" {
  zone_id = var.service_dns_zone_id
}

# Declare any locals
locals {
    service_fqdn = "${var.service_host_name}.${data.aws_route53_zone.dns_zone.name}"
}

# Build our AppRunner Service
module "appruner_service" {
    source = "../apprunner"
    ecr_access_role_arn = var.ecr_access_role_arn
    service_name = var.service_name
}

# Create the custom domain name association
resource "aws_apprunner_custom_domain_association" "apprunner_service_domain_association" {
  domain_name = local.service_fqdn
  service_arn = module.appruner_service.apprunner_service_arn
}

## Add our validation Records
resource "aws_route53_record" "apprunner_service_domain_validation" {
  for_each = {
    for cvr in aws_apprunner_custom_domain_association.apprunner_service_domain_association.certificate_validation_records : cvr.name => {
      name   = cvr.name
      record = cvr.value
      type   = cvr.type
    }
  }

  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = data.aws_route53_zone.dns_zone.zone_id
}

# Finally, create a record in Route53 to the new endpoint
resource "aws_route53_record" "apprunner_service_custom_domain_record" {
  name    = aws_apprunner_custom_domain_association.apprunner_service_domain_association.domain_name
  type    = "CNAME"
  records = [module.appruner_service.apprunner_service_url]
  ttl     = 300
  zone_id = data.aws_route53_zone.dns_zone.zone_id
}

Note: I sub-module out the setup of aws_apprunner_service to encapsulate some defaults I have in my projects, otherwise, it's fairly stock.

What I find pecular is that my ability to for_each works... but only when my assocation looks like:

resource "aws_apprunner_custom_domain_association" "apprunner_service_domain_association" {
  domain_name = local.service_fqdn
  service_arn = module.appruner_service.apprunner_service_arn
}

If I enable_www_subdomain = false on the aws_apprunner_custom_domain_association:

# Create the custom domain name association
resource "aws_apprunner_custom_domain_association" "apprunner_service_domain_association" {
  domain_name = local.service_fqdn
  service_arn = module.appruner_service.apprunner_service_arn
  enable_www_subdomain = false
}

Then I receive the similar error to those above.:

│ Error: Invalid for_each argument
│
│   on ../../modules/common/apprunner_with_domain/main.tf line 27, in resource "aws_route53_record" "apprunner_service_domain_validation":
│   27:   for_each = {
│   28:     for cvr in aws_apprunner_custom_domain_association.apprunner_service_domain_association.certificate_validation_records : cvr.name => {
│   29:       name   = cvr.name
│   30:       record = cvr.value
│   31:       type   = cvr.type
│   32:     }
│   33:   }
│     ├────────────────
│     │ aws_apprunner_custom_domain_association.apprunner_service_domain_association.certificate_validation_records is a set of object, known only after apply
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this
│ resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.

Enabling or disabling www is not worth the effort for me to look into this further as without DNS resolution setup, it's an benign setting. But wanted to share my functional code, with a condition, for anyone interested.

[AndreMiras]

Refactored @fracampit example leveraging count.index:

resource "aws_apprunner_service" "this" {
  . . .
}

resource "aws_apprunner_custom_domain_association" "this" {
  domain_name = "custom-sub-domain.my-domain.com"
  service_arn = aws_apprunner_service.this.arn
}

data "aws_route53_zone" "this" {
  name = "my-domain.com"
}

resource "aws_route53_record" "validation_records_linglinger" {
  count = 3
  name     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[count.index].name
  type     = tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[count.index].type
  records  = [tolist(aws_apprunner_custom_domain_association.this.certificate_validation_records)[count.index].value]
  ttl      = 300
  zone_id  = data.aws_route53_zone.this.id
}

resource "aws_route53_record" "custom_domain" {
  name    = aws_apprunner_custom_domain_association.this.domain_name
  type    = "CNAME"
  records = [aws_apprunner_service.this.service_url]
  ttl     = 300
  zone_id = data.aws_route53_zone.this.id
}

[ewbankkit]

It looks like we will need to do something similar to https://github.com/hashicorp/terraform-provider-aws/pull/14199 and attempt to compute domain_validation_options during the plan (in a CustomizeDiff function), else it remains unknown.

[em-jones]

Is there an established workaround for this issue? Past the two-year mark gives me pause on whether/not I should keep app runner as a solution.