search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.74k stars 9.1k forks source link

[Bug]: ACM certificate which failed to import ends up in Terraform state #34968

Open mewa opened 8 months ago

mewa commented 8 months ago

Terraform Core Version

1.6.5

AWS Provider Version

3.76.0,5.31.0

Affected Resource(s)

Expected Behavior

ACM state is not updated if it fails to update.

Actual Behavior

ACM certificate_body is updated to value rejected by AWS during update.

Relevant Error/Panic Output Snippet

aws_acm_certificate.this: Modifying... [id=arn:aws:acm:eu-central-1:<accountId>:certificate/<certificateId>]
╷
│ Error: importing ACM Certificate (arn:aws:acm:eu-central-1:<accountId>:certificate/<certificateId>): operation error ACM: ImportCertificate, https response error StatusCode: 400, RequestID: <requestId>, api error ValidationException: New certificate is missing one or more Key Usages supported by the currently imported certificate
│
│   with aws_acm_certificate.this,
│   on main.tf line 28, in resource "aws_acm_certificate" "this":
│   28: resource "aws_acm_certificate" "this" {
│

Terraform Configuration Files

variable "validity" {
  default = 3600
}

resource "tls_private_key" "this" {
  algorithm = "RSA"
}

resource "tls_self_signed_cert" "this" {
  private_key_pem = tls_private_key.this.private_key_pem

  subject {
    common_name  = "example.com"
    organization = "Buggy Inc."
  }

  is_ca_certificate = true

  validity_period_hours = var.validity
  early_renewal_hours   = var.validity / 6

  allowed_uses = [
    "cert_signing",
    "digital_signature"
  ]
}

resource "aws_acm_certificate" "this" {
  private_key      = tls_private_key.this.private_key_pem
  certificate_body = tls_self_signed_cert.this.cert_pem

  lifecycle {
    create_before_destroy = true
  }
}

output "acm_certificate" {
  value = {
    not_after = aws_acm_certificate.this.not_after
    cert_hash = md5(aws_acm_certificate.this.certificate_body)
  }
}

output "local_certificate" {
  value = {
    not_after = formatdate("YYYY-MM-DD'T'hh:mm:ssZ", tls_self_signed_cert.this.validity_end_time)
    cert_hash = md5(tls_self_signed_cert.this.cert_pem)
  }
}

Steps to Reproduce

  1. Run terraform apply 
    
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

aws_acm_certificate.this will be created

Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

acm_certificate = { "cert_hash" = "b2f4057d0212c2e8c359120b37e3966b" "not_after" = "2023-12-21T15:31:07Z" } local_certificate = { "cert_hash" = "b2f4057d0212c2e8c359120b37e3966b" "not_after" = "2023-12-21T16:31:07+01:00" }

4. Comment one of `allowed_uses`, e.g. `digital_signature`
5. Run `terraform apply`

tls_private_key.this: Refreshing state... [id=541a7bea42183808e07377d0374f88f996357825] tls_self_signed_cert.this: Refreshing state... [id=267788273987092730818475432989880164488] aws_acm_certificate.this: Refreshing state... [id=arn:aws:acm:::certificate/]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: ~ update in-place +/- create replacement and then destroy

Terraform will perform the following actions:

aws_acm_certificate.this will be updated in-place

~ resource "aws_acm_certificate" "this" { ~ certificate_body = <<-EOT -----BEGIN CERTIFICATE----- MIIDIzCCAgugAwIBAgIRAMl2JUOZuV8nvKdLkVIXIIgwDQYJKoZIhvcNAQELBQAw KzETMBEGA1UEChMKQnVnZ3kgSW5jLjEUMBIGA1UEAxMLZXhhbXBsZS5jb20wHhcN MjMxMjE4MTUzMTA3WhcNMjMxMjIxMTUzMTA3WjArMRMwEQYDVQQKEwpCdWdneSBJ bmMuMRQwEgYDVQQDEwtleGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP ADCCAQoCggEBAKYbVd5YZQI6PWGoT8ePSnzIKKnPlQj0fnd+ULDZs6ccLw3iylTh f18t42J3ijLwJTBNJGwHOIguXoQZoFWx6MzsTdiCQs2l1EQrUX7q/t7Ya2Rcw4iR 9gWFCOM/JOc2tSvJSO6rZMJGIWMftoKbWgWXgZanGdPS2buD5PBXzVuNjppV2O5T Zve2iSbvjSCRgZQpMxYoMsd3iXihnq9T8ouviVpmFBKdkCdQlpgAbkrmbKvaZVrX BIkDogtYIC2SKIS+VU4st3P9VcPOLeyHAtpoV9czovLdi0JL9XHqcMQQojTuoQqb XxyxWObXX1+CWylU5rOvhpG1dMZ1bnj8ArsCAwEAAaNCMEAwDgYDVR0PAQH/BAQD AgKEMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOMYwN/tyJBC2kmMeA+NY6LD 5Mk7MA0GCSqGSIb3DQEBCwUAA4IBAQCdEPaul34YOgRCU6VaVPzJR2Rp+1Jr5/Zg tzVWCoN/X8TpxzxL+luLgY9HA/RAfD2312eGqPxbnviAHsDyvGYCgrucAuWtcp8+ F82fJqRacODghuRgPA6yAhVo052263x045Ax3/K574MuG63CZajHhf9vtBwfoLmN X0nRpweRlPS2RaYKjRvz+hRKZUl3eF9lXwsyk5szZrfafwoLGTIbfp9WD9GlKPAv yIangdLm60IXf0dFcRV2CCDCfwavaOogrojF7rPo8L32GALmDOlNLfumZvcvPgWG XgIDJr7YaDNPmGwfWvrfrKYMzHInNyLFHPLeW5YApqkydn6Bq3Hl -----END CERTIFICATE----- EOT -> (known after apply) id = "arn:aws:acm:::certificate/" tags = {}

(16 unchanged attributes hidden)

    # (1 unchanged block hidden)
}

tls_self_signed_cert.this must be replaced

+/- resource "tls_self_signed_cert" "this" { ~ allowed_uses = [ # forces replacement "cert_signing",

Plan: 1 to add, 1 to change, 1 to destroy.

Changes to Outputs: ~ acm_certificate = { ~ cert_hash = "b2f4057d0212c2e8c359120b37e3966b" -> (known after apply)

(1 unchanged attribute hidden)

}

~ local_certificate = { ~ cert_hash = "b2f4057d0212c2e8c359120b37e3966b" -> (known after apply) ~ not_after = "2023-12-21T16:31:07+01:00" -> (known after apply) }

7. Observe AWS error rejecting the updated certificate

tls_self_signed_cert.this: Creating... tls_self_signed_cert.this: Creation complete after 0s [id=302349904251140011571768920538688197264] aws_acm_certificate.this: Modifying... [id=arn:aws:acm:::certificate/] ╷ │ Error: importing ACM Certificate (arn:aws:acm:::certificate/): operation error ACM: ImportCertificate, https response error StatusCode: 400, RequestID: da4af74b-a878-42dd-83b0-ee0ed5f13795, api error ValidationException: New certificate is missing one or more Key Usages supported by the currently imported certificate │ │ with aws_acm_certificate.this, │ on main.tf line 28, in resource "aws_acm_certificate" "this": │ 28: resource "aws_acm_certificate" "this" { │

8. Run `terraform apply` once more

tls_private_key.this: Refreshing state... [id=541a7bea42183808e07377d0374f88f996357825] tls_self_signed_cert.this (deposed object 8843fec5): Refreshing state... [id=267788273987092730818475432989880164488] tls_self_signed_cert.this: Refreshing state... [id=302349904251140011571768920538688197264] aws_acm_certificate.this: Refreshing state... [id=arn:aws:acm:::certificate/]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:

Terraform will perform the following actions:

tls_self_signed_cert.this (deposed object 8843fec5) will be destroyed

(left over from a partially-failed replacement of this instance)

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs: ~ acm_certificate = { ~ cert_hash = "b2f4057d0212c2e8c359120b37e3966b" -> "7ecaca8b9c044c727cb4244946f20d94"

(1 unchanged attribute hidden)

}
9. Observe invalid `certificate_body` reported from `aws_acm_certificate` resource

tls_self_signed_cert.this (deposed object 8843fec5): Destroying... [id=267788273987092730818475432989880164488] tls_self_signed_cert.this: Destruction complete after 0s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Outputs:

acm_certificate = { "cert_hash" = "7ecaca8b9c044c727cb4244946f20d94" "not_after" = "2023-12-21T15:31:07Z" } local_certificate = { "cert_hash" = "7ecaca8b9c044c727cb4244946f20d94" "not_after" = "2023-12-21T16:31:37+01:00" }



### Debug Output

_No response_

### Panic Output

_No response_

### Important Factoids

_No response_

### References

_No response_

### Would you like to implement a fix?

None
github-actions[bot] commented 8 months ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

justinretzolk commented 8 months ago

Hey @mewa 👋 Thank you for taking the time to raise this, and for the great reproduction information! That was super helpful in letting me reproduce this and try to wrap my head around it.

I believe that this is the result of a Terraform Core behavior paired with an unfortunate AWS API limitation, but I'd like to leave this issue open so that someone from the team or community can take another look before I remove the bug label. The main clue to that being these lines from the plan output in step 8:

  # tls_self_signed_cert.this (deposed object 8843fec5) will be destroyed
  # (left over from a partially-failed replacement of this instance)

From the Terraform Glossary's definition of deposed:

... Terraform expected to replace the existing resource by creating a new resource, then destroying the existing resource, but an error occurred in the apply before the destruction. Existing references to the resource refer to the new resource. Terraform will destroy the deposed resource on the next apply. This only can occur in resource configurations that have the lifecycle configuration block create_before_destroy argument set to true.

While tls_self_signed_cert.this doesn't explicitly have create_before_destroy set, aws_acm_certificate.this does. Since aws_acm_certificate.this is implicitly dependent on tls_self_signed_cert.this, tls_self_signed_cert.this inherits the create_before_destroy configuration.

In step 7's logs, the new instance of tls_self_signed_cert.this was successfully created, but the apply failed during the modification of aws_acm_certificate.this -- before the previous instance of tls_self_signed_cert.this could be destroyed. This led to the deposed instance seen in the logs in step 8.

As mentioned in the documentation linked above, any references to a resource using create_before_destroy are updated immediately after the new resource is created. With that in mind, any interpolations of tls_self_signed_cert.this's attributes are updated in the state despite the modification's failure (pivotally, aws_acm_certificate.this.certificate_body). The AWS API used to manage this resource doesn't return the certificate body, which means that the difference between state and reality for that particular attribute is not detected on subsequent runs.

The difference in behavior as far as the aws_acm_certificate outputs go is due to the not_after attribute being Computed. Those types of attributes are always read from the resource directly, leading to a difference in the not_after outputs if you inspect the state.

visit1985 commented 3 months ago

@justinretzolk, thanks for you analysis. But I experienced the same issue in a different scenario, which leads me to the conclusion that deposed resources are not the root cause of this issue.

I was trying to update an existing aws_acm_certificate without create_before_destroy

resource "aws_acm_certificate" "xxx" {
  certificate_body = file("cert.pem")
  certificate_chain = file("chain.pem")
  private_key = file("key.pem")
}
  # aws_acm_certificate.xxx will be updated in-place
  ~ resource "aws_acm_certificate" "xxx" {
      ~ certificate_body          = (sensitive value)
        id                        = "arn:aws:acm:eu-central-1:xxx:certificate/xxx"
      ~ private_key               = (sensitive value)
        tags                      = {}
        # (16 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

but didn't have permission to, which resulted in an

Error: importing ACM Certificate (arn:aws:acm:eu-central-1:xxx:certificate/xxx): operation error ACM: ImportCertificate, https response error StatusCode: 400, RequestID: xxx, api error AccessDeniedException: User: arn:aws:sts::xxx:assumed-role/xxx/aws-go-sdk-xxx is not authorized to perform: acm:ImportCertificate on resource: arn:aws:acm:eu-central-1:xxx:certificate/xxx with an explicit deny in a service control policy

So I changed my IAM role and tried to apply again, which gave me

No changes. Your infrastructure matches the configuration.

I compared the state (versioned in S3) and I can confirm, that the first apply updated the certificate_body and private_key in state without any other resource involved.