Certificate keeps recreating

[41jules1967]

I have a recurrent random issues with aws_iam_server_certificate. I run an apply and it runs through without error. Subsequent runs however identify that the certificate needs to be recreated again (even though it is unchanged). As it already exists and certficates must be replaced rather than updated, I get

• aws_iam_server_certificate.ipfs_cert1: [WARN] Error uploading server certificate, error: EntityAlreadyExists: The Server Certificate with name ipfs-cert1 already exists. 2 apply runs immediately after each other result in this. Once this starts happening, I can't get around it, but it doesn't always occur, even on the same scripts against another environment, hence annoyingly, providing you with a clean test example is impossible.

To my mind, the root cause is terraform mis-identifying that the certificate object needs to be changed in the first place, thereby dropping us into the second wider AWS issue that you can't update a certificate object. I have attached a debug TFLOG.txt

[41jules1967]

I have a theory. The issue started when I swapped my self signed cert for a proper Geotrust one. The Geotrust cert is a chain and is 4226 bytes whereas my original self signed one is 1338 bytes. Is terraform failing to recognise a file over a certain size is unchanged?

[41jules1967]

Anyone got any thoughts?

[41jules1967]

Bit more in case you need it. I have run terraform apply 4 times this hour. The cert file hasn't been touched yet it reports this. arn: "arn:aws:iam::107210838220:server-certificate/ipfs-cert" => "" certificate_body: "3418efc0e290bfb0a08fac4d49ad3ebd39a70ebd" => "af26a111805dcc36ced216bfdf8a98a3d5c05860" (forces new resource) certificate_chain: "0c331ed5c65f2e465fb19d0977f6400b7eb9d2c4" => "" name: "ipfs-cert" => "ipfs-cert" path: "/" => "/" private_key: "39659b8e1c343e14986bade75bc33010c0eb074f" => "39659b8e1c343e14986bade75bc33010c0eb074f"

So it seems to recognise the smaller PK file is unchanged, but is generating a different (checksum?) each time for the certificate body. The only workaround is to rename the cert store, and the corresponding entry on the elb each run, which is a bit of a pain. I'd appreciate something from the devs to say at least they're aware. Thanks

[apparentlymart]

Hi @41jules1967. Sorry for the problem here and for the delay in replying.

I'm not in a good spot to try to repro this right now (in transit) but given that the object does exist but yet Terraform is losing track of it this smells like an "eventual consistency" bug with the IAM API, where after creating a certificate for a brief time when we read it back we get a 404 or some other sort of response that makes us think the cert doesn't exist, and so Terraform drops it from the state. We could confirm this if you run Terraform with TF_LOG=debug set and look in the log output for a message like the following:

[WARN] IAM Server Cert (...) not found, removing from state

Assuming that is the problem, I'd normally suggest for now a workaround of manually importing the certificate using terraform import after Terraform loses track of it, but sadly this resource has not yet had an import process implemented and so that solution won't work here.

[41jules1967]

Hi, Thanks for the response. I can't find that line in the debug log I'm afraid. I'm not clear on the bug you describe but if it's a brief time sync issue in AWS I'm not sure that's it as the cert can be created days earlier, it will still throw the same error.

[donnoman]

I think you may be hitting this that terraform knows about

certificate_chain: "0c331ed5c65f2e465fb19d0977f6400b7eb9d2c4" => ""

is the clue.

NOTE: AWS performs behind-the-scenes modifications to some certificate files if they do not adhere to a specific format. These modifications will result in terraform forever believing that it needs to update the resources since the local and AWS file contents will not match after theses modifications occur. In order to prevent this from happening you must ensure that all your PEM-encoded files use UNIX line-breaks and that certificate_body contains only one certificate. All other certificates should go in certificate_chain. It is common for some Certificate Authorities to issue certificate files that have DOS line-breaks and that are actually multiple certificates concatenated together in order to form a full certificate chain.

this kinda stuff drives me crazy... if you know this then fix it on your side terraform. In my case I'm use the same cert files for nginx, nginx needs the combined. but terraform keeps recreating them.

so now I have to make "SPECIAL" set of certs to satisfy terraform :rage1:

the moment I created that special set where the chain was in a separate file the plan stopped trying to recreate it; I didn't even have to apply again.

[41jules1967]

Thanks Donnoman. You are exactly right. Discovered this a couple of weeks ago (but lost track of this post to update it so thanks) . It's actually documented, though whether that's a recent thing or because when I originally wrote my scripts, I was using a simple cert so didn't have the issue, then didn't revisit the docs when I changed the certs and the problem started I'm not sure. But yes, I broke the cert into a cert and a chain file , and removed the comment lines and as you say, it stops doing it.

[klokkie]

This was driving me potty too, seems I was reading over it and though that it was re-doing the checksum. Nice spot of certificate_chain, that should be an easy fix.

[apparentlymart]

Given @donnoman's explanation above I think we could do better here:

• We can use a StateFunc on the relevant attributes that have certificates that will normalize the line endings and comments so that Terraform won't consider these differences as significant.
• We could explicitly catch the problem of putting multiple certificates in certificate_body and generate a helpful error message for that case.

So I'm going to reopen this to represent the above changes. This sort of issue where AWS normalizes something and Terraform sees it as a perpetual diff are an ongoing challenge but one that is, thankfully, usually somewhat easy to address.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.