Closed scott-doyland-burrows closed 1 year ago
Hi @scott-doyland-burrows,
From what's described in your plan it seems like the route53 record resource is in "create before destroy" mode, but the configuration of that resource is not compatible with that mode:
The underlying hashicorp/aws
provider doesn't seem to be able to detect that it's being asked to create a record that already exists -- probably a limitation of the underlying API only supporting a single combined "create or replace" operation -- and so the provider thinks it's creating a new record but it's actually overwriting the one that already exists. It then tries to delete the old one, but the old one was already overwritten by the new one and so the new one gets deleted.
This is technically an AWS provider bug that could be reported in the provider's own repository, but the design of the underlying API might mean that it isn't fixable. I'll leave the provider development team to decide that.
With the provider as it is currently implemented, I think you will need to make sure that "create before destroy" is never enabled for this resource, and instead accept that the record will briefly be unavailable during an update of this kind.
Part of the problem here is that aws_acm_certificate
seems to be calculating the validation record names and values only during the apply phase, so they appear as unknown during planning and thus Terraform must assume that they are going to change. If the AWS provider could instead fix the values of those attributes during planning this would not be such a problem, because it would be clear from the values that the hostnames are not changing. Again, whether that is possible in practice depends on how the underlying ACM API is designed, so I would suggest taking this to the AWS provider repository so that the provider team can evaluate it based on their knowledge of these APIs.
From Terraform Core's perspective unfortunately things behaved as designed here given this particular set of proposed changes from the provider. There are two potential improvements to the AWS provider that I'll re-summarize here:
aws_route53_record
fail if it's asked to create a record that already exists in the specified zone, rather than just silently overwriting the existing record. Correct Terraform behavior relies on providers not treating a create action as an update/replace action.aws_acm_certificate
set at least the name
and type
of each validation record request to known values during planning, so that the downstream record resource instances can see that those values are not changing and thus not plan to replace the record in the first place.The feasibility of each of these will depend on the design of the underlying AWS APIs and so I suggest opening an issue in the AWS provider repository so that the provider team can evaluate whether each of these is viable, or if they can find other solutions that would avoid this situation.
I don't think there's anything we could change in Terraform Core to improve this situation, and so I'm going to close this. Please open an issue about it in the AWS provider repository.
The route53 record resource isn't in create-before-destroy mode but it does have this setting:
allow_overwrite = true
which allows records to be overwritten. This is due to the domain CNAME cert validation records being the same for "*.example.com" and for "example.com", so route53 effectively has to create two records the same (or in other words create one, and then overwrite with the second).
Potentially I may be able to prevent this, but I don't think it will actually make any difference overall.
Thanks for the detailed reply. I will raise an issue with the AWS provider.
# module.terraform-module-environment.aws_acm_certificate_validation.primary_domain must be replaced
+/- resource "aws_acm_certificate_validation" "primary_domain" {
The +/-
at the start of the second line above is the shorthand icon for "create and then destroy", so this resource does seem to be in create-before-destroy mode. If you didn't explicitly enable it for this resource then this must be a situation where this resource is in a dependency chain with another resource that has it enabled, and so Terraform must propagate the setting along the chain so that there is a viable order of creating and destroying.
Thanks for pointing out the allow_overwrite = true
setting. I wasn't aware of that feature and it's good to know that silently overwriting is not the default behavior. I think this is the root cause of the problem in your case, though: the behavior of allow_overwrite = true
is incompatible with "create before destroy" because it prevents Terraform (really: the AWS provider) from noticing that the new object it's creating conflicts with the one that hasn't been destroyed yet. If you disable that setting then you will presumably see the AWS provider notice that it would need to overwrite the existing object to apply this change, and so the create step would fail rather than it succeeding and then having its result deleted by the subsequent destroy step.
Of course, disabling allow_override
will just cause it to "fail better" -- it will return an error describing the conflict rather than just leaving the remote system with no record at all -- so that second point about the aws_acm_certificate
resource being able to produce known values for the validation settings during planning would be required to fully resolve this. That would presumably avoid the presumed need to replace the Route53 record, since Terraform will be able to see that the hostname and record type are not changing. Hopefully the AWS provider team can find a way to make that work!
One last comment - we use Cloudposse modules and they have create_before_destroy
set in multiple locations.
I ran a terraform init
and then commented out all the create_before_destroy
from the imported modules, and then ran a plan.
I can then see that the route53 records will be destoyed/created rather than created/destroyed.
So, as you say, the issue is the create_before_destroy
.
I think the only sensible way to fix this, is for us to move away from Cloudposse modules (even then we may find reasons to use create_before_destroy
in our own code).
I don't think it would be possible to fix this any other way, eg even if a plan could calculate the certificate validation values that will be passed to route53 records, this still will cause issues if we are creating/deleting domains from the certificate, as then route53 will still need to add/delete these records (regardless of whether they are known in the plan or not).
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Terraform Version
Terraform Configuration Files
Debug Output
Here is an apply when I am adding in a new
local.alternative_domains
to the list, as can be see, terraform destroys the route53 records, and there are mentions ofdeposed object
.It is a lot of output, but essentially at the end you can see it is destroying all the route53 records, even though they should be replaced (destroyed/created) according to the output from the plan.
Expected Behavior
Route53 records do not get destroyed at the end of the apply.
Actual Behavior
Route53 records are getting destroyed.
Steps to Reproduce
terraform apply
Additional Context
The code generates a certificate, route53 certificate validation records, and lastly a certificate validation.
There are a few other things that happen in other
.tf
files, essentially various AWS infrastructure uses the certifcates.If I update the
local.alternative_domains
by adding/deleting a domain, and then apply, it all works OK, except the route53 records are destroyed at the end of the apply, and I need a second apply to add them back again.I cannot work out why the route53 records are destroyed. The plan does not say this will happen.
Note the mention of various
deposed object
as well.This happens every time I make a
local.alternative_domains
update - and of course I have made sure I start from clean environment to check there is nothing broken in the statefile.References
N/A