Closed syphernl closed 2 years ago
This is not only "requires 2 apply", upgrade from 0.39.0
to 0.40.0
causes downtime because of the reason described.
Did both of you do terraform state mv
for existing elasticache groups or were elasticache group modules upgraded w/o state moves ?
Or was the above applied on a brand new elasticache ?
Did both of you do
terraform state mv
for existing elasticache groups or were elasticache group modules upgraded w/o state moves ?
I did not, just terraform apply
. Tomorrow I can test a production-like upgrade procedure and check whether we can achieve that without downtime if it helps?
@nitrocode I did not do a terraform state mv
either for the Elasticache groups.
@syphernl Can you give us more details about why the upgrade to v0.40.0 causes 2 runs? I am curious if it is because of a propagation delay or a consequence of apply_immediately = false
or something else.
What version of Terraform were you using?
As mentioned elsewhere, we have marked v0.40.0 pre-release and recommend using 0.39.0 for now. However, this bug could persist if it is because of something we overlooked, which appears to be the case, so we appreciate your help in understanding this issue better.
@Nuru At the time we were using Terraform 1.0.0.
In order to get this change to apply I had to login to the console, go to the Redis Cluster and change the security groups by hand to something else. Otherwise Terraform could not remove the old Security Group and would eventually fail. This change was executed before running this state change.
I don't know why it didn't apply the new security group (and remove the one's that shouldn't be there) in the first run.
@syphernl Terraform is not aware that to remove a security group, you have to remove all its associations first, so I would expect its plan to be
I would expect step 1 to fail because the security group was still associated, step 2 to succeed, and step 3 to fail because step 1 failed. Then you manually disassociated the old SG, so on the next run Terraform could succeed at step 1, skip step 2 because that had already succeeded, and move on to step 3. Do I have that right?
@Nuru Sounds logical. But the whole process only failed for the first environment we updated. For all subsequent environments we first manually disassociated the old (to be removed) security group from the Elasticache Cluster, temporarily assigned the VPC SG to prevent this behavior.
I have looked into this extensively and believe it is a bug in the AWS Terraform provider. I have not been able to find a workaround. Please upvote https://github.com/hashicorp/terraform-provider-aws/issues/20139 and if you have any additional information, please share it there.
Opened Terraform issue https://github.com/hashicorp/terraform/issues/29799 to track the problem. Will publish changes that work around the issue.
Describe the Bug
The v0.40.0 release requires two plan/apply stages to be executed to be fully useful.
In the first run, the security group is created:
This security group is however not applied until the second run:
Expected Behavior
Security Group to be created & assigned in the same run.
Configuration
Environment (please complete the following information):
Anything that will help us triage the bug will help. Here are some ideas:
Additional Context
Add any other context about the problem here.