Open reifnir opened 1 year ago
Voting for Prioritization
Volunteering to Work on This Issue
I don't know if this is related to my issue, but I'm seeing something similar where I create the stack set instance for the OU (ours has 15 accounts) and while it finishes in a few minutes, the console just keeps polling:
aws_cloudformation_stack_set_instance.<name>: Still creating... [39m41s elapsed]
Hi @reifnir,
I encountered the same issue when trying to deploy the stack set to a large number of accounts. To address this, I modified the max_retries parameter on the AWS provider, which overrides the default number of retries for AWS API calls.
FYI: https://registry.terraform.io/providers/hashicorp/aws/latest/docs#max_retries
Same issue here, after deployment to the bigger ou, terraform is retrying stackset instance creation that is causing error Status (FAILED) Status Reason: Attempt to perform create operation on already existing stack
. So first creation attempt is succeeding but in the meantime terraform starts 2nd one. As 2nd attempt fails it marks that instance as tained (which forces it's replacement during next plan). On certain size OU this is actually occurs more frequent - so only way for dealing with it currently is to untain failed instances manually.
Hi @wtc1230 I tried your solution, but this parameter seems to be ignored for me. Also I have found a possible root cause of the issues, which I think should be reverted or parametrizied.
For larger OUs operation still in progress is actually in progress, I have been investigating errors on both Terraform and AWS end. And this is what's happening for me.
Status (FAILED) Status Reason: Attempt to perform create operation on already existing stack
are thrown, because cloudformation in the 2nd operation tries to create the same resources one more time.End result:
Also interesting First successful operation have a different operation ID, in my example:
Status (FAILED) Status Reason: Attempt to perform create operation on already existing stack
So ideally terraform reporting should be bound to the specific operation ID that it creates, at the moment terraform is guessing.
Terraform Core Version
1.3.7
AWS Provider Version
4.48.0
Affected Resource(s)
Expected Behavior
While Terraform is waiting for a StackSet to be deployed, it should continue waiting without failing if it receives an
OperationInProgressException
error that shares the same operation id as the one it's trying to perform.In my example:
terraform-20230104195015363300000003
Actual Behavior
After almost an hour of the stacks being newly deployed, Terraform bombs with a 409 error stating
OperationInProgressException: Another Operation on StackSet {the arn} is in progress
.This occurred the first time after 50m41s and a second time at 57m20s. So, I'm able to repeat this.
Relevant Error/Panic Output Snippet
Terraform Configuration Files
Steps to Reproduce
Debug Output
Panic Output
No response
Important Factoids
From the trace log, it looks as though the provider is set to only query for progress 25 times, then retry the operation (which fails because it's still in progress).
The error occurs after around 70 accounts have been successfully applied or 50-60 minutes have passed.
When selecting the AWS OU for the stack set, I'd recommend having about 90 or 100 accounts. The error only shows itself once about 90% of the accounts in my 80 account org have been successfully deployed to. It takes a long time and you probably don't want to run this repeatedly because you're unsure if the deployment happened to complete in time.
References
No response
Would you like to implement a fix?
None