v1.0.11 - Error: Provider produced inconsistent result after apply

[psharm53]

Creating this issue in continuation with issues https://github.com/hashicorp/terraform/issues/29949

Based on feedback in issue https://github.com/hashicorp/terraform/issues/29949, we have worked on updating the terraform version to latest and using latest provider version as well as follows:

Terraform version - 1.0.11 AWS Provider Version - 3.66.0

The frequency of error "Error: Provider produced inconsistent result after apply" is much less compared to what we got with terraform version 0.12.31 but it is still there.

We are gtting this error for the following type of resources:

1. Route Tables
2. Subnets
3. Route Entries
4. Peering connection

NOTE - This error is not static but intermittent so it will require re-executions to be reproduced. However, it does happens multiple times when it starts coming. We got it 7 times in about 100 executions done over a period of 48 hours with a gap of at least 15 minutes between each execution. The error happened multiple times in a row when ever we faced it.

Procedure to reproduce issue Pre-requisites:

1. Use Region us-east-1 only
2. IAM role attached to EC2 instance on which terraform will be executed with account administrator access.
3. One VPC , it's CIDR . The VPC should have 2 private route table and one public route table . Keep their route table entires , VPC ID and CIDR handy.
4. Second VPC , it's CIDR . The VPC should have 2 private route table and one public route table . Keep their route table entires , VPC ID and CIDR handy.
5. One S3 bucket with folder names State-Files and FlowLogs in us-east-1

Steps to follow

1. Go to Folder Path Scripts/Terraform/Reinvent/StackLdapPeering

2. Edit File for following fields

   => Replace AWS Account ID ldap_vpc_access_role => Set to IAM role ARN from pre-requisutes 2. ldap_ops_vpc_id = Set to VPC ID from pre-requisutes 3. ldap_ops_vpc_cidr = Set to VPC CIDR from pre-requisutes 3. => Replace with one private Route Table ID from pre-requisutes 3. => Replace with second private Route Table ID from pre-requisutes 3. => Replace with public Route Table ID from pre-requisutes 3. ops_vpc_id = Set to VPC ID from pre-requisutes 4. ops_vpc_cidr = Set to VPC CIDR from pre-requisutes 4. => Replace with one private Route Table ID from pre-requisutes 4. => Replace with second private Route Table ID from pre-requisutes 4. => Replace the string with S3 bucket from pre-requisutes 5.

3. Execute Commands as follows: => export AWS_DEFAULT_REGION="us-east-1"

   => terraform init -reconfigure -input=false -backend-config="encrypt=true" -backend-config="max_retries=100" -backend-config="bucket=<Valid S3 Bucket Name from pre req 5>/State-Files/" -backend-config="region=us-east-1" -backend-config="key=CIOPS-test.tfstate"

   =>terraform get

   =>terraform plan -input=false -var-file=CIOPS.tfvars This will show 761 resources to be created

   => terraform apply -auto-approve -input=false -var-file=CIOPS.tfvars

terraform Configuration: Scripts.zip

[jbardin]

Hi @psharm53,

Thanks for filing the issue. The error Provider produced inconsistent result after apply is almost always caused by a provider outside of the control of Terraform. However, since the configuration here is not quite a minimal reproducible example, re-submitting this to the appropriate provider team is probably not going to result in a solution either.

Can you supply the exact error output from Terraform, so that we can narrow down the scope of where the problem might be? Once we know what resource is erroring out, and what that error is specifically, we may be able to pinpoint the dependency causing the error so that a more specific issue can be filed with the AWS provider.

Thanks!

[psharm53]

Error Messages for different types of resources that could be reproduced today in about 10 attempts :

1. Peering Connection:

│ Error: Provider produced inconsistent result after apply │ │ When applying changes to │ module.peering_extapp_data.aws_vpc_peering_connection.src_peering, provider │ "provider[\"registry.terraform.io/hashicorp/aws\"].stack" produced an │ unexpected new value: Root resource was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's │ own issue tracker. ╵

2. Route Table: │ Error: Provider produced inconsistent result after apply | |Provider produced inconsistent result after apply |When applying changes to |module.tms_vpc.aws_route_table.private_route_table["us-east-1a"], provider |"registry.terraform.io/-/aws" produced an unexpected new value for was |present, but now absent.

[jbardin]

These errors indicate the provider planned changes for certain resources, but applying those changes resulted in the resource returning a null value. This is often something like an uncaught error condition in the provider, but it also may be caused by eventual consistency issues in the remote service. The problem here however does need to be fixed in the provider itself, as Terraform cannot determine the cause of the missing resources.

Thanks!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.