Open bgalkows opened 6 months ago
pebo
commented
6 months ago This issue occurs several times a day also with version 5.x. The problem might be that the providers is not using long enough timeouts for the 404 retries to deal with eventual consistency while creating service accounts. Adding a sleep after resource creation won't help as the provider fails during creation .
Terraform 1.8.3
provider "registry.terraform.io/hashicorp/google" {
version = "5.25.0"
dahel8
commented
6 months ago To provide some further information on the issue
Terraform 1.8.1
Google provider 5.25.0Some debug logs from when this occurs
ggtisc
commented
6 months ago Unable to reproduce, many users are reporting the same issue so I'm directly forwarding this
TJM
commented
6 months ago It seems like it is easier to reproduce this problem in the morning (US timezones), like from 9a-12p MDT. It seems like, maybe the provider tries to create the resource, then lookup some additional information or something, the way the error message is written. It doesn't exist for the info query, so it fails. It would be nice if it would create it as tainted at the very least in the state file so that a retry of the pipeline would work.
We are using provider version v5.27.0 and TF 1.8.3.
michaellzc
commented
5 months ago It seems like it is easier to reproduce this problem in the morning (US timezones), like from 9a-12p MDT.
Confirmed we've been observing similar problems for the past months at least 5 times. Coincidentally, it happened to be within this time range for the most recent instance.
TPG v5.30.0
Terraform v1.5.6provider "google" {
request_timeout = "120s"
}
resource "google_service_account" "<>" {
// ...
}
oavner
commented
4 months ago google-beta provider v5.38.0 waits 10 sec prior to reading the created service account (vs 5 sec at the google provider) and solve the inconsistency issue (most of the time).
the google IAM api is eventually consistent hence reading the service account metadata right after writing a new service account might not result with the latest mutation of the resource.
https://google.aip.dev/121#strong-consistency
https://cloud.google.com/iam/docs/overview#consistency
this behavior violates the Terraform Resource Instance Change Lifecycle and that's pretty much the only solution currently for applying changes to eventually consistent API's. https://github.com/hashicorp/terraform/blob/main/docs/resource-instance-change-lifecycle.md
upgrade ur version to the latest one and use google-beta provider to provision service account's and bind them to IAM policies, this way the provider would wait 10 sec before reading its state and writing it to state file.
Ameausoone
commented
3 months ago This bug has been "dodged" in the release v5.32.0 of the "regular" provider, with an unexpected time.Sleep(10 * time.Second) :sweat_smile:.
iam: added a 10 second sleep when creating a 'google_service_account' resource to reduce eventual consistency errors(https://github.com/hashicorp/terraform-provider-google/pull/18261)
"Fixed" by https://github.com/hashicorp/terraform-provider-google/pull/18261
Exagone313
commented
3 months ago Increasing the timeout is not a real fix. Tomorrow you'll have to increase it again.
https://www.commitstrip.com/en/2017/05/22/a-story-about-callbacks/
Exagone313
commented
3 months ago To make things more clear, we are hit by this issue on other resources too, such as a google_sourcerepo_repository, where no time.Sleep is done in the implementation. Most of the API are eventually consistent. But it is difficult to deal with this with Terraform.
Please do not close this issue until a more long term solution is implemented for all resources.
Community Note
Terraform Version & Provider Version(s)
Terraform v1.5.7 on Ubuntu
Affected Resource(s)
google_service_account
Terraform Configuration
Debug Output
│ Error: Provider produced inconsistent result after apply │ │ When applying changes to │ module.[...].google_service_account.service_accounts[...], │ provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an │ unexpected new value: Root resource was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's │ own issue tracker.
Expected Behavior
A service account should have been actuated normally.
Actual Behavior
After an apply error, the service account was created successfully in GCP but never acknowledged in Terraform state. An import block was needed to resolve the situation.
Steps to reproduce
terraform apply, creating a google_service_account resourceImportant Factoids
No response
References
No response
b/341145887