Closed mromascanu123 closed 4 months ago
Here is the funny thing - to "fix" (kind of) the issue simply bumped random_project_id_length from 4 to 6 in module org_secrets in 1-org/envs/shared/projects.tf
Does not mean this is no longer a bug but I don't have an explanation - in the tfstate file the resource "random_string" "random_project_id_suffix" was existing and as a result there was no real randomization of the project_id between subsequent plan & apply operations. Once I bumped the hength here you go:
-/+ resource "random_string" "random_project_id_suffix" { ~ id = "zfzs" -> (known after apply) ~ length = 4 -> 6 # forces replacement ~ result = "zfzs" -> (known after apply)
}
This is a terraform pain point when you're creating GCP project IDs with a random suffix. This happens when a terraform apply
step fails midway through: your terraform state has generated the name of a project ID, but the project has not been created. This causes subsequent attempts to use terraform plan
or terraform apply
to fail because they're looking for a project ID that does not exist.
This error is not unique to the code in this repo, but it is an error you're likely to encounter because this repo creates a large number of projects, and a failed apply step due to project quota limitations often leads to this error.
You might also see errors like this (including these for others who might be searching for the specific error text):
converting TF resource to CAI: getting resource ancestry or parent failed: user does not have the correct permissions for projects/prj-c-secrets-zfzs. For more info: https://cloud.google.com/docs/terraform/policy-validation/troubleshooting#ProjectCallerForbidden]. Additional details: [terraform-validator-internal.git.corp.google.com/terraform-tools.git/cmd.Execute
Or errors like this:
Error: Error when reading or editing GCS service account not found: googleapi: Error 400: Unknown project id: 'prj-c-secrets-zfzs', invalid
Unfortunately we have to unpick terraform state to resolve the error. The terraform destroy
command doesn't help because it will fail with the same error trying to find the invalid project.
Your workaround to force replacement of the random suffix by changing it's length works because it gets rid of the generated suffix in state.
I prefer to use the Terraform cli to remove the suffix from state, then run again
terraform plan
# returns error like Error: Error when reading or editing GCS service account not found: googleapi: Error 400: Unknown project id: 'prj-c-secrets-zfzs', invalid
terraform state list
# look for the state object ending in random_project_id_suffix[0]. Some steps have a few of these, choose the one aligned to the module for the project in the error.
terraform state show module.env.module.base_shared_vpc_project.module.project.module.project-factory.random_string.random_project_id_suffix[0]
# check that you choose the correct suffix as the project in the error, this should contain id = "zfzs" (or the same value as your suffix)
# After you have verified this is the correct ID, remove it.
terraform state rm module.env.module.base_shared_vpc_project.module.project.module.project-factory.random_string.random_project_id_suffix[0]
terraform plan
# it should work now
I'll create an issue to improve readme guidance on fixing this error state because it's non-intuitive to workaround.
I will track this in #1195
Sounds good will track 1195 in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/380
TL;DR
Creation of the prj-c-secrets project (in 1-org package) fails with "already exists" issue. Not clear what is "already existing" because the project ID at stake ("prj-c-secrets-zfzs") does not exist gcloud projects list | grep prj-c-secret prj-c-secrets-1q2n prj-c-secrets 560518446142 prj-c-secrets-bb5k prj-c-secrets 536016672477 prj-c-secrets-cb7v prj-c-secrets 298129689217
Possibly a red-herring because just before this error there is a reference to a null project-id
module.org_secrets.module.project-factory.google_project.main: Creating... 2024-04-10T23:56:30.305Z [INFO] Starting apply for module.org_secrets.module.project-factory.google_project.main 2024-04-10T23:56:30.305Z [DEBUG] module.org_secrets.module.project-factory.google_project.main: applying the planned Create change ... etc...
2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Request Details: 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ REQUEST ]--------------------------------------- 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: GET /v1/projects/00000000000/services/serviceusage.googleapis.com?alt=json&prettyPrint=false HTTP/1.1 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Host: serviceusage.googleapis.com 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.6.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.84.0 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Goog-Api-Client: gl-go/1.19.9 gdcl/0.139.0 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Accept-Encoding: gzip 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ----------------------------------------------------- 2024-04-10T23:56:31.421Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Response Details: 2024-04-10T23:56:31.421Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ RESPONSE ]-------------------------------------- 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: HTTP/2.0 403 Forbidden 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Cache-Control: private 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json; charset=UTF-8 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Date: Wed, 10 Apr 2024 23:56:31 GMT 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server: ESF 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Origin 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: X-Origin 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Referer 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Content-Type-Options: nosniff 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Frame-Options: SAMEORIGIN 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Xss-Protection: 0 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "error": { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "code": 403, 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Project '00000000000' not found or permission denied.\nHelp Token: ARqICROPrg0klnTMfbX-zyzQRDdy-qJCED0hJszNt9xjA5AnmIeV1fZUgU4pPmWcOvcabDygKdUAHz2uf5PkqqZ9LmRMKnPPnnJZJkEyY5B5zk", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "errors": [ 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Project '00000000000' not found or permission denied.\nHelp Token: ARqICROPrg0klnTMfbX-zyzQRDdy-qJCED0hJszNt9xjA5AnmIeV1fZUgU4pPmWcOvcabDygKdUAHz2uf5PkqqZ9LmRMKnPPnnJZJkEyY5B5zk", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "global", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "forbidden" 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ], 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "status": "PERMISSION_DENIED", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "details": [ 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "@type": "type.googleapis.com/google.rpc.PreconditionFailure", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "violations": [ 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "googleapis.com", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "subject": "?error_code=210002&type=Project&resource_id=00000000000" 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ] 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }, 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "@type": "type.googleapis.com/google.rpc.ErrorInfo", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "RESOURCES_NOT_FOUND", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "serviceusage.googleapis.com", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "metadata": { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "Project", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "resource_id": "00000000000" 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ] 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }
Everything happens inside the attempted call to project-factory module in 1-org/envs/shared/projects.tf
/** Project for Org-wide Secrets *****/
module "org_secrets" { source = "terraform-google-modules/project-factory/google"
random_project_id = true random_project_id_length = 4 default_service_account = "deprivilege" name = "${local.project_prefix}-c-secrets" org_id = local.org_id billing_account = local.billing_account folder_id = google_folder.common.id activate_apis = ["logging.googleapis.com", "secretmanager.googleapis.com", "billingbudgets.googleapis.com"]
labels = { environment = "production" application_name = "org-secrets" billing_code = "1234" primary_contact = "example1" secondary_contact = "example2" business_code = "abcd" env_code = "p" } budget_alert_pubsub_topic = var.project_budget.org_secrets_alert_pubsub_topic budget_alert_spent_percents = var.project_budget.org_secrets_alert_spent_percents budget_amount = var.project_budget.org_secrets_budget_amount budget_alert_spend_basis = var.project_budget.org_secrets_budget_alert_spend_basis }
Terraform log attached
Expected behavior
As all the other projects have been created successfully in 1-org, all using project-factory (see attached screenshot) , so should be prj-c-secrets
Observed behavior
Creation fails and the NULL strange project ID might be related to issue 1186
Terraform Configuration
Terraform Version
Additional information
Here is a log extract: 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ REQUEST ]--------------------------------------- 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: POST /v1/projects?alt=json&prettyPrint=false HTTP/1.1 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Host: cloudresourcemanager.googleapis.com 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.6.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.84.0 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Length: 296 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Goog-Api-Client: gl-go/1.19.9 gdcl/0.139.0 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Accept-Encoding: gzip 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "labels": { 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "application_name": "org-secrets", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "billing_code": "1234", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "business_code": "abcd", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "env_code": "p", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "environment": "production", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "primary_contact": "example1", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "secondary_contact": "example2" 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }, 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "name": "prj-c-secrets", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "parent": { 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "id": "384831136297", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "folder" 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }, 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "projectId": "prj-c-secrets-zfzs" 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ----------------------------------------------------- 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Response Details: 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ RESPONSE ]-------------------------------------- 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: HTTP/2.0 409 Conflict 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Cache-Control: private 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json; charset=UTF-8 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Date: Wed, 10 Apr 2024 23:56:31 GMT 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server: ESF 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server-Timing: gfet4t7; dur=195 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Origin 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: X-Origin 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Referer 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Content-Type-Options: nosniff 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Frame-Options: SAMEORIGIN 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Xss-Protection: 0 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "error": { 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "code": 409, 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Requested entity already exists", 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "errors": [ 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Requested entity already exists", 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "global", 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "alreadyExists" 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ], 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "status": "ALREADY_EXISTS" 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5
terraform_debug.log