search

terraform-google-modules / terraform-example-foundation

Shows how the CFT modules can be composed to build a secure cloud foundation
https://cloud.google.com/architecture/security-foundations
Apache License 2.0
1.2k stars 706 forks source link

Creation of prj-c-secrets fails with error "already exists" #1187

Closed mromascanu123 closed 4 months ago

mromascanu123 commented 4 months ago

TL;DR

Creation of the prj-c-secrets project (in 1-org package) fails with "already exists" issue. Not clear what is "already existing" because the project ID at stake ("prj-c-secrets-zfzs") does not exist gcloud projects list | grep prj-c-secret prj-c-secrets-1q2n prj-c-secrets 560518446142 prj-c-secrets-bb5k prj-c-secrets 536016672477 prj-c-secrets-cb7v prj-c-secrets 298129689217

Possibly a red-herring because just before this error there is a reference to a null project-id

module.org_secrets.module.project-factory.google_project.main: Creating... 2024-04-10T23:56:30.305Z [INFO] Starting apply for module.org_secrets.module.project-factory.google_project.main 2024-04-10T23:56:30.305Z [DEBUG] module.org_secrets.module.project-factory.google_project.main: applying the planned Create change ... etc...

2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Request Details: 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ REQUEST ]--------------------------------------- 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: GET /v1/projects/00000000000/services/serviceusage.googleapis.com?alt=json&prettyPrint=false HTTP/1.1 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Host: serviceusage.googleapis.com 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.6.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.84.0 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Goog-Api-Client: gl-go/1.19.9 gdcl/0.139.0 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Accept-Encoding: gzip 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.024Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ----------------------------------------------------- 2024-04-10T23:56:31.421Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Response Details: 2024-04-10T23:56:31.421Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ RESPONSE ]-------------------------------------- 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: HTTP/2.0 403 Forbidden 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Cache-Control: private 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json; charset=UTF-8 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Date: Wed, 10 Apr 2024 23:56:31 GMT 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server: ESF 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Origin 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: X-Origin 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Referer 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Content-Type-Options: nosniff 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Frame-Options: SAMEORIGIN 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Xss-Protection: 0 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "error": { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "code": 403, 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Project '00000000000' not found or permission denied.\nHelp Token: ARqICROPrg0klnTMfbX-zyzQRDdy-qJCED0hJszNt9xjA5AnmIeV1fZUgU4pPmWcOvcabDygKdUAHz2uf5PkqqZ9LmRMKnPPnnJZJkEyY5B5zk", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "errors": [ 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Project '00000000000' not found or permission denied.\nHelp Token: ARqICROPrg0klnTMfbX-zyzQRDdy-qJCED0hJszNt9xjA5AnmIeV1fZUgU4pPmWcOvcabDygKdUAHz2uf5PkqqZ9LmRMKnPPnnJZJkEyY5B5zk", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "global", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "forbidden" 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ], 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "status": "PERMISSION_DENIED", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "details": [ 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "@type": "type.googleapis.com/google.rpc.PreconditionFailure", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "violations": [ 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "googleapis.com", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "subject": "?error_code=210002&type=Project&resource_id=00000000000" 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ] 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }, 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "@type": "type.googleapis.com/google.rpc.ErrorInfo", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "RESOURCES_NOT_FOUND", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "serviceusage.googleapis.com", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "metadata": { 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "Project", 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "resource_id": "00000000000" 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ] 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.422Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }

Everything happens inside the attempted call to project-factory module in 1-org/envs/shared/projects.tf

/** Project for Org-wide Secrets *****/

module "org_secrets" { source = "terraform-google-modules/project-factory/google"

random_project_id = true random_project_id_length = 4 default_service_account = "deprivilege" name = "${local.project_prefix}-c-secrets" org_id = local.org_id billing_account = local.billing_account folder_id = google_folder.common.id activate_apis = ["logging.googleapis.com", "secretmanager.googleapis.com", "billingbudgets.googleapis.com"]

labels = { environment = "production" application_name = "org-secrets" billing_code = "1234" primary_contact = "example1" secondary_contact = "example2" business_code = "abcd" env_code = "p" } budget_alert_pubsub_topic = var.project_budget.org_secrets_alert_pubsub_topic budget_alert_spent_percents = var.project_budget.org_secrets_alert_spent_percents budget_amount = var.project_budget.org_secrets_budget_amount budget_alert_spend_basis = var.project_budget.org_secrets_budget_alert_spend_basis }

Terraform log attached

Expected behavior

As all the other projects have been created successfully in 1-org, all using project-factory (see attached screenshot) , so should be prj-c-secrets

Observed behavior

Creation fails and the NULL strange project ID might be related to issue 1186

Terraform Configuration

Nothing special - see attached screenshot

Terraform Version

$ terraform version
Terraform v1.6.0
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.8.0.

Additional information

Here is a log extract: 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ REQUEST ]--------------------------------------- 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: POST /v1/projects?alt=json&prettyPrint=false HTTP/1.1 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Host: cloudresourcemanager.googleapis.com 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: User-Agent: google-api-go-client/0.5 Terraform/1.6.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/4.84.0 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Length: 296 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Goog-Api-Client: gl-go/1.19.9 gdcl/0.139.0 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Accept-Encoding: gzip 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "labels": { 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "application_name": "org-secrets", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "billing_code": "1234", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "business_code": "abcd", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "env_code": "p", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "environment": "production", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "primary_contact": "example1", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "secondary_contact": "example2" 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }, 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "name": "prj-c-secrets", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "parent": { 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "id": "384831136297", 2024-04-10T23:56:31.423Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "type": "folder" 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: }, 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "projectId": "prj-c-secrets-zfzs" 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.424Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ----------------------------------------------------- 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: 2024/04/10 23:56:31 [DEBUG] Google API Response Details: 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ---[ RESPONSE ]-------------------------------------- 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: HTTP/2.0 409 Conflict 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Cache-Control: private 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Content-Type: application/json; charset=UTF-8 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Date: Wed, 10 Apr 2024 23:56:31 GMT 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server: ESF 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Server-Timing: gfet4t7; dur=195 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Origin 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: X-Origin 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: Vary: Referer 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Content-Type-Options: nosniff 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Frame-Options: SAMEORIGIN 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: X-Xss-Protection: 0 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5 2024-04-10T23:56:31.715Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "error": { 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "code": 409, 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Requested entity already exists", 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "errors": [ 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: { 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "message": "Requested entity already exists", 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "domain": "global", 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "reason": "alreadyExists" 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: ], 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: "status": "ALREADY_EXISTS" 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5: } 2024-04-10T23:56:31.716Z [DEBUG] provider.terraform-provider-google_v4.84.0_x5

terraform_debug.log

other_org_projects_created

terraform_tfvars

mromascanu123 commented 4 months ago

Here is the funny thing - to "fix" (kind of) the issue simply bumped random_project_id_length from 4 to 6 in module org_secrets in 1-org/envs/shared/projects.tf

Does not mean this is no longer a bug but I don't have an explanation - in the tfstate file the resource "random_string" "random_project_id_suffix" was existing and as a result there was no real randomization of the project_id between subsequent plan & apply operations. Once I bumped the hength here you go:

module.org_secrets.module.project-factory.random_string.random_project_id_suffix[0] must be replaced

-/+ resource "random_string" "random_project_id_suffix" { ~ id = "zfzs" -> (known after apply) ~ length = 4 -> 6 # forces replacement ~ result = "zfzs" -> (known after apply)

(9 unchanged attributes hidden)

}
eeaton commented 4 months ago

This is a terraform pain point when you're creating GCP project IDs with a random suffix. This happens when a terraform apply step fails midway through: your terraform state has generated the name of a project ID, but the project has not been created. This causes subsequent attempts to use terraform plan or terraform apply to fail because they're looking for a project ID that does not exist.

This error is not unique to the code in this repo, but it is an error you're likely to encounter because this repo creates a large number of projects, and a failed apply step due to project quota limitations often leads to this error.

error messages

You might also see errors like this (including these for others who might be searching for the specific error text):

converting TF resource to CAI: getting resource ancestry or parent failed: user does not have the correct permissions for projects/prj-c-secrets-zfzs. For more info: https://cloud.google.com/docs/terraform/policy-validation/troubleshooting#ProjectCallerForbidden]. Additional details: [terraform-validator-internal.git.corp.google.com/terraform-tools.git/cmd.Execute

Or errors like this:

Error: Error when reading or editing GCS service account not found: googleapi: Error 400: Unknown project id: 'prj-c-secrets-zfzs', invalid

fix

Unfortunately we have to unpick terraform state to resolve the error. The terraform destroy command doesn't help because it will fail with the same error trying to find the invalid project.

Your workaround to force replacement of the random suffix by changing it's length works because it gets rid of the generated suffix in state.

I prefer to use the Terraform cli to remove the suffix from state, then run again

terraform plan 
# returns error like Error: Error when reading or editing GCS service account not found: googleapi: Error 400: Unknown project id: 'prj-c-secrets-zfzs', invalid

terraform state list
# look for the state object ending in random_project_id_suffix[0]. Some steps have a few of these, choose the one aligned to the module for the project in the error. 

terraform state show module.env.module.base_shared_vpc_project.module.project.module.project-factory.random_string.random_project_id_suffix[0]
# check that you choose the correct suffix as the project in the error, this should contain id = "zfzs" (or the same value as your suffix) 

# After you have verified this is the correct ID, remove it.
terraform state rm module.env.module.base_shared_vpc_project.module.project.module.project-factory.random_string.random_project_id_suffix[0]

terraform plan
# it should work now

I'll create an issue to improve readme guidance on fixing this error state because it's non-intuitive to workaround.

eeaton commented 4 months ago

I will track this in #1195

fmichaelobrien commented 4 months ago

Sounds good will track 1195 in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/380