The project-factory can use a service account with a misconfigured host project, which leads to a deadlocked Terraform configuration with confusing errors.
Description
The project factory requires that the host project associated with an instantiating service account has certain APIs enabled, but the project factory does not verify that these APIs are enabled before generating a new project. This can cause the project factory to partially generate a new project and then fail, and subsequent calls to terraform apply will fail with a number of related but unhelpful errors.
When terraform apply is run with the misconfigured environment, it will create a new project, attempt to associate a billing account with that project, and fail since cloudbilling.googleapis.com is disabled. Following runs of terraform apply will see that the google_project was created and then attempt to look up the default compute service account with the google_service_account data source. This will fail because the compute.googleapis.com API is not enabled, but this hides the more interesting errors - that the service project doesn't have a billing account enabled, and that the host project doesn't have the Cloud Billing API enabled and thus can't remedy the issue through Terraform.
Expected behavior
When the project-factory runs with a service account whose host project doesn't have the right APIs enabled, it terminates before generating resources.
Actual behavior
First invocation of terraform apply:
module.project-factory.google_project.project: Still creating... (10s elapsed)
Error: Error applying plan:
1 error(s) occurred:
* module.project-factory.google_project.project: 1 error(s) occurred:
* google_project.project: Error setting billing account "<BILLING ACCOUNT>" for project "projects/tftest-1-4901": googleapi: Error 403: Cloud Billing API has not been used in project <HOST NUMBER> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=<HOST NUMBER> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
Note: that the error generated references the host project number on the first failed run.
Running terraform plan or terraform apply
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
random_id.random_project_id_suffix: Refreshing state... (ID: SQE)
data.google_organization.org: Refreshing state...
google_project.project: Refreshing state... (ID: tftest-1-4901)
data.null_data_source.data_given_group_email: Refreshing state...
data.null_data_source.data_final_group_email: Refreshing state...
data.null_data_source.data_group_email_format: Refreshing state...
data.google_compute_default_service_account.default: Refreshing state...
Error: Error refreshing state: 1 error(s) occurred:
* module.project-factory.data.google_compute_default_service_account.default: 1 error(s) occurred:
* module.project-factory.data.google_compute_default_service_account.default: data.google_compute_default_service_account.default: Error reading GCE service account not found: googleapi: Error 403: Project <SERVICE NUMBER> is not found and cannot be used for API calls. If it is recently created, enable Compute Engine API by visiting https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=<SERVICE NUMBER> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured
Note: in subsequent runs the error references the service project.
Steps to reproduce
note: these steps are approximate and should be checked to make sure that this reproduces correctly.
Create a new host project, but don't enable any APIs.
Create a service account within that project with helpers/setup-sa.sh.
This creates a service account with the correct rights to create projects, but without the APIs enabled that it needs to work
Create a terraform configuration with the project factory.
Run terraform apply; the command should fail.
Verify that terraform can no longer generate a plan with terraform plan
Workarounds
If a terraform configuration is deadlocked in the describe manner, the correct APIs can be enabled retroactively and terraform will be able to run.
Enable cloudbilling.googleapis.com on the host project through the Google Cloud console.
Enable compute.googleapis.com on the service project through the Google Cloud console.
Using the console to enable this API will automatically prompt the user to add a billing account.
(Maybe) enable serviceusage.googleapis.com on the host project?
adrienthebo
commented
5 years ago
Summary
The project-factory can use a service account with a misconfigured host project, which leads to a deadlocked Terraform configuration with confusing errors.
Description
The project factory requires that the host project associated with an instantiating service account has certain APIs enabled, but the project factory does not verify that these APIs are enabled before generating a new project. This can cause the project factory to partially generate a new project and then fail, and subsequent calls to
terraform applywill fail with a number of related but unhelpful errors.When
terraform applyis run with the misconfigured environment, it will create a new project, attempt to associate a billing account with that project, and fail sincecloudbilling.googleapis.comis disabled. Following runs ofterraform applywill see that the google_project was created and then attempt to look up the default compute service account with thegoogle_service_accountdata source. This will fail because thecompute.googleapis.comAPI is not enabled, but this hides the more interesting errors - that the service project doesn't have a billing account enabled, and that the host project doesn't have the Cloud Billing API enabled and thus can't remedy the issue through Terraform.Expected behavior
When the project-factory runs with a service account whose host project doesn't have the right APIs enabled, it terminates before generating resources.
Actual behavior
First invocation of
terraform apply:Note: that the error generated references the host project number on the first failed run.
Running
terraform planorterraform applyNote: in subsequent runs the error references the service project.
Steps to reproduce
note: these steps are approximate and should be checked to make sure that this reproduces correctly.
helpers/setup-sa.sh.terraform apply; the command should fail.terraform planWorkarounds
If a terraform configuration is deadlocked in the describe manner, the correct APIs can be enabled retroactively and terraform will be able to run.
cloudbilling.googleapis.comon the host project through the Google Cloud console.compute.googleapis.comon the service project through the Google Cloud console.serviceusage.googleapis.comon the host project?