Missing cloudbuild services enable on bootstrap requires a 5 min wait state before a terraform re apply on 0-bootstrap - delete the existing cloudbuild worker pool to proceed #425
For some reason I missed one of the cloudbuild service enablements on the seed project - the last CB and testing ADO builds were fine - I had them on before
rerun tf apply before 5 min after cloudbuild services enable - and we get an eventually consistent error
3rd run will encounter an existing cloudbuild pool - delete it manually to proceed
421 421
Testing on existing org - no TEF run before - oldev
downgrade terraform and prep project
michael@cloudshell:~$ gcloud config set project tef-oldev
Updated property [core/project].
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ git checkout gh421-cb-enable
Branch 'gh421-cb-enable' set up to track remote branch 'gh421-cb-enable' from 'origin'.
Switched to a new branch 'gh421-cb-enable'
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ git status
On branch gh421-cb-enable
Your branch is up to date with 'origin/gh421-cb-enable'.
michael@cloudshell:~/tef-oldev/terraform1310 (tef-oldev)$ sudo cp terraform /usr/bin
michael@cloudshell:~/tef-oldev/terraform1310 (tef-oldev)$ cd ..
michael@cloudshell:~/tef-oldev (tef-oldev)$ ls
github terraform1310
michael@cloudshell:~/tef-oldev (tef-oldev)$ terraform --version
Terraform v1.3.10
on linux_amd64
Your version of Terraform is out of date! The latest version
is 1.8.2. You can update by downloading from https://www.terraform.io/downloads.html
michael@cloudshell:~/tef-oldev (tef-oldev)$ cd github/pbmm-on-gcp-onboarding/
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ gcloud services enable cloudresourcemanager.googleapis.com
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ gcloud services enable cloudbilling.googleapis.com
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ gcloud services enable iam.googleapis.com
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ gcloud services enable cloudkms.googleapis.com
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ gcloud services enable servicenetworking.googleapis.com
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding (tef-oldev)$ cd 0-bootstrap/
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev)$ mv terraform.example.tfvars terraform.tfvars
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev)$ mv ./cb.tf.example ./cb.tf
mv: cannot stat './cb.tf.example': No such file or directory
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev)$ terraform init
terraform init / plan / apply
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev)$ terraform init
Initializing modules...
Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for bootstrap_csr_repo...
terraform plan -input=false -out bootstrap.tfplan
Plan: 271 to add, 0 to change, 0 to destroy.
total time actual: 50 min to 1821
total time without errors: 23 min
1731 apply - 1 min and group eventually consistent error
3 - plan / apply 1743
Plan: 248 to add, 0 to change, 0 to destroy.
michael@cloudshell:~/tef-oldev/github/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev)$ terraform apply bootstrap.tfplan
3:30 min for project factory
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Still creating... [2m0s elapsed]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Creation complete after 2m2s [id=projects%2Fprj-b-cicd-khce%2Fglobal%2Fnetworks%2Fvpc-b-cbpools:servicenetworking.googleapis.com]
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Creating...
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Creating...
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Still creating... [10s elapsed]
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Creation complete after 11s [id=projects/prj-b-cicd-khce/global/networks/vpc-b-cbpools/networkPeerings/servicenetworking-googleapis-com]
╷
│ Error: Error creating WorkerPool: googleapi: Error 403: Cloud Build API has not been used in project tef-oldev before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=tef-oldev then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│ {
│ "@type": "type.googleapis.com/google.rpc.Help",
│ "links": [
│ {
│ "description": "Google developers console API activation",
│ "url": "https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=tef-oldev"
│ }
│ ]
│ },
│ {
│ "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│ "domain": "googleapis.com",
│ "metadata": {
│ "consumer": "projects/tef-oldev",
│ "service": "cloudbuild.googleapis.com"
│ },
│ "reason": "SERVICE_DISABLED"
│ }
│ ]
│
│ with module.tf_private_pool.google_cloudbuild_worker_pool.private_pool,
│ on modules/cb-private-pool/main.tf line 30, in resource "google_cloudbuild_worker_pool" "private_pool":
│ 30: resource "google_cloudbuild_worker_pool" "private_pool" {
│
╵
╷
│ Error: Error waiting to create Repository: Error waiting for Creating Repository: Error code 3, message: Request contains an invalid argument.
│
│ with module.tf_cloud_builder.google_artifact_registry_repository.tf-image-repo,
│ on .terraform/modules/tf_cloud_builder/modules/tf_cloudbuild_builder/gar.tf line 21, in resource "google_artifact_registry_repository" "tf-image-repo":
│ 21: resource "google_artifact_registry_repository" "tf-image-repo" {
│
╵
1759
In past deployments - missed adding cb service enablement
For some reason I missed one of the cloudbuild service enablements on the seed project - the last CB and testing ADO builds were fine - I had them on before
Add to existing
3 issue steps
421 421
Testing on existing org - no TEF run before - oldev
downgrade terraform and prep project
https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/gh421-cb-enable/0-bootstrap/README.md#deploying-with-cloud-build
delete groups
terraform init / plan / apply
total time actual: 50 min to 1821 total time without errors: 23 min
1731 apply - 1 min and group eventually consistent error
3 - plan / apply 1743
1759
In past deployments - missed adding cb service enablement
4 - plan / apply 1800
wait 5 min for cloudbuild enablement to propagate 1804-1809
5 - plan / apply 1809
1810
idempotent error - either full terraform destroy - or orphan/rename the resource
private-pool-rtbh northamerica-northeast1 e2-medium 100 GB vpc-b-cbpools
6 - plan/apply after cb worker pool delete - 1817
1819 50 min overall with 23 min terraform run time
bootstrap project
https://github.com/terraform-google-modules/terraform-example-foundation/issues/1139 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1140 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1142 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1143
See latest 3rd org testing 50 min for #421 in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/421#issuecomment-2093522430