fmichaelobrien commented 1 year ago

20240307: update Got past 0-bootstrap - working 1-org below

1133

see https://github.com/terraform-google-modules/terraform-example-foundation/issues/940 https://github.com/terraform-google-modules/terraform-example-foundation/issues/967 https://github.com/terraform-google-modules/terraform-example-foundation/issues/966 https://github.com/terraform-google-modules/terraform-example-foundation/issues/965 https://github.com/terraform-google-modules/terraform-example-foundation/issues/964

assigned fmichaelobrien branch/fork in https://github.com/CloudLandingZone/terraform-example-foundation

TL;DR

This issue will document deploying the existing TEF and any additional/modified artifacts around adding a PBMM overlay.

Parts of this repo https://github.com/terraform-google-modules/terraform-example-foundation were copied around mid 2021.

Having 2 separate repos causes obvious issues around rebasing, parallel development, library updates... An effort to backport/rebase changes specific to the PBMM variant are underway so that we have 1 Terraform based LZ with optional PBMM modules.

see KCC LZ https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/298 see TF LZ https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/243

sced shadow of 298 https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/249

Work Items

WI0: deploy the TEF (Selectively comment business user 2 projects - TB under quota)

WI1: Diff between TEF and PBMM repos - (code/deployment) using both deployments

WI2: Selective rebase

Terraform Resources

No response

Detailed design

No response

Additional information

Also rebase with/to https://github.com/GoogleCloudPlatform/blueprints/tree/main/catalog/landing-zone

No response

obriensystems commented 1 year ago

Clean org prereq: cloud-identity (no workspaces this run) 3rd party subdomain org: pbmm-landing-systems boot: landingzone-tef/lz-tef-tlz project quota increase: from default 15 to 40 billing/project quota increase: from default 5 to 30 IAM permissions added to admin on top of Organization Administrator

Compute Shared VPC Admin
Folder Admin
Folder Creator
Organization Policy Administrator
Project Billing Manager
Project Creator
Project Deleter
Project IAM Admin
Service Account Token Creator

CLI tracking using 20230506 version under release 3.0.0 (Dec 2022)

create project example

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud projects create $CC_PROJECT_ID --name="${CC_PROJECTID}" --set-as-default Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/bootstrap-plz]. Waiting for [operations/cp.5122135151997130492] to finish...done.
Enabling service [cloudapis.googleapis.com] on project [bootstrap-plz]... Operation "operations/acat.p2-208036100419-ba66496b-4a17-45fe-806c-c7f154c9bca2" finished successfully. Updated property [core/project] to [bootstrap-plz]. root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ export BILLING_ID=$(gcloud alpha billing projects describe $CC_PROJECTID '--format=value(billingAccountName)' | sed 's/.*\///') root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ echo $BILLING_ID

notice billing is not set - even though we are either a "Billing Account Administrator" or "Billing Account User" - we get the billing ID from another project with billing set - or we set the BILLINGID variable manually root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ export BILLINGID=$(gcloud alpha billing projects describe lz-tef-plz '--format=value(billingAccountName)' | sed 's/.*\///') root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ echo $BILLINGID 01906F-.....-859F42 root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ export ORGANIZATION_ID=$(gcloud projects get-ancestors $CC_PROJECTID --format='get(id)' | tail -1) root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ echo $ORGANIZATIONID 93...09 root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ gcloud beta billing projects link ${CC_PROJECT_ID} --billing-account ${BILLING_ID} billingAccountName: billingAccounts/01906F-...-859F42 billingEnabled: true name: projects/bootstrap-plz/billingInfo projectId: bootstrap-plz
now recheck the billing id on the new project and we are good to use it root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ export BILLING_ID=$(gcloud alpha billing projects describe $CC_PROJECTID '--format=value(billingAccountName)' | sed 's/.*\///') root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$ echo $BILLINGID 01906F-...-859F42 root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (bootstrap-plz)$

root@cloudshell:~$ mkdir lz-tef-plz root@cloudshell:~$ cd lz-tef-plz/ root@cloudshell:~$ gcloud config set project lz-tef-plz root@cloudshell:~/lz-tef-plz (lz-tef-plz)$ mkdir CloudLandingZone root@cloudshell:~/lz-tef-plz (lz-tef-plz)$ cd CloudLandingZone/ root@cloudshell:~/lz-tef-plz/CloudLandingZone (lz-tef-plz)$ git clone https://github.com/CloudLandingZone/terraform-example-foundation.git

root_@cloudshell:~/lz-tef-plz/CloudLandingZone (lz-tef-plz)$ terraform version Terraform v1.4.6 on linuxamd64 root@cloudshell:~/lz-tef-plz/CloudLandingZone (lz-tef-plz)$ gcloud version Google Cloud SDK 428.0.0 root_@cloudshell:~/lz-tef-plz/CloudLandingZone (lz-tef-plz)$ git version git version 2.30.2

all good (above 1.3.0, 393, 2.28)

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation (lz-tef-plz)$ git checkout pbmm

prepare vars - uncomment automatic group creation and parent folder in terraform-example.tfvars first

root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation (lz-tef-plz)$ cd 0-bootstrap/ root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ cp terraform.example.tfvars terraform.tfvars

fill vars

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ export PROJECTID=lz-tef-plz root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ export ORGANIZATION_ID=$(gcloud projects get-ancestors $PROJECTID --format='get(id)' | tail -1) root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ echo $ORGANIZATIONID 93....309 root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ export BILLING_ID=$(gcloud alpha billing projects describe $PROJECTID '--format=value(billingAccountName)' | sed 's/.*\///') root@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ echo $BILLING_ID 01906F-....1-859F42


terraform.tfvars

https://cloud.google.com/architecture/security-foundations/authentication-authorization#users_and_groups

org_id = "93..09" # format "000000000000" billing_account = "019...F42" # format "000000-000000-000000" group_org_admins = "group_org_admins@pbmm..." group_billing_admins = "group_billing_admins@pbmm..." default_region = "northamerica-northeast1" parent_folder = "24...6" groups = { create_groups = true, billing_project = "billing-project", required_groups = { group_org_admins = "group_org_admins@pbmm..." group_billing_admins = "group_billing_admins@pbmm..." billing_data_users = "billing_data_users@pbmm..." audit_data_users = "audit_data_users@pbmm..." monitoring_workspace_users = "monitoring_workspace_users@pbmm...." }, optional_groups = { gcp_platform_viewer = "gcp_platform_viewer@pbmm...." gcp_security_reviewer = "gcp_security_reviewer@pbmm...." gcp_network_viewer = "gcp_network_viewer@pbmm...." gcp_scc_admin = "gcp_scc_admin@pbmm...." gcp_global_secrets_admin = "gcp_global_secrets_admin@pbmm...." gcp_audit_viewer = "gcp_audit_viewer@pbmm...." } }


validate

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ export EMAIL=$(gcloud config list --format json|jq .core.account | sed 's/"//g')

../scripts/validate-requirements.sh -o $ORGANIZATION_ID -b $BILLING_ID -u $EMAIL

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ ../scripts/validate-requirements.sh -o $ORGANIZATION_ID -b $BILLING_ID -u $EMAIL Validating required utility tools... Validating Terraform installation... Validating Google Cloud SDK installation... Validating Git installation... git default branch must be configured as main. See the instructions at https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting . Validating local gcloud configuration... Validating roles assignment for current end user credential... Validating 0-bootstrap configuration... ....................................... Validation failed! Errors found: git default branch must be configured as main.

Relaxing branch requirement...

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ ../scripts/validate-requirements.sh -o $ORGANIZATION_ID -b $BILLING_ID -u $EMAIL Validating required utility tools... Validating Terraform installation... Validating Google Cloud SDK installation... Validating Git installation... git default branch must be configured as main. See the instructions at https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting . Validating local gcloud configuration... Validating roles assignment for current end user credential... Validating 0-bootstrap configuration... ....................................... Validation successful! No errors found.


continue with
https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md#deploying-with-cloud-build

terraform init

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform init

Initializing the backend... Initializing modules... Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.1.2 for bootstrap_csr_repo...

bootstrap_csr_repo in .terraform/modules/bootstrap_csr_repo
bootstrap_projects_remove_editor in modules/parent-iam-remove-role Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.1.2 for build_terraform_image...
build_terraform_image in .terraform/modules/build_terraform_image
cicd_project_iam_member in modules/parent-iam-member Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 4.0.0 for gcp_projects_state_bucket...
gcp_projects_state_bucket in .terraform/modules/gcp_projects_state_bucket/modules/simple_bucket Downloading registry.terraform.io/terraform-google-modules/group/google 0.5.0 for optional_group...
optional_group in .terraform/modules/optional_group
org_iam_member in modules/parent-iam-member
parent_iam_member in modules/parent-iam-member Downloading registry.terraform.io/terraform-google-modules/group/google 0.5.0 for required_group...
required_group in .terraform/modules/required_group Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 6.4.0 for seed_bootstrap...
seed_bootstrap in .terraform/modules/seed_bootstrap Downloading registry.terraform.io/terraform-google-modules/org-policy/google 5.2.2 for seed_bootstrap.enable_cross_project_service_account_usage...
seed_bootstrap.enable_cross_project_service_account_usage in .terraform/modules/seed_bootstrap.enable_cross_project_service_account_usage Downloading registry.terraform.io/terraform-google-modules/kms/google 2.2.2 for seed_bootstrap.kms...
seed_bootstrap.kms in .terraform/modules/seed_bootstrap.kms Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.2.0 for seed_bootstrap.seed_project...
seed_bootstrap.seed_project in .terraform/modules/seed_bootstrap.seed_project
seed_bootstrap.seed_project.budget in .terraform/modules/seed_bootstrap.seed_project/modules/budget
seed_bootstrap.seed_project.essential_contacts in .terraform/modules/seed_bootstrap.seed_project/modules/essential_contacts
seed_bootstrap.seed_project.gsuite_group in .terraform/modules/seed_bootstrap.seed_project/modules/gsuite_group
seed_bootstrap.seed_project.project-factory in .terraform/modules/seed_bootstrap.seed_project/modules/core_project_factory
seed_bootstrap.seed_project.project-factory.project_services in .terraform/modules/seed_bootstrap.seed_project/modules/project_services
seed_bootstrap.seed_project.quotas in .terraform/modules/seed_bootstrap.seed_project/modules/quota_manager
seed_bootstrap.seed_project.shared_vpc_access in .terraform/modules/seed_bootstrap.seed_project/modules/shared_vpc_access
seed_project_iam_member in modules/parent-iam-member Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 6.4.0 for tf_cloud_builder...
tf_cloud_builder in .terraform/modules/tf_cloud_builder/modules/tf_cloudbuild_builder Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 3.4.1 for tf_cloud_builder.bucket...
tf_cloud_builder.bucket in .terraform/modules/tf_cloud_builder.bucket/modules/simple_bucket
tf_private_pool in modules/cb-private-pool Downloading registry.terraform.io/terraform-google-modules/network/google 5.2.0 for tf_private_pool.firewall_rules...
tf_private_pool.firewall_rules in .terraform/modules/tf_private_pool.firewall_rules/modules/firewall-rules Downloading registry.terraform.io/terraform-google-modules/network/google 5.2.0 for tf_private_pool.peered_network...
tf_private_pool.peered_network in .terraform/modules/tf_private_pool.peered_network
tf_private_pool.peered_network.firewall_rules in .terraform/modules/tf_private_pool.peered_network/modules/firewall-rules
tf_private_pool.peered_network.routes in .terraform/modules/tf_private_pool.peered_network/modules/routes
tf_private_pool.peered_network.subnets in .terraform/modules/tf_private_pool.peered_network/modules/subnets
tf_private_pool.peered_network.vpc in .terraform/modules/tf_private_pool.peered_network/modules/vpc Downloading registry.terraform.io/terraform-google-modules/vpn/google 2.3.1 for tf_private_pool.vpn_ha_cb_to_onprem...
tf_private_pool.vpn_ha_cb_to_onprem in .terraform/modules/tf_private_pool.vpn_ha_cb_to_onprem/modules/vpn_ha Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 6.4.0 for tf_source...
tf_source in .terraform/modules/tf_source/modules/tf_cloudbuild_source Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 3.4.1 for tf_source.cloudbuild_bucket...
tf_source.cloudbuild_bucket in .terraform/modules/tf_source.cloudbuild_bucket/modules/simple_bucket Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.2.0 for tf_source.cloudbuild_project...
tf_source.cloudbuild_project in .terraform/modules/tf_source.cloudbuild_project
tf_source.cloudbuild_project.gsuite_group in .terraform/modules/tf_source.cloudbuild_project/modules/gsuite_group
tf_source.cloudbuild_project.project-factory in .terraform/modules/tf_source.cloudbuild_project/modules/core_project_factory
tf_source.cloudbuild_project.project-factory.project_services in .terraform/modules/tf_source.cloudbuild_project/modules/project_services
tf_source.cloudbuild_project.quotas in .terraform/modules/tf_source.cloudbuild_project/modules/quota_manager
tf_source.cloudbuild_project.shared_vpc_access in .terraform/modules/tf_source.cloudbuild_project/modules/shared_vpc_access Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 6.4.0 for tf_workspace...
tf_workspace in .terraform/modules/tf_workspace/modules/tf_cloudbuild_workspace Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 3.4.1 for tf_workspace.artifacts_bucket...
tf_workspace.artifacts_bucket in .terraform/modules/tf_workspace.artifacts_bucket/modules/simple_bucket Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 3.4.1 for tf_workspace.log_bucket...
tf_workspace.log_bucket in .terraform/modules/tf_workspace.log_bucket/modules/simple_bucket Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 3.4.1 for tf_workspace.state_bucket...
tf_workspace.state_bucket in .terraform/modules/tf_workspace.state_bucket/modules/simple_bucket

Initializing provider plugins...

Finding hashicorp/google-beta versions matching ">= 3.30.0, >= 3.43.0, >= 3.45.0, >= 3.50.0, >= 3.67.0, >= 3.77.0, ~> 4.11, >= 4.17.0, ~> 4.28, != 4.31.0, < 5.0.0"...
Finding hashicorp/external versions matching ">= 2.2.2"...
Finding hashicorp/null versions matching ">= 2.1.0"...

Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.

If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.


terraform plan has errors on IAM permissions - 2 were my missing cloudidentity API enablement and roles/serviceusage.serviceUsageConsumer grant - but the 3rd was a missing cloudresourcemanager.googleapis.com API enablement

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform plan -input=false -out bootstrap.tfplan

gcs_bucket_tfstate = (known after apply)
networks_step_terraform_service_account_email = (known after apply)
organization_step_terraform_service_account_email = (known after apply)
projects_gcs_bucket_tfstate = (known after apply)
projects_step_terraform_service_account_email = (known after apply)
seed_project_id = (known after apply) ╷ │ Error: Error when reading or editing Organization Not Found : 9....309: googleapi: Error 403: Cloud Resource Manager API has not been used in project 46....3 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=46....63 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. │ Details: │ [ │ { │ "@type": "type.googleapis.com/google.rpc.Help", │ "links": [ │ { │ "description": "Google developers console API activation", │ "url": "https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=46...3" │ } │ ] │ }, │ { │ "@type": "type.googleapis.com/google.rpc.ErrorInfo", │ "domain": "googleapis.com", │ "metadata": { │ "consumer": "projects/46...63", │ "service": "cloudresourcemanager.googleapis.com" │ }, │ "reason": "SERVICE_DISABLED" │ } │ ] │ , accessNotConfigured │ │ with data.google_organization.org[0], │ on groups.tf line 30, in data "google_organization" "org": │ 30: data "google_organization" "org" { │

missed Enable the Cloud Identity API (cloudidentity.googleapis.com) on the billing project. Grant role roles/serviceusage.serviceUsageConsumer to the user running Terraform on the billing project.

on https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md#optional---automatic-creation-of-google-cloud-identity-groups

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud organizations add-iam-policy-binding "${ORGANIZATION_ID}" --member "user:${EMAIL}" --role roles/serviceusage.serviceUsageConsumer

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services enable cloudidentity.googleapis.com
Operation "operations/acat.p2-464286101163-44fe637f-13fb-4e86-abb9-8aa6fd3e5405" finished successfully.

verifying APIs enabled

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services list | grep NAME
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

missing cloudresourcemanager

enabling

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services enable cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-464286101163-ab55063b-66d9-4dfb-a6b2-016e8d8ac84a" finished successfully.

rerun terraform plan

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform plan -input=false -out bootstrap.tfplan

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform plan -input=false -out bootstrap.tfplan

        }
      + monitoring_workspace_users = {
          + id            = "monitoring_workspace_users@pbmm.landing.systems"
          + name          = "monitoring_workspace_users"
          + resource_name = (known after apply)
        }
    }
  + seed_project_id                                   = (known after apply)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: bootstrap.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "bootstrap.tfplan"

raised https://github.com/terraform-google-modules/terraform-example-foundation/issues/965

continue with terraform apply

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ export VET_PROJECT_ID=$PROJECT_ID
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform show -json bootstrap.tfplan > bootstrap.json
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud beta terraform vet bootstrap.json --policy-library="../policy-library" --project ${VET_PROJECT_ID}
Pausing command execution:

This command requires the `terraform-tools` component to be installed. Would you like to install the `terraform-tools` component to continue command execution? (Y/n)?  y

ERROR: (gcloud.beta.terraform.vet) 
You cannot perform this action because the Google Cloud CLI component manager 
is disabled for this installation. You can run the following command 
to achieve the same result for this installation: 

sudo apt-get install google-cloud-sdk-terraform-tools

Do you want to opt-in (y/N)?  y-cloud-sdk-cloud-run-proxy google-cloud-sdk-gke-gcloud-auth-plugin google-cloud-sdk-minikube google-cloud-sdk-skaffold google-cloud-sdk-package-go-module google-cloud-sdk-local-extract google-cloud-sdk-app-engine-grpc 
You cannot perform this action because the Google Cloud CLI component manager 
is disabled for this installation. You can run the following command 
to achieve the same result for this installation: 

sudo apt-get install google-cloud-sdk-terraform-tools

********************************************************************************
You are running apt-get inside of Cloud Shell. Note that your Cloud Shell  
The command will automatically proceed in 5 seconds or on any key. 

Visit https://cloud.google.com/shell/help for more information.                 
********************************************************************************
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  libpcre2-posix2
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  google-cloud-sdk-terraform-tools
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 23.1 MB of archives.
After this operation, 113 MB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt cloud-sdk-bullseye/main amd64 google-cloud-sdk-terraform-tools amd64 428.0.0-0 [23.1 MB]
Fetched 23.1 MB in 2s (11.7 MB/s)                            
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package google-cloud-sdk-terraform-tools.
(Reading database ... 141868 files and directories currently installed.)
Preparing to unpack .../google-cloud-sdk-terraform-tools_428.0.0-0_amd64.deb ...
Unpacking google-cloud-sdk-terraform-tools (428.0.0-0) ...
Setting up google-cloud-sdk-terraform-tools (428.0.0-0) ...

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud beta terraform vet bootstrap.json --policy-library="../policy-library" --project ${VET_PROJECT_ID}
Validating resources...done.                                                                                                                                                                                             

Note: I didn't rename but copied the tfvar file - so I pass in -var-file or delete the example.tfvars - deleted it

terraform apply

1602
terraform apply bootstrap.tfplan

module.bootstrap_csr_repo.null_resource.run_destroy_command[0]: Creation complete after 0s [id=7391127032857258936]
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Creation complete after 0s [id=wic]
random_string.suffix: Creating...
random_string.suffix: Creation complete after 0s [id=7trd]
module.tf_private_pool.random_string.suffix: Creating...
module.seed_bootstrap.random_id.suffix: Creation complete after 0s [id=yz8]
module.tf_private_pool.random_string.suffix: Creation complete after 0s [id=72bz]
module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Creating...
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creating...
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creating...
google_folder.bootstrap: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creating...
module.optional_group["gcp_scc_admin"].google_cloud_identity_group.group: Creating...
module.required_group["group_org_admins"].google_cloud_identity_group.group: Creating...
module.optional_group["gcp_audit_viewer"].google_cloud_identity_group.group: Creating...
module.optional_group["gcp_global_secrets_admin"].google_cloud_identity_group.group: Creating...
module.optional_group["gcp_security_reviewer"].google_cloud_identity_group.group: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Creation complete after 5s [id=folders/241483971186/roles/iam.serviceAccountUser/group:group_org_admins@pbmm.landing.systems]
module.optional_group["gcp_network_viewer"].google_cloud_identity_group.group: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creation complete after 6s [id=939763342309/roles/billing.user/group:group_org_admins@pbmm.landing.systems]
module.optional_group["gcp_platform_viewer"].google_cloud_identity_group.group: Creating...
module.required_group["group_org_admins"].google_cloud_identity_group.group: Creation complete after 8s [id=groups/02bn6wsx2rmxjqw]
module.optional_group["gcp_scc_admin"].google_cloud_identity_group.group: Creation complete after 8s [id=groups/01ci93xb0p7xrvr]
module.optional_group["gcp_audit_viewer"].google_cloud_identity_group.group: Creation complete after 8s [id=groups/00haapch3otbpq4]
module.required_group["monitoring_workspace_users"].google_cloud_identity_group.group: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Still creating... [10s elapsed]
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Still creating... [10s elapsed]
google_folder.bootstrap: Still creating... [10s elapsed]
module.optional_group["gcp_security_reviewer"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/02koq656452w6fg]
google_folder.bootstrap: Creation complete after 12s [id=folders/721726436675]
module.optional_group["gcp_network_viewer"].google_cloud_identity_group.group: Creation complete after 8s [id=groups/01pxezwc0qtu2pv]
module.optional_group["gcp_platform_viewer"].google_cloud_identity_group.group: Creation complete after 7s [id=groups/01opuj5n35f1i03]
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creation complete after 14s [id=folders/241483971186/roles/resourcemanager.projectCreator/group:group_org_admins@pbmm.landing.systems]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Creation complete after 14s [id=folders/241483971186/roles/serviceusage.serviceUsageConsumer/group:group_org_admins@pbmm.landing.system
module.required_group["audit_data_users"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/019c6y18142bjy1]
module.required_group["billing_data_users"].google_cloud_identity_group.group: Creation complete after 8s [id=groups/01fob9te4dvfo1e]
module.required_group["monitoring_workspace_users"].google_cloud_identity_group.group: Creation complete after 10s [id=groups/01mrcu092xoa41j]
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_billing_admins@pbmm.landing.systems does not exist., badRequest
│ 
│  156: resource "google_organization_iam_binding" "billing_creator" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., badRequest
│ 
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_billing_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_billing_admin,
│   on .terraform/modules/seed_bootstrap/main.tf line 196, in resource "google_organization_iam_member" "org_billing_admin":
│  196: resource "google_organization_iam_member" "org_billing_admin" {
│ 
╵
╷
│ Error: failed pre-requisites: failed to check permissions on billing account "billingAccounts/01906F-C5E311-859F42": googleapi: Error 403: Cloud Billing API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:

│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/464286101163",
│       "service": "cloudbilling.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main,
│   on .terraform/modules/seed_bootstrap.seed_project/modules/core_project_factory/main.tf line 73, in resource "google_project" "main":
│   73: resource "google_project" "main" {
│ 
╵

it looks like a post cloud identity group creation issue specific to billing further than https://github.com/terraform-google-modules/terraform-example-foundation/issues/959

Groups were created - single admin was added

Billing is enabled on the project

Triaging what was created / pending on the apply folder created fldr-bootstrap

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform plan -input=false -out bootstrap.tfplan
random_string.suffix: Refreshing state... [id=7trd]
module.seed_bootstrap.random_id.suffix: Refreshing state... [id=yz8]
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=wic]
module.tf_private_pool.random_string.suffix: Refreshing state... [id=72bz]
module.bootstrap_csr_repo.data.external.env_override[0]: Reading...
module.build_terraform_image.data.external.env_override[0]: Reading...
module.bootstrap_csr_repo.data.external.env_override[0]: Read complete after 0s [id=-]
module.build_terraform_image.data.external.env_override[0]: Read complete after 0s [id=-]
google_folder.bootstrap: Refreshing state... [id=folders/721726436675]
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.projectCreator/group:group_org_admins@pbmm.landing.systems]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Refreshing state... [id=939763342309/roles/billing.user/group:group_org_admins@pbmm.landing.systems]
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Refreshing state... [id=folders/241483971186/roles/serviceusage.serviceUsageConsumer/group:group_org_admins@pbmm.landing.systems]
      + proj      = (known after apply)
    }
  + gcs_bucket_tfstate                                = "bkt-prj-b-seed-tfstate-cb3f"
  + networks_step_terraform_service_account_email     = (known after apply)
  + organization_step_terraform_service_account_email = (known after apply)
  + projects_gcs_bucket_tfstate                       = "bkt-prj-b-seed-c227-gcp-projects-tfstate"
  + projects_step_terraform_service_account_email     = (known after apply)
  + seed_project_id                                   = "prj-b-seed-c227"

oot_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform apply bootstrap.tfplan
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creating...
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creating...
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Creation complete after 9s [id=939763342309/roles/billing.creator]
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Still creating... [10s elapsed]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creation complete after 14s [id=939763342309/roles/billing.admin/group:group_billing_admins@pbmm.landing.systems]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creation complete after 14s [id=939763342309/roles/resourcemanager.organizationAdmin/group:group_org_admins@pbmm.landing.systems]
╷
│ Error: failed pre-requisites: failed to check permissions on billing account "billingAccounts/01906F-C5E311-859F42": googleapi: Error 403: Cloud Billing API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/464286101163",
│       "service": "cloudbilling.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main,
│   on .terraform/modules/seed_bootstrap.seed_project/modules/core_project_factory/main.tf line 73, in resource "google_project" "main":
│   73: resource "google_project" "main" {
│

Billing is enabled - however enabling the billing api

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services list | grep NAME
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services enable cloudbilling.googleapis.com
Operation "operations/acat.p2-464286101163-00703144-22eb-44e1-88d1-ca14bc70a658" finished successfully.

rerun plan and apply

terraform plan -input=false -out bootstrap.tfplan
terraform apply bootstrap.tfplan

1701

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform apply bootstrap.tfplan
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [30s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [40s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [50s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m0s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m30s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m40s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m40s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m50s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [3m0s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [3m10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [3m20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creation complete after 3m22s [id=projects/prj-b-seed-c227]
module.seed_bootstrap.module.seed_project.module.project-factory.google_resource_manager_lien.lien[0]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["essentialcontacts.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["monitoring.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.google_resource_manager_lien.lien[0]: Creation complete after 1s [id=p709038195674-led40930c-8d8c-47f2-98f5-0ebceb91a24e]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["assuredworkloads.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iamcredentials.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["monitoring.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Creation complete after 18s [id=prj-b-seed-c227/cloudresourcemanager.googleapis.com]leapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["assuredworkloads.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-c227/assuredworkloads.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-c227/cloudbilling.googleapleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["essentialcontacts.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-c227/essentialcontacts.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["monitoring.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-c227/monitoring.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudkms.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iamcredentials.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-c227/iamcredentials.goog
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-c227/storage-api.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["pubsub.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["accesscontextmanager.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["pubsub.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudkms.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["accesscontextmanager.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["pubsub.googleapis.com"]: Still creating... [20s elapsed]

4 min

module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Creation complete after 31s [id=prj-b-seed-c227/cloudbuild.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["accesscontextmanager.googleapis.com"]: Creation complete after 31s [id=prj-b-seed-c227/accesscontextmanager.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creation complete after 31s [id=prj-b-seed-c227/appengine.googleapis.com
module.seed_bootstrap.module.seed_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creating...
module.seed_bootstrap.data.google_storage_project_service_account.gcs_account: Reading...
module.seed_bootstrap.module.seed_project.module.budget.data.google_project.project[0]: Reading...
google_service_account.terraform-env-sa["env"]: Creating...
module.seed_bootstrap.module.enable_cross_project_service_account_usage.google_project_organization_policy.project_policy_boolean[0]: Creating...
google_service_account.terraform-env-sa["org"]: Creating...
google_service_account.terraform-env-sa["proj"]: Creating...
module.seed_bootstrap.google_storage_bucket.org_terraform_state: Creating...
module.gcp_projects_state_bucket.google_storage_bucket.bucket: Creating...
module.seed_bootstrap.module.seed_project.module.budget.data.google_project.project[0]: Read complete after 0s [id=projects/prj-b-seed-c227]
module.seed_bootstrap.data.google_storage_project_service_account.gcs_account: Read complete after 1s [id=service-709038195674@gs-project-accounts.iam.gserviceaccount.com]
module.seed_bootstrap.module.enable_cross_project_service_account_usage.google_project_organization_policy.project_policy_boolean[0]: Creation complete after 1s [id=prj-b-seed-c227:constraints/iam.disableCrossProjectServiceAccountUsage]
module.gcp_projects_state_bucket.google_storage_bucket.bucket: Creation complete after 1s [id=bkt-prj-b-seed-c227-gcp-projects-tfstate]
module.seed_bootstrap.google_storage_bucket.org_terraform_state: Creation complete after 1s [id=bkt-prj-b-seed-tfstate-cb3f]
module.seed_bootstrap.google_storage_bucket_iam_member.orgadmins_state_iam[0]: Creating...
module.seed_bootstrap.google_storage_bucket_iam_member.orgadmins_state_iam[0]: Creation complete after 5s [id=b/bkt-prj-b-seed-tfstate-cb3f/roles/storage.admin/group:group_org_admins@pbmm.landing.systems]
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["org"],
│   on sa.tf line 137, in resource "google_service_account" "terraform-env-sa":
│  137: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["net"],
│   on sa.tf line 137, in resource "google_service_account" "terraform-env-sa":
│  137: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["env"],
│   on sa.tf line 137, in resource "google_service_account" "terraform-env-sa":
│  137: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["proj"],
│   on sa.tf line 137, in resource "google_service_account" "terraform-env-sa":
│  137: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["bootstrap"],
│   on sa.tf line 137, in resource "google_service_account" "terraform-env-sa":
│  137: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: error listing service accounts on project prj-b-seed-c227: failed to list service accounts on project "prj-b-seed-c227": googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/464286101163",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.seed_bootstrap.module.seed_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0],
│   on .terraform/modules/seed_bootstrap.seed_project/modules/core_project_factory/main.tf line 134, in resource "google_project_default_service_accounts" "default_service_accounts":
│  134: resource "google_project_default_service_accounts" "default_service_accounts" {
│ 
╵

add iam enablement

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services enable iam.googleapis.com
Operation "operations/acat.p2-464286101163-c4b35446-433c-4eef-9d33-8197f81107cb" finished successfully.

rerun plan and apply 1711

google_service_account.terraform-env-sa["bootstrap"]: Creating...
google_service_account.terraform-env-sa["org"]: Creation complete after 1s [id=projects/prj-b-seed-c227/serviceAccounts/sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
google_service_account.terraform-env-sa["bootstrap"]: Creation complete after 1s [id=projects/prj-b-seed-c227/serviceAccounts/sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.tf_billing_user["net"]: Creating...
google_billing_account_iam_member.tf_billing_user["org"]: Creating...
google_billing_account_iam_member.tf_billing_user["proj"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Creating...
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Creation complete after 4s [id=939763342309/roles/serviceusage.serviceUsageConsumer/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
google_billing_account_iam_member.tf_billing_user["net"]: Creation complete after 4s [id=01906F-C5E311-859F42/roles/billing.user/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
google_billing_account_iam_member.tf_billing_user["org"]: Still creating... [10s elapsed]
google_billing_account_iam_member.tf_billing_user["env"]: Still creating... [10s elapsed]
google_billing_account_iam_member.tf_billing_user["bootstrap"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]
google_billing_account_iam_member.tf_billing_user["proj"]: Still creating... [10s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]

module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [20s elapsed]
google_billing_account_iam_member.tf_billing_user["org"]: Creation complete after 21s [id=01906F-C5E311-859F42/roles/billing.user/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creating...
google_billing_account_iam_member.tf_billing_user["proj"]: Creation complete after 21s [id=01906F-C5E311-859F42/roles/billing.user/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Creating...
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creation complete after 29s [id=939763342309/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.notificationConfigEditor"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creation complete after 29s [id=939763342309/roles/resourcemanager.organizationAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/compute.xpnAdmin"]: Creating...
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creation complete after 29s [id=939763342309/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/essentialcontacts.admin"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creation complete after 30s [id=939763342309/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [10s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [10s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.notificationConfigEditor"]: Still creating... [10s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/compute.xpnAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/essentialcontacts.admin"]: Still creating... [10s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [20s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Still creating... [20s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [40s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [40s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 42s [id=939763342309/roles/browser/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 42s [id=939763342309/roles/browser/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.notificationConfigEditor"]: Still creating... [20s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/compute.xpnAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/essentialcontacts.admin"]: Still creating... [20s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.notificationConfigEditor"]: Still creating... [30s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [30s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [40s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [40s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [40s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Still creating... [40s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Creation complete after 41s [id=939763342309/roles/serviceusage.serviceUsageConsumer/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creation complete after 41s [id=939763342309/roles/resourcemanager.organizationAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 41s [id=939763342309/roles/browser/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Creation complete after 42s [id=939763342309/roles/resourcemanager.organizationViewer/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [20s elapsed]

module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/compute.xpnAdmin"]: Creation complete after 42s [id=939763342309/roles/compute.xpnAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/orgpolicy.policyAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/essentialcontacts.admin"]: Creation complete after 43s [id=939763342309/roles/essentialcontacts.admin/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creating...
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 42s [id=939763342309/roles/browser/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
module.seed_project_iam_member["org"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [30s elapsed]

module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Still creating... [20s elapsed]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creation complete after 7s [id=prj-b-seed-c227/roles/resourcemanager.projectDeleter/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [40s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [40s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creation complete after 43s [id=939763342309/roles/resourcemanager.tagUser/serviceAccount:sa-terraform-org@prj-b-seed-c22
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Creation complete after 4s [id=folders/241483971186/roles/resourcemanager.folderViewer/serviceAccount:sa-terraform-n
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/logging.configWriter"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Still creating... [30s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Still creating... [30s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Creating...
module.seed_project_iam_member["proj"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creation complete after 8s [id=prj-b-seed-c227/roles/storage.objectAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]227.iam.gserviceaccount.com]c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Creating...
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creation complete after 39s [id=939763342309/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityPolicyAdmin"]: Creating...
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Creation complete after 39s [id=939763342309/roles/assuredworkloads.admin/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
module.seed_project_iam_member["env"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creating...
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/logging.configWriter"]: Creation complete after 31s [id=939763342309/roles/logging.configWriter/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.seed_project_iam_member["net"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/orgpolicy.policyAdmin"]: Creation complete after 31s [id=939763342309/roles/orgpolicy.policyAdmin/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderIamAdmin"]: Creating...
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creation complete after 30s [id=939763342309/roles/resourcemanager.tagUser/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Creating...

module.seed_project_iam_member["env"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creation complete after 7s [id=prj-b-seed-c227/roles/storage.objectAdmin/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityPolicyAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderIamAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Still creating... [10s elapsed]
module.parent_iam_member["org"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Still creating... [20s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Still creating... [10s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Still creating... [10s elapsed]

module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Still creating... [30s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Still creating... [30s elapsed]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Still creating... [30s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Still creating... [30s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Still creating... [40s elapsed]
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [40s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Still creating... [40s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityPolicyAdmin"]: Still creating... [40s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderIamAdmin"]: Still creating... [40s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Still creating... [40s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Creation complete after 43s [id=folders/241483971186/roles/compute.orgSecurityResourceAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["org"]: Creating...
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creation complete after 42s [id=folders/241483971186/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["net"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Creation complete after 42s [id=folders/241483971186/roles/compute.networkAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["bootstrap"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityPolicyAdmin"]: Creation complete after 43s [id=folders/241483971186/roles/compute.orgSecurityPolicyAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["env"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderIamAdmin"]: Creation complete after 42s [id=folders/241483971186/roles/resourcemanager.folderIamAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["proj"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Creation complete after 42s [id=folders/241483971186/roles/artifactregistry.admin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [10s elapsed]
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["org"]: Creation complete after 5s [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Still creating... [40s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Still creating... [40s elapsed]

google_billing_account_iam_member.billing_admin_user["net"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["bootstrap"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["env"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["proj"]: Still creating... [10s elapsed]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [20s elapsed]
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creation complete after 24s [id=folders/241483971186/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Creation complete after 24s [id=folders/241483971186/roles/compute.securityAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Creation complete after 14s [id=folders/241483971186/roles/resourcemanager.projectCreator]
module.tf_source.module.cloudbuild_project.module.project-factory.random_id.random_project_id_suffix: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.random_id.random_project_id_suffix: Creation complete after 0s [id=rSQ]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Creating...
google_billing_account_iam_member.billing_admin_user["net"]: Still creating... [20s elapsed]

google_billing_account_iam_member.billing_admin_user["env"]: Still creating... [20s elapsed]
google_billing_account_iam_member.billing_admin_user["proj"]: Still creating... [20s elapsed]
google_billing_account_iam_member.billing_admin_user["net"]: Creation complete after 21s [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["bootstrap"]: Creation complete after 22s [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["env"]: Creation complete after 21s [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["proj"]: Creation complete after 21s [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [40s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [50s elapsed]

module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m0s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m40s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m50s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m0s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m40s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m50s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [3m0s elapsed]

module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [3m30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Creation complete after 3m32s [id=projects/prj-b-cicd-7trd]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.google_service_account.default_service_account[0]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["dns.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudscheduler.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.google_service_account.default_service_account[0]: Creation complete after 1s [id=projects/prj-b-cicd-7trd/serviceAccounts/project-service-account@prj-b-cicd-7trd.iam.gserviceaccount.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creating...

odule.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudscheduler.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["dns.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudscheduler.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Still creating... [20s elapsed]

module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Creation complete after 32s [id=prj-b-cicd-7trd/workflows.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creation complete after 32s [id=prj-b-cicd-7trd/cloudbilling.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Creation complete after 32s [id=prj-b-cicd-7trd/serviceusage.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["sourcerepo.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Creation complete after 3s [id=prj-b-cicd-7trd/logging.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Creation complete after 3s [id=prj-b-cicd-7trd/storage-api.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Creation complete after 3s [id=prj-b-cicd-7trd/compute.googleapis.com]

odule.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Creation complete after 21s [id=prj-b-cicd-7trd/artifactregistry.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creating...
module.tf_source.module.cloudbuild_project.module.budget.data.google_project.project[0]: Reading...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"]: Creating...
module.tf_source.google_project_iam_member.org_admins_source_repo_admin[0]: Creating...
module.tf_source.google_project_iam_member.org_admins_cloudbuild_viewer: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creation complete after 0s [id=projects/prj-b-cicd-7trd]
module.tf_source.module.cloudbuild_project.module.budget.data.google_project.project[0]: Read complete after 0s [id=projects/prj-b-cicd-7trd]
module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"]: Creating...
module.tf_source.google_project_iam_member.org_admins_cloudbuild_editor: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-policies]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-networks]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-bootstrap]
module.tf_source.module.cloudbuild_bucket.google_storage_bucket.bucket: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-environments]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-projects]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-org]
module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/tf-cloudbuilder]
module.tf_source.module.cloudbuild_bucket.google_storage_bucket.bucket: Creation complete after 1s [id=prj-b-cicd-7trd_cloudbuild]
module.tf_source.google_storage_bucket_iam_member.cloudbuild_iam: Creating...

odule.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Creation complete after 21s [id=prj-b-cicd-7trd/artifactregistry.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creating...
module.tf_source.module.cloudbuild_project.module.budget.data.google_project.project[0]: Reading...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"]: Creating...
module.tf_source.google_project_iam_member.org_admins_source_repo_admin[0]: Creating...
module.tf_source.google_project_iam_member.org_admins_cloudbuild_viewer: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creation complete after 0s [id=projects/prj-b-cicd-7trd]
module.tf_source.module.cloudbuild_project.module.budget.data.google_project.project[0]: Read complete after 0s [id=projects/prj-b-cicd-7trd]
module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"]: Creating...
module.tf_source.google_project_iam_member.org_admins_cloudbuild_editor: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-policies]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-networks]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-bootstrap]
module.tf_source.module.cloudbuild_bucket.google_storage_bucket.bucket: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-environments]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-projects]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/gcp-org]
module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"]: Creation complete after 2s [id=projects/prj-b-cicd-7trd/repos/tf-cloudbuilder]
module.tf_source.module.cloudbuild_bucket.google_storage_bucket.bucket: Creation complete after 1s [id=prj-b-cicd-7trd_cloudbuild]
module.tf_source.google_storage_bucket_iam_member.cloudbuild_iam: Creating...

module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creation complete after 12s [id=prj-b-cicd-7trd/roles/resourcemanager.projectDeleter/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_sourcerepo_repository_iam_member.member["net"]: Creation complete after 20s [id=projects/prj-b-cicd-7trd/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/artifactregistry.admin"]: Creation complete after 12s [id=prj-b-cicd-7trd/roles/artifactregistry.admin/serviceAccount:sa-terraform-boo
google_sourcerepo_repository_iam_member.member["org"]: Creation complete after 20s [id=projects/prj-b-cicd-7trd/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]eed-c227.iam.gserviceaccount.com]
google_sourcerepo_repository_iam_member.member["env"]: Creation complete after 20s [id=projects/prj-b-cicd-7trd/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.workflow_list: Creating...orm-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_cloud_builder.google_workflows_workflow.builder: Creating...
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creation complete after 9s [id=prj-b-cicd-7trd/roles/iam.serviceAccountAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_cloud_builder.module.bucket.google_storage_bucket.bucket: Creating...
module.tf_cloud_builder.google_sourcerepo_repository_iam_member.member[0]: Creating...
module.tf_cloud_builder.google_project_iam_member.logs_writer: Creating...
module.tf_cloud_builder.module.bucket.google_storage_bucket.bucket: Creation complete after 1s [id=bkt-prj-b-cicd-7trd-tf-cloudbuilder-build-logs]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.push_images: Creating...
module.tf_cloud_builder.google_workflows_workflow.builder: Creation complete after 2s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/workflows/terraform-runner-workflow]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Creating...
module.tf_cloud_builder.google_service_account_iam_member.use_cb_sa: Creation complete after 4s [id=projects/prj-b-cicd-7trd/serviceAccounts/tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["env"]: Creating...
module.tf_cloud_builder.google_sourcerepo_repository_iam_member.member[0]: Creation complete after 4s [id=projects/prj-b-cicd-7trd/repos/tf-cloudbuilder/roles/viewer/serviceAccount:tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Creating...
module.tf_cloud_builder.google_project_iam_member.logs_writer: Creation complete after 7s [id=prj-b-cicd-7trd/roles/logging.logWriter/serviceAccount:tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["org"]: Creating...
module.tf_cloud_builder.google_project_iam_member.invoke_workflow_scheduler: Creation complete after 7s [id=prj-b-cicd-7trd/roles/workflows.invoker/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Creation complete after 5s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudscheduler.admin"]: Creation complete after 7s [id=prj-b-cicd-7trd/roles/cloudscheduler.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["proj"]: Creating...
module.tf_cloud_builder.google_storage_bucket_iam_member.member: Creating...
module.tf_cloud_builder.google_cloud_scheduler_job.trigger_workflow: Creating...
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Creation complete after 7s [id=prj-b-cicd-7trd/roles/cloudbuild.builds.editor/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
module.bootstrap_projects_remove_editor["cicd"].google_project_iam_binding.iam_remove["roles/editor"]: Creating...
module.tf_cloud_builder.google_cloud_scheduler_job.trigger_workflow: Creation complete after 1s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/jobs/trigger-terraform-runner-workflow]
module.bootstrap_projects_remove_editor["seed"].google_project_iam_binding.iam_remove["roles/editor"]: Creating...

module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creation complete after 12s [id=prj-b-cicd-7trd/roles/resourcemanager.projectDeleter/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_sourcerepo_repository_iam_member.member["net"]: Creation complete after 20s [id=projects/prj-b-cicd-7trd/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/artifactregistry.admin"]: Creation complete after 12s [id=prj-b-cicd-7trd/roles/artifactregistry.admin/serviceAccount:sa-terraform-boo
google_sourcerepo_repository_iam_member.member["org"]: Creation complete after 20s [id=projects/prj-b-cicd-7trd/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]eed-c227.iam.gserviceaccount.com]
google_sourcerepo_repository_iam_member.member["env"]: Creation complete after 20s [id=projects/prj-b-cicd-7trd/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.workflow_list: Creating...orm-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_cloud_builder.google_workflows_workflow.builder: Creating...
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creation complete after 9s [id=prj-b-cicd-7trd/roles/iam.serviceAccountAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_cloud_builder.module.bucket.google_storage_bucket.bucket: Creating...
module.tf_cloud_builder.google_sourcerepo_repository_iam_member.member[0]: Creating...
module.tf_cloud_builder.google_project_iam_member.logs_writer: Creating...
module.tf_cloud_builder.module.bucket.google_storage_bucket.bucket: Creation complete after 1s [id=bkt-prj-b-cicd-7trd-tf-cloudbuilder-build-logs]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.push_images: Creating...
module.tf_cloud_builder.google_workflows_workflow.builder: Creation complete after 2s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/workflows/terraform-runner-workflow]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Creating...
module.tf_cloud_builder.google_service_account_iam_member.use_cb_sa: Creation complete after 4s [id=projects/prj-b-cicd-7trd/serviceAccounts/tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["env"]: Creating...
module.tf_cloud_builder.google_sourcerepo_repository_iam_member.member[0]: Creation complete after 4s [id=projects/prj-b-cicd-7trd/repos/tf-cloudbuilder/roles/viewer/serviceAccount:tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Creating...
module.tf_cloud_builder.google_project_iam_member.logs_writer: Creation complete after 7s [id=prj-b-cicd-7trd/roles/logging.logWriter/serviceAccount:tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["org"]: Creating...
module.tf_cloud_builder.google_project_iam_member.invoke_workflow_scheduler: Creation complete after 7s [id=prj-b-cicd-7trd/roles/workflows.invoker/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Creation complete after 5s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudscheduler.admin"]: Creation complete after 7s [id=prj-b-cicd-7trd/roles/cloudscheduler.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["proj"]: Creating...
module.tf_cloud_builder.google_storage_bucket_iam_member.member: Creating...
module.tf_cloud_builder.google_cloud_scheduler_job.trigger_workflow: Creating...
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Creation complete after 7s [id=prj-b-cicd-7trd/roles/cloudbuild.builds.editor/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
module.bootstrap_projects_remove_editor["cicd"].google_project_iam_binding.iam_remove["roles/editor"]: Creating...
module.tf_cloud_builder.google_cloud_scheduler_job.trigger_workflow: Creation complete after 1s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/jobs/trigger-terraform-runner-workflow]
module.bootstrap_projects_remove_editor["seed"].google_project_iam_binding.iam_remove["roles/editor"]: Creating...

module.tf_private_pool.module.peered_network[0].module.vpc.google_compute_network.network: Creation complete after 22s [id=projects/prj-b-cicd-7trd/global/networks/vpc-b-cbpools]
module.tf_private_pool.google_dns_policy.default_policy[0]: Creation complete after 1s [id=projects/prj-b-cicd-7trd/policies/dp-b-cbpools-default-policy]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["env"]: Still creating... [20s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Still creating... [20s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["env"]: Creation complete after 22s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Creation complete after 22s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["org"]: Creation complete after 19s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["proj"]: Creation complete after 19s [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.tf_private_pool.google_compute_global_address.worker_pool_range[0]: Still creating... [10s elapsed]
module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["northamerica-northeast1/sb-b-cbpools-northamerica-northeast1"]: Still creating... [10s elapsed]
module.tf_private_pool.google_compute_global_address.worker_pool_range[0]: Creation complete after 11s [id=projects/prj-b-cicd-7trd/global/addresses/ga-b-cbpools-worker-pool-range]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Creating...
module.tf_private_pool.module.firewall_rules[0].google_compute_firewall.rules["fw-b-cbpools-100-i-a-all-all-all-service-networking"]: Creating...
module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["northamerica-northeast1/sb-b-cbpools-northamerica-northeast1"]: Still creating... [20s elapsed]
module.tf_private_pool.module.firewall_rules[0].google_compute_firewall.rules["fw-b-cbpools-100-i-a-all-all-all-service-networking"]: Still creating... [10s elapsed]
module.tf_private_pool.module.firewall_rules[0].google_compute_firewall.rules["fw-b-cbpools-100-i-a-all-all-all-service-networking"]: Creation complete after 11s [id=projects/prj-b-cicd-7trd/global/firewalls/fw-b-cbpools-100-i-a-all-all-all-service-networking]
module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["northamerica-northeast1/sb-b-cbpools-northamerica-northeast1"]: Creation complete after 24s [id=projects/prj-b-cicd-7trd/regions/northamerica-northeast1/subnetworks/sb-b-cbpools-northamerica-northeast1]
╷
│ Error: Error waiting for Create Service Networking Connection: error while retrieving operation: googleapi: Error 403: Service Networking API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/servicenetworking.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/servicenetworking.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/464286101163",
│       "service": "servicenetworking.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0],
│   on modules/cb-private-pool/network.tf line 67, in resource "google_service_networking_connection" "worker_pool_conn":
│   67: resource "google_service_networking_connection" "worker_pool_conn" {
│ 
╵

1721 enabling networking (we will need compute as well)

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services list | grep NAME
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudbilling.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: datastore.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services enable servicenetworking.googleapis.com
Operation "operations/acat.p2-464286101163-f5c25269-242d-41d7-8291-f922734edb2c" finished successfully.

elapsed time = 13 min wait 2 min rerun plan/apply

1724

oot_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform apply bootstrap.tfplan
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Creating...

module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Still creating... [10s elapsed]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Still creating... [20s elapsed]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Creation complete after 21s [id=projects%2Fprj-b-cicd-7trd%2Fglobal%2Fnetworks%2Fvpc-b-cbpools:servicenetworking.googleapis.com]
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Creating...
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Creating...

╷
│ Error: Error creating WorkerPool: googleapi: Error 403: Cloud Build API has not been used in project 464286101163 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=464286101163 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=464286101163"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/464286101163",
│       "service": "cloudbuild.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ 
│   with module.tf_private_pool.google_cloudbuild_worker_pool.private_pool,
│   on modules/cb-private-pool/main.tf line 30, in resource "google_cloudbuild_worker_pool" "private_pool":
│   30: resource "google_cloudbuild_worker_pool" "private_pool" {
│ 
╵

1726 enable cloudbuild running into same prereq as https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/deployment.sh#L92 or https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/canary/solutions/document-processing/gcloud/deployment.sh#L238

enable apis

 gcloud services enable cloudbuild.googleapis.com

rerun plan/apply 1729

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform apply bootstrap.tfplan
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Creating...
╷
│ Error: Error creating WorkerPool: Resource already exists - apply blocked by lifecycle params: &cloudbuild.WorkerPool{Name:(*string)(0xc00240cd50), DisplayName:(*string)(0xc00240cda0), Uid:(*string)(0xc002334d20), Annotations:map[string]string{}, CreateTime:(*string)(0xc002334d30), UpdateTime:(*string)(0xc002334d40), DeleteTime:(*string)(nil), State:(*cloudbuild.WorkerPoolStateEnum)(0xc002334d50), PrivatePoolV1Config:(*cloudbuild.WorkerPoolPrivatePoolV1Config)(0xc0022e5f98), Etag:(*string)(0xc002334df0), WorkerConfig:(*cloudbuild.WorkerPoolWorkerConfig)(nil), NetworkConfig:(*cloudbuild.WorkerPoolNetworkConfig)(nil), Project:(*string)(0xc00240ce80), Location:(*string)(0xc00240cd10)}.
│ 
│   with module.tf_private_pool.google_cloudbuild_worker_pool.private_pool,
│   on modules/cb-private-pool/main.tf line 30, in resource "google_cloudbuild_worker_pool" "private_pool":
│   30: resource "google_cloudbuild_worker_pool" "private_pool" {
│ 
╵

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform apply bootstrap.tfplan
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Creating...
╷
│ Error: Error creating WorkerPool: Resource already exists - apply blocked by lifecycle params: &cloudbuild.WorkerPool{Name:(*string)(0xc000bf1c40), DisplayName:(*string)(0xc000bf1cb0), Uid:(*string)(0xc000da1c90), Annotations:map[string]string{}, CreateTime:(*string)(0xc000da1ca0), UpdateTime:(*string)(0xc000da1cb0), DeleteTime:(*string)(nil), State:(*cloudbuild.WorkerPoolStateEnum)(0xc000da1cc0), PrivatePoolV1Config:(*cloudbuild.WorkerPoolPrivatePoolV1Config)(0xc0017da7b0), Etag:(*string)(0xc000da1d70), WorkerConfig:(*cloudbuild.WorkerPoolWorkerConfig)(nil), NetworkConfig:(*cloudbuild.WorkerPoolNetworkConfig)(nil), Project:(*string)(0xc000bf1d90), Location:(*string)(0xc000bf1c00)}.
│ 
│   with module.tf_private_pool.google_cloudbuild_worker_pool.private_pool,
│   on modules/cb-private-pool/main.tf line 30, in resource "google_cloudbuild_worker_pool" "private_pool":
│   30: resource "google_cloudbuild_worker_pool" "private_pool" {
│

manually delete

disable billing on the cicd project - then delete it - hopefully the random identifier recycles (worst case we redeploy in folder 2) -sometimes resources that are deletes still show in asset inventory

on plan
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/tf-cloudbuilder": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [

starting from scratch with service enablements up front

Removing projects - delete the lien first

I usually script the lien - but this one has 2
export LIEN_PROJECT_ID=prj-b-seed-c227
gcloud config set project $LIEN_PROJECT_ID
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud alpha resource-manager liens list | grep NAME
NAME: p709038195674-lc6270422-68bb-4673-8062-aaa4f18781ab
NAME: p709038195674-led40930c-8d8c-47f2-98f5-0ebceb91a24e

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud alpha resource-manager liens delete p709038195674-lc6270422-68bb-4673-8062-aaa4f18781ab
Deleted [liens/p709038195674-lc6270422-68bb-4673-8062-aaa4f18781ab].
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud alpha resource-manager liens list | grep NAME
NAME: p709038195674-led40930c-8d8c-47f2-98f5-0ebceb91a24e
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud alpha resource-manager liens delete p709038195674-led40930c-8d8c-47f2-98f5-0ebceb91a24e
Deleted [liens/p709038195674-led40930c-8d8c-47f2-98f5-0ebceb91a24e].
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud alpha resource-manager liens list | grep NAME
Listed 0 items.

delete project
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud alpha billing projects unlink ${LIEN_PROJECT_ID}
billingAccountName: ''
billingEnabled: false
name: projects/prj-b-seed-c227/billingInfo
projectId: prj-b-seed-c227
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud projects delete ${LIEN_PROJECT_ID} --quiet
Deleted [https://cloudresourcemanager.googleapis.com/v1/projects/prj-b-seed-c227].

You can undo this operation for a limited period by running the command below.
    $ gcloud projects undelete prj-b-seed-c227

obriensystems commented 1 year ago

Run 2

#parent_folder = "241483971186"
parent_folder = "200904469514"

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (prj-b-seed-c227)$ gcloud config set project lz-tef-plz
Updated property [core/project].
root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ gcloud services list | grep NAME
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudbilling.googleapis.com
NAME: cloudbuild.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: containerregistry.googleapis.com
NAME: datastore.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: pubsub.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: servicenetworking.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

We will see how the existing org level and iam groups get handled - including the existing service accounts

1800

root_@cloudshell:~/lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-plz)$ terraform plan -input=false -out bootstrap.tfplan
module.build_terraform_image.data.external.env_override[0]: Reading...
module.bootstrap_csr_repo.data.external.env_override[0]: Reading...
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Refreshing state... [id=wic]
random_string.suffix: Refreshing state... [id=7trd]
module.seed_bootstrap.random_id.suffix: Refreshing state... [id=yz8]
module.tf_private_pool.random_string.suffix: Refreshing state... [id=72bz]
module.build_terraform_image.data.external.env_override[0]: Read complete after 0s [id=-]
module.bootstrap_csr_repo.data.external.env_override[0]: Read complete after 0s [id=-]
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Refreshing state... [id=folders/241483971186/roles/serviceusage.serviceUsageConsumer/group:group_org_admins@pbmm.landing.systems]
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.projectCreator/group:group_org_admins@pbmm.landing.systems]
module.required_group["group_org_admins"].google_cloud_identity_group.group: Refreshing state... [id=groups/02bn6wsx2rmxjqw]
module.required_group["audit_data_users"].google_cloud_identity_group.group: Refreshing state... [id=groups/019c6y18142bjy1]
module.optional_group["gcp_audit_viewer"].google_cloud_identity_group.group: Refreshing state... [id=groups/00haapch3otbpq4]
module.optional_group["gcp_platform_viewer"].google_cloud_identity_group.group: Refreshing state... [id=groups/01opuj5n35f1i03]
module.optional_group["gcp_scc_admin"].google_cloud_identity_group.group: Refreshing state... [id=groups/01ci93xb0p7xrvr]
module.required_group["group_billing_admins"].google_cloud_identity_group.group: Refreshing state... [id=groups/04i7ojhp1pp161r]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Refreshing state... [id=projects/prj-b-seed-c227]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Refreshing state... [id=prj-b-seed-c227/admin.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Refreshing state... [id=prj-b-seed-c227/securitycenter.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Refreshing state... [id=prj-b-seed-c227/compute.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iamcredentials.googleapis.com"]: Refreshing state... [id=prj-b-seed-c227/iamcredentials.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["accesscontextmanager.googleapis.com"]: Refreshing state... [id=prj-b-seed-c227/accesscontextmanager.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Refreshing state... [id=prj-b-seed-c227/cloudresourcemanager.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Refreshing state... [id=prj-b-seed-c227/cloudbilling.googleapis.com]
module.seed_project_iam_member["proj"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Refreshing state... [id=prj-b-seed-c227/roles/storage.objectAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.i@prj-b-seed-c227.iam.gserviceaccount.com]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Refreshing state... [id=prj-b-seed-c227/roles/resourcemanager.projectDeleter/serviceAccount:sa-terraj-b-seed-c227.iam.gserviceaccount.com]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Refreshing state... [id=939763342309/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-net@prj-
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Refreshing state... [id=939763342309/roles/resourcemanager.organizationViewer/serviceAccount:sa-terraform-org@j@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityPolicyAdmin"]: Refreshing state... [id=folders/241483971186/roles/compute.orgSecurityPolicyAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["org"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.folderViewer/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Refreshing state... [id=folders/241483971186/roles/compute.xpnAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Refreshing state... [id=folders/241483971186/roles/compute.networkAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Refreshing state... [id=folders/241483971186/roles/compute.orgSecurityResourceAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Refreshing state... [id=folders/241483971186/roles/compute.securityAdmin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Refreshing state... [id=folders/241483971186/roles/artifactregistry.admin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Refreshing state... [id=folders/241483971186/roles/compute.networkAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.folderViewer/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Refreshing state... [id=folders/241483971186/roles/dns.admin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["env"]: Refreshing state... [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-env@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["net"]: Refreshing state... [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-net@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["proj"]: Refreshing state... [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-proj@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["bootstrap"]: Refreshing state... [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-c227.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["org"]: Refreshing state... [id=01906F-C5E311-859F42/roles/billing.admin/serviceAccount:sa-terraform-org@prj-b-seed-c227.iam.gserviceaccount.com]
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Refreshing state... [id=folders/241483971186/roles/resourcemanager.projectCreator]

hanging

module.tf_source.module.cloudbuild_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Refreshing state... [id=projects/prj-b-cicd-7trd]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"]: Refreshing state... [id=projects/prj-b-cicd-7trd/repos/gcp-projects]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"]: Refreshing state... [id=projects/prj-b-cicd-7trd/repos/gcp-environments]
module.tf_source.google_project_iam_member.org_admins_cloudbuild_editor: Refreshing state... [id=prj-b-cicd-7trd/roles/cloudbuild.builds.editor/group:group_org_admins@pbmm.landing.systems]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"]: Refreshing state... [id=projects/prj-b-cicd-7trd/repos/gcp-bootstrap]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"]: Refreshing state... [id=projects/prj-b-cicd-7trd/repos/gcp-networks]
module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"]: Refreshing state... [id=projects/prj-b-cicd-7trd/repos/tf-cloudbuilder]
module.tf_source.google_project_iam_member.org_admins_source_repo_admin[0]: Refreshing state... [id=prj-b-cicd-7trd/roles/source.admin/group:group_org_admins@pbmm.landing.systems]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"]: Refreshing state... [id=projects/prj-b-cicd-7trd/repos/gcp-policies]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"]: Refreshing state... [id=projects/prj-b-cicd-7trd/repos/gcp-org]
module.tf_source.module.cloudbuild_bucket.google_storage_bucket.bucket: Refreshing state... [id=prj-b-cicd-7trd_cloudbuild]
module.tf_source.google_project_iam_member.org_admins_cloudbuild_viewer: Refreshing state... [id=prj-b-cicd-7trd/roles/viewer/group:group_org_admins@pbmm.landing.systems]

1805 hanging

module.tf_cloud_builder.google_project_iam_member.invoke_workflow_scheduler: Refreshing state... [id=prj-b-cicd-7trd/roles/workflows.invoker/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Refreshing state... [id=prj-b-cicd-7trd/roles/cloudbuild.builds.editor/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
module.tf_cloud_builder.google_service_account_iam_member.use_cb_sa: Refreshing state... [id=projects/prj-b-cicd-7trd/serviceAccounts/tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
module.tf_cloud_builder.google_workflows_workflow.builder: Refreshing state... [id=projects/prj-b-cicd-7trd/locations/northamerica-northeast1/workflows/terraform-runner-workflow]
module.tf_cloud_builder.google_project_iam_member.logs_writer: Refreshing state... [id=prj-b-cicd-7trd/roles/logging.logWriter/serviceAccount:tf-cb-builder-sa@prj-b-cicd-7trd.iam.gserviceaccount.com]
module.tf_private_pool.google_dns_policy.default_policy[0]: Refreshing state... [id=projects/prj-b-cicd-7trd/policies/dp-b-cbpools-default-policy]
module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["northamerica-northeast1/sb-b-cbpools-northamerica-northeast1"]: Refreshing state... [id=projects/prj-b-cicd-7trd/regions/northamerica-northeast1/subnetworks/sb-b-cbpools-northamerica-northeast1]
module.tf_private_pool.google_compute_global_address.worker_pool_range[0]: Refreshing state... [id=projects/prj-b-cicd-7trd/global/addresses/ga-b-cbpools-worker-pool-range]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Refreshing state... [id=projects%2Fprj-b-cicd-7trd%2Fglobal%2Fnetworks%2Fvpc-b-cbpools:servicenetworking.googleapis.com]
module.tf_private_pool.module.firewall_rules[0].google_compute_firewall.rules["fw-b-cbpools-100-i-a-all-all-all-service-networking"]: Refreshing state... [id=projects/prj-b-cicd-7trd/global/firewalls/fw-b-cbpools-100-i-a-all-all-all-service-networking]
module.bootstrap_projects_remove_editor["cicd"].google_project_iam_binding.iam_remove["roles/editor"]: Refreshing state... [id=prj-b-cicd-7trd/roles/editor]
module.bootstrap_projects_remove_editor["seed"].google_project_iam_binding.iam_remove["roles/editor"]: Refreshing state... [id=prj-b-seed-c227/roles/editor]

1807

  # module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"] will be created
  + resource "google_project_service" "project_services" {
      + disable_dependent_services = true
      + disable_on_destroy         = false
      + id                         = (known after apply)
      + project                    = "prj-b-cicd-7trd"
      + service                    = "workflows.googleapis.com"
    }

Plan: 75 to add, 1 to change, 21 to destroy.

Changes to Outputs:
  ~ cloud_build_peered_network_id                     = "projects/prj-b-cicd-7trd/global/networks/vpc-b-cbpools" -> (known after apply)
  ~ cloud_build_worker_range_id                       = "projects/prj-b-cicd-7trd/global/addresses/ga-b-cbpools-worker-pool-range" -> (known after apply)
  ~ common_config                                     = {
      ~ parent_folder         = "241483971186" -> "200904469514"
      ~ parent_id             = "folders/241483971186" -> "folders/200904469514"
        # (6 unchanged attributes hidden)
    }
╷
│ Error: Error when reading or editing DNSPolicy "projects/prj-b-cicd-7trd/policies/dp-b-cbpools-default-policy": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "dns.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ , forbidden
│ 
│   with module.tf_private_pool.google_dns_policy.default_policy[0],
│   on modules/cb-private-pool/network.tf line 43, in resource "google_dns_policy" "default_policy":
│   43: resource "google_dns_policy" "default_policy" {
│ 
╵
╷
│ Error: googleapi: Error 403: Permission denied on resource project #1026400783057.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/1026400783057/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/1026400783057",
│       "service": "servicenetworking.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ , forbidden
│ 
│   with module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0],
│   on modules/cb-private-pool/network.tf line 67, in resource "google_service_networking_connection" "worker_pool_conn":
│   67: resource "google_service_networking_connection" "worker_pool_conn" {
│ 
╵
╷
│ Error: Error when reading or editing ArtifactRegistryRepository "projects/prj-b-cicd-7trd/locations/northamerica-northeast1/repositories/tf-runners": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "artifactregistry.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_cloud_builder.google_artifact_registry_repository.tf-image-repo,
│   on .terraform/modules/tf_cloud_builder/modules/tf_cloudbuild_builder/gar.tf line 21, in resource "google_artifact_registry_repository" "tf-image-repo":
│   21: resource "google_artifact_registry_repository" "tf-image-repo" {
│ 
╵
╷
│ Error: Error when reading or editing WorkflowsWorkflow "projects/prj-b-cicd-7trd/locations/northamerica-northeast1/workflows/terraform-runner-workflow": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "workflows.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_cloud_builder.google_workflows_workflow.builder,
│   on .terraform/modules/tf_cloud_builder/modules/tf_cloudbuild_builder/workflow.tf line 35, in resource "google_workflows_workflow" "builder":
│   35: resource "google_workflows_workflow" "builder" {
│ 
╵
╷
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/gcp-networks": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "sourcerepo.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"],
│   on .terraform/modules/tf_source/modules/tf_cloudbuild_source/main.tf line 63, in resource "google_sourcerepo_repository" "gcp_repo":
│   63: resource "google_sourcerepo_repository" "gcp_repo" {
│ 
╵
╷
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/tf-cloudbuilder": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "sourcerepo.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"],
│   on .terraform/modules/tf_source/modules/tf_cloudbuild_source/main.tf line 63, in resource "google_sourcerepo_repository" "gcp_repo":
│   63: resource "google_sourcerepo_repository" "gcp_repo" {
│ 
╵
╷
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/gcp-environments": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "sourcerepo.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"],
│   on .terraform/modules/tf_source/modules/tf_cloudbuild_source/main.tf line 63, in resource "google_sourcerepo_repository" "gcp_repo":
│   63: resource "google_sourcerepo_repository" "gcp_repo" {
│ 
╵
╷
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/gcp-policies": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "sourcerepo.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"],
│   on .terraform/modules/tf_source/modules/tf_cloudbuild_source/main.tf line 63, in resource "google_sourcerepo_repository" "gcp_repo":
│   63: resource "google_sourcerepo_repository" "gcp_repo" {
│ 
╵
╷
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/gcp-projects": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "sourcerepo.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"],
│   on .terraform/modules/tf_source/modules/tf_cloudbuild_source/main.tf line 63, in resource "google_sourcerepo_repository" "gcp_repo":
│   63: resource "google_sourcerepo_repository" "gcp_repo" {
│ 
╵
╷
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/gcp-bootstrap": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "sourcerepo.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"],
│   on .terraform/modules/tf_source/modules/tf_cloudbuild_source/main.tf line 63, in resource "google_sourcerepo_repository" "gcp_repo":
│   63: resource "google_sourcerepo_repository" "gcp_repo" {
│ 
╵
╷
│ Error: Error when reading or editing SourceRepoRepository "projects/prj-b-cicd-7trd/repos/gcp-org": googleapi: Error 403: Permission denied on resource project prj-b-cicd-7trd.
│ Details:
│ [
│   {
│     "links": [
│         "description": "Google developer console API key",
│         "url": "https://console.developers.google.com/project/prj-b-cicd-7trd/apiui/credential"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/prj-b-cicd-7trd",
│       "service": "sourcerepo.googleapis.com"
│     },
│     "reason": "CONSUMER_INVALID"
│   }
│ ]
│ 
│   with module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"],
│   on .terraform/modules/tf_source/modules/tf_cloudbuild_source/main.tf line 63, in resource "google_sourcerepo_repository" "gcp_repo":
│   63: resource "google_sourcerepo_repository" "gcp_repo" {
│

switch projects, repo folder and tfstate file

obriensystems commented 1 year ago

delete previous SA's

Create new project off new folder

root_@cloudshell:~ (lz-tef-plz)$ mkdir lz-tef-pls
root_@cloudshell:~ (lz-tef-plz)$ cd lz-tef-pls
root_@cloudshell:~/lz-tef-pls (lz-tef-plz)$ mkdir CloudLandingSystems
root_@cloudshell:~/lz-tef-pls (lz-tef-plz)$ cd CloudLandingSystems/
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-plz)$ export PROJECT_ID=lz-tef-pls
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-plz)$ gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default --folder=200904469514
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/lz-tef-pls].
Waiting for [operations/cp.8227492116560663364] to finish...done.                                                                                                                                                                    
Enabling service [cloudapis.googleapis.com] on project [lz-tef-pls]...
Operation "operations/acat.p2-274207777486-80a22ed8-0930-4dcd-9a41-83b05a6e6e79" finished successfully.
Updated property [core/project] to [lz-tef-pls].
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ 
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ export BILLING_ID=$(gcloud alpha billing projects describe lz-tef-plz '--format=value(billingAccountName)' | sed 's/.*\///')
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ export ORGANIZATION_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud beta billing projects link ${PROJECT_ID} --billing-account ${BILLING_ID}
billingAccountName: billingAccounts/01906F-...-859F42
billingEnabled: true
name: projects/lz-tef-pls/billingInfo
projectId: lz-tef-pls

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services list | grep NAME
NAME: bigquery.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: clouddebugger.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

adding API enablements

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudidentity.googleapis.com
Operation "operations/acat.p2-274207777486-c2e35ef4-2886-4a97-aeb0-31a0deebc2e1" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-274207777486-38db3533-2d05-415e-9aef-fb898ad75557" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudbilling.googleapis.com
Operation "operations/acat.p2-274207777486-2dafc37a-4ebe-430b-9622-65a84c1b124b" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable iam.googleapis.com
Operation "operations/acat.p2-274207777486-cff7130a-19d8-4a74-857f-c1acb37b0bdb" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable servicenetworking.googleapis.com
Operation "operations/acat.p2-274207777486-dc3b9587-ecdb-4b40-ab36-ed3441e06004" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable accesscontextmanager.googleapis.com
Operation "operations/acat.p2-274207777486-b8c358e9-8464-4663-a137-51e1d48c903a" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable sourcerepo.googleapis.com
Operation "operations/acat.p2-274207777486-f5352256-0046-42b0-882f-32919ccf5fa9" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable artifactregistry.googleapis.com
Operation "operations/acat.p2-274207777486-7da442f6-7da1-4009-ad0e-809bba162e34" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable containerregistry.googleapis.com
Operation "operations/acf.p2-274207777486-b5755d7c-6532-4c1d-8bb3-fd6fe987c993" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable compute.googleapis.com
Operation "operations/acf.p2-274207777486-23fdc89e-121d-4e33-abe0-5eee03e09e44" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable run.googleapis.com
Operation "operations/acf.p2-274207777486-b97f4ffd-8ef3-457d-a90b-513a080f5759" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudapis.googleapis.com
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudbuild.googleapis.com
Operation "operations/acf.p2-274207777486-414049b5-95f6-41f6-93d1-2e386243bbbf" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable storage-component.googleapis.com
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudkms.googleapis.com
Operation "operations/acat.p2-274207777486-c9e823e7-ac93-4872-bc1a-aab3b9a8f2f6" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable logging.googleapis.com
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudfunctions.googleapis.com
Operation "operations/acf.p2-274207777486-39511de6-6163-423a-947c-14f01d2b614c" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable container.googleapis.com
Operation "operations/acf.p2-274207777486-366d354e-377c-4ebf-97c2-815dc266ee71" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable containeranalysis.googleapis.com
Operation "operations/acf.p2-274207777486-5f49c2be-8d12-48e5-af23-1b6b4bc4dd27" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable krmapihosting.googleapis.com
Operation "operations/acat.p2-274207777486-b1ee6b69-1ff5-4303-8f55-2086844a42d2" finished successfully.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ gcloud services enable cloudasset.googleapis.com
Operation "operations/acat.p2-274207777486-7cecf87e-7ad6-428b-851a-f4ae9b959c76" finished successfully.

clone existing

oot_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ ls
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ git clone https://github.com/CloudLandingZone/terraform-example-foundation.git
Cloning into 'terraform-example-foundation'...
remote: Enumerating objects: 8545, done.
remote: Counting objects: 100% (202/202), done.
remote: Compressing objects: 100% (141/141), done.
remote: Total 8545 (delta 79), reused 133 (delta 49), pack-reused 8343
Receiving objects: 100% (8545/8545), 2.26 MiB | 7.70 MiB/s, done.
Resolving deltas: 100% (6158/6158), done.
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems (lz-tef-pls)$ cd terraform-example-foundation/
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation (lz-tef-pls)$ git checkout pbmm
Branch 'pbmm' set up to track remote branch 'pbmm' from 'origin'.
Switched to a new branch 'pbmm'

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation (lz-tef-pls)$ cp ../../../lz-tef-plz/CloudLandingZone/terraform-example-foundation/scripts/validate-requirements.sh scripts/
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation (lz-tef-pls)$ cp ../../../lz-tef-plz/CloudLandingZone/terraform-example-foundation/0-bootstrap/terraform.tfvars 0-bootstrap/
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation (lz-tef-pls)$ rm -rf 0-bootstrap/terraform.example.tfvars 
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation (lz-tef-pls)$ cd 0-bootstrap/
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ 

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ ../scripts/validate-requirements.sh -o $ORGANIZATION_ID -b $BILLING_ID -u $EMAIL
Validating required utility tools...
Validating Terraform installation...
Validating Google Cloud SDK installation...
Validating Git installation...
  git default branch must be configured as main.
  See the instructions at https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting .
Validating local gcloud configuration...
Validating roles assignment for current end user credential...
Validating 0-bootstrap configuration...
.......................................
Validation successful!
No errors found.

terraform init

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ terraform plan -input=false -out bootstrap.tfplan

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ export VET_PROJECT_ID=$PROJECT_ID
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ echo $VET_PROJECT_ID
lz-tef-pls
root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ terraform show -json bootstrap.tfplan > bootstrap.json

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ gcloud beta terraform vet bootstrap.json --policy-library="../policy-library" --project ${VET_PROJECT_ID}
Validating resources...done.  

1841

root_@cloudshell:~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap (lz-tef-pls)$ terraform apply bootstrap.tfplan
module.seed_bootstrap.random_id.suffix: Creating...
random_string.suffix: Creating...

sa's added
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Creation complete after 4s [id=folders/200904469514/roles/resourcemanager.folderViewer/serviceAccount:sa-terraform-proj@prj-b-seed-4772.iam.gserviceaccount.com]

1850
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m30s elapsed]

1853
dule.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["dns.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Still creating... [10s elapsed]
mod

odule.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/source.admin"]: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Project [prj-b-cicd-r0di] repository [tf-cloudbuilder] was cloned to [/tmp/tmp.OMgoTUiBau].
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): /tmp/tmp.OMgoTUiBau ~/lz-tef-pls/CloudLandingSystems/terraform-example-foundation/0-bootstrap
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config user.email terraform-robot@example.com
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): error: pathspec 'main' did not match any file(s) known to git
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git checkout -b main
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Switched to a new branch 'main'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git add Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git commit -m 'Initialize tf dockerfile repo'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): [main (root-commit) 4a97d4f] Initialize tf dockerfile repo
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  1 file changed, 39 insertions(+)
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  create mode 100644 Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git push origin main -f
google_sourcerepo_repository_iam_member.member["net"]: Creation complete after 4s [id=projects/prj-b-cicd-r0di/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-net@prj-b-seed-4772.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/workflows.admin"]: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): To https://source.developers.google.com/p/prj-b-cicd-r0di/r/tf-cloudbuilder
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  * [new branch]      main -> main
module.bootstrap_csr_repo.null_resource.run_command[0]: Creation complete after 7s [id=1257885978800531261]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/source.admin"]: Creation complete after 6s [id=prj-b-cicd-r0di/roles/source.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-4772.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/storage.admin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creation complete after 7s [id=prj-b-cicd-r0di/roles/iam.serviceAccountAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-4772.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/artifactregistry.admin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.workerPoolOwner"]: Creation complete after 7s [id=prj-b-cicd-r0di/roles/cloudbuild.workerPoolOwner/serviceAccount:sa-terraform-bootstrap@prj-b-seed-4772.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudscheduler.admin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/dns.admin"]: Creation complete after 8s [id=prj-b-cicd-r0di/roles/dns.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-4772.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.builds.editor"]: Creating...

module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["northamerica-northeast1/sb-b-cbpools-northamerica-northeast1"]: Creation complete after 14s [id=projects/prj-b-cicd-r0di/regions/northamerica-northeast1/subnetworks/sb-b-cbpools-northamerica-northeast1]

module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Creation complete after 11s [id=projects/prj-b-cicd-r0di/global/networks/vpc-b-cbpools/networkPeerings/servicenetworking-googleapis-com]

1900 as expected - group collisions

odule.build_terraform_image.null_resource.run_command[0] (local-exec): name: operations/build/prj-b-cicd-r0di/NDYyM2VmMTAtNDA3NC00YjQzLTliMjYtY2JlYzA5MDVmNDQ4
module.build_terraform_image.null_resource.run_command[0]: Creation complete after 3s [id=8891116333862544152]
module.tf_workspace["env"].google_storage_bucket_iam_member.log_admin: Creation complete after 5s [id=b/bkt-prj-b-cicd-r0di-gcp-environments-build-logs/roles/storage.admin/serviceAccount:sa-terraform-env@prj-b-seed-4772.iam.gserviceaccount.com]
╷
│ Error: Error creating Group: googleapi: Error 409: Error(2018): Cannot create group 'gcp_scc_admin@pbmm.landing.systems' because it already exists.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
│     "description": "Error(2018): Cannot create group 'gcp_scc_admin@pbmm.landing.systems' because it already exists.",

deleting only group* groups - for the rest just deleted section in tfvars

will test recreation first

deleting

as expected - disabled

rerun plan/apply

Plan: 20 to add, 2 to change, 10 to destroy.
module.seed_bootstrap.module.seed_project.module.budget.data.google_project.project[0]: Read complete after 1s [id=projects/prj-b-seed-4772]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creating...
╷
│ Error: Error applying IAM policy for folder "folders/200904469514": Error setting IAM policy for folder "folders/200904469514": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 47, in resource "google_folder_iam_member" "tmp_project_creator":
│   47: resource "google_folder_iam_member" "tmp_project_creator" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_billing_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_binding.billing_creator,
│   on .terraform/modules/seed_bootstrap/main.tf line 156, in resource "google_organization_iam_binding" "billing_creator":
│  156: resource "google_organization_iam_binding" "billing_creator" {
│ 
╵
╷
│ Error: Error applying IAM policy for folder "folders/200904469514": Error setting IAM policy for folder "folders/200904469514": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_binding.project_creator[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 172, in resource "google_folder_iam_binding" "project_creator":
│  172: resource "google_folder_iam_binding" "project_creator" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"],
│   on .terraform/modules/seed_bootstrap/main.tf line 184, in resource "google_organization_iam_member" "org_admins_group":
│  184: resource "google_organization_iam_member" "org_admins_group" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"],
│   on .terraform/modules/seed_bootstrap/main.tf line 184, in resource "google_organization_iam_member" "org_admins_group":
│  184: resource "google_organization_iam_member" "org_admins_group" {
│ 
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_billing_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_billing_admin,
│   on .terraform/modules/seed_bootstrap/main.tf line 196, in resource "google_organization_iam_member" "org_billing_admin":
│  196: resource "google_organization_iam_member" "org_billing_admin" {
│ 
╵
╷
│ Error: Error applying IAM policy for folder "folders/200904469514": Error setting IAM policy for folder "folders/200904469514": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 259, in resource "google_folder_iam_member" "org_admin_service_account_user":
│  259: resource "google_folder_iam_member" "org_admin_service_account_user" {
│ 
╵
╷
│ Error: Error applying IAM policy for folder "folders/200904469514": Error setting IAM policy for folder "folders/200904469514": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 267, in resource "google_folder_iam_member" "org_admin_serviceusage_consumer":
│  267: resource "google_folder_iam_member" "org_admin_serviceusage_consumer" {
│ 
╵
╷
│ Error: Error applying IAM policy for storage bucket "b/bkt-prj-b-seed-tfstate-15c9": Error setting IAM policy for storage bucket "b/bkt-prj-b-seed-tfstate-15c9": googleapi: Error 400: Group group_org_admins@pbmm.landing.systems does not exist., invalid
│ 
│   with module.seed_bootstrap.google_storage_bucket_iam_member.orgadmins_state_iam[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 275, in resource "google_storage_bucket_iam_member" "orgadmins_state_iam":
│  275: resource "google_storage_bucket_iam_member" "orgadmins_state_iam" {
│

turning groups back on - but all renamed like

gcp_audit_viewer2

1907

 Error: Error applying IAM policy for storage bucket "b/bkt-prj-b-seed-tfstate-15c9": Error setting IAM policy for storage bucket "b/bkt-prj-b-seed-tfstate-15c9": googleapi: Error 400: Group group_org_admins2@pbmm.landing.systems does not exist., invalid
│

deleting all groups

expected

│ Error: Error when reading or editing CloudIdentityGroup "groups/04anzqyu4fhe22x": googleapi: Error 403: Error(2017): Permission denied for group resource 'groups/04anzqyu4fhe22x' (or it may not exist).
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
│     "description": "Error(2017): Permission denied for group resource 'groups/04anzqyu4fhe22x' (or it may not exist).",

doing a pass with groups commented out again

rerun clean 3

obriensystems commented 1 year ago

Run 3 using preliminary script

set -ex
export PROJECT_ID=lz-tef-pls3
mkdir CloudLandingZone
cd CloudLandingZone
gcloud projects create $PROJECT_ID --name="${PROJECT_ID}" --set-as-default --folder=287605518452
export BILLING_ID=$(gcloud alpha billing projects describe lz-tef-plz '--format=value(billingAccountName)' | sed 's/.*\///')
export ORGANIZATION_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)
gcloud beta billing projects link ${PROJECT_ID} --billing-account ${BILLING_ID}
gcloud services list | grep NAME
gcloud services enable cloudidentity.googleapis.com
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable cloudbilling.googleapis.com
gcloud services enable iam.googleapis.com
gcloud services enable servicenetworking.googleapis.com
gcloud services enable accesscontextmanager.googleapis.com
gcloud services enable sourcerepo.googleapis.com
gcloud services enable artifactregistry.googleapis.com
gcloud services enable containerregistry.googleapis.com
gcloud services enable cloudbilling.googleapis.com
gcloud services enable cloudidentity.googleapis.com
gcloud services enable cloudresourcemanager.googleapis.com
gcloud services enable cloudbilling.googleapis.com
gcloud services enable iam.googleapis.com
gcloud services enable servicenetworking.googleapis.com
gcloud services enable accesscontextmanager.googleapis.com
gcloud services enable sourcerepo.googleapis.com
gcloud services enable artifactregistry.googleapis.com
gcloud services enable containerregistry.googleapis.com
gcloud services enable compute.googleapis.com
gcloud services enable run.googleapis.com
gcloud services enable cloudapis.googleapis.com
gcloud services enable cloudbuild.googleapis.com
gcloud services enable storage-component.googleapis.com
gcloud services enable cloudkms.googleapis.com
gcloud services enable logging.googleapis.com
gcloud services enable cloudfunctions.googleapis.com
gcloud services enable container.googleapis.com
gcloud services enable containeranalysis.googleapis.com
gcloud services enable krmapihosting.googleapis.com
gcloud services enable cloudasset.googleapis.com
git clone https://github.com/CloudLandingZone/terraform-example-foundation.git
cd terraform-example-foundation/
git checkout pbmm
cp ../../terraform.tfvars 0-bootstrap/
rm -rf 0-bootstrap/terraform.example.tfvars 
cd 0-bootstrap/

terraform init
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ terraform plan -input=false -out bootstrap.tfplan
oot_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ terraform apply bootstrap.tfplan

1942

oogle_folder.bootstrap: Still creating... [10s elapsed]
google_folder.bootstrap: Creation complete after 11s [id=folders/310036786253]
module.required_group["billing_data_users"].google_cloud_identity_group.group: Still creating... [10s elapsed]
module.required_group["group_billing_admins"].google_cloud_identity_group.group: Creation complete after 10s [id=groups/01x0gk370zmofdp]
module.required_group["billing_data_users"].google_cloud_identity_group.group: Creation complete after 10s [id=groups/02w5ecyt1h0cokd]
module.required_group["audit_data_users"].google_cloud_identity_group.group: Creation complete after 7s [id=groups/034g0dwd2abwumb]
module.required_group["group_org_admins"].google_cloud_identity_group.group: Creation complete after 7s [id=groups/02afmg2816usjpz]
╷
│ Error: Error applying IAM policy for folder "folders/287605518452": Error setting IAM policy for folder "folders/287605518452": googleapi: Error 400: Group group_org_admins3@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 47, in resource "google_folder_iam_member" "tmp_project_creator":
│   47: resource "google_folder_iam_member" "tmp_project_creator" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_billing_admins3@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_binding.billing_creator,
│   on .terraform/modules/seed_bootstrap/main.tf line 156, in resource "google_organization_iam_binding" "billing_creator":
│  156: resource "google_organization_iam_binding" "billing_creator" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_org_admins3@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"],
│   on .terraform/modules/seed_bootstrap/main.tf line 184, in resource "google_organization_iam_member" "org_admins_group":
│  184: resource "google_organization_iam_member" "org_admins_group" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_org_admins3@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"],
│   on .terraform/modules/seed_bootstrap/main.tf line 184, in resource "google_organization_iam_member" "org_admins_group":
│  184: resource "google_organization_iam_member" "org_admins_group" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "939763342309": Error setting IAM policy for organization "939763342309": googleapi: Error 400: Group group_billing_admins3@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_billing_admin,
│   on .terraform/modules/seed_bootstrap/main.tf line 196, in resource "google_organization_iam_member" "org_billing_admin":
│  196: resource "google_organization_iam_member" "org_billing_admin" {
│ 
╵
╷
│ Error: Error applying IAM policy for folder "folders/287605518452": Error setting IAM policy for folder "folders/287605518452": googleapi: Error 400: Group group_org_admins3@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 259, in resource "google_folder_iam_member" "org_admin_service_account_user":
│  259: resource "google_folder_iam_member" "org_admin_service_account_user" {
│ 
╵
╷
│ Error: Error applying IAM policy for folder "folders/287605518452": Error setting IAM policy for folder "folders/287605518452": googleapi: Error 400: Group group_org_admins3@pbmm.landing.systems does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 267, in resource "google_folder_iam_member" "org_admin_serviceusage_consumer":
│  267: resource "google_folder_iam_member" "org_admin_serviceusage_consumer" {
│

rerun anyway

odule.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Still creating... [10s elapsed]
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creation complete after 13s [id=folders/287605518452/roles/resourcemanager.projectCreator/group:group_org_admins3@pbmm.landing.systems]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Creation complete after 13s [id=folders/287605518452/roles/iam.serviceAccountUser/group:group_org_admins3@pbmm.landing.systems]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creation complete after 17s [id=939763342309/roles/resourcemanager.organizationAdmin/group:group_org_admins3@pbmm.landing.systems]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creation complete after 18s [id=939763342309/roles/billing.user/group:group_org_admins3@pbmm.landing.systems]
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creation complete after 18s [id=939763342309/roles/billing.admin/group:group_billing_admins3@pbmm.landing.systems]

1949

module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m10s elapsed]

1952
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudkms.googleapis.com"]: Still creating... [10s 

2002
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Still creating... [10s elapsed]
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Creation complete after 11s [id=projects/prj-b-cicd-anjl/global/networks/vpc-b-cbpools/networkPeerings/servicenetworking-googleapis-com]

2004 complete
module.tf_workspace["net"].google_storage_bucket_iam_member.log_admin: Creation complete after 5s [id=b/bkt-prj-b-cicd-anjl-gcp-networks-build-logs/roles/storage.admin/serviceAccount:sa-terraform-net@prj-b-seed-06dd.iam.gserviceaccount.com]
module.build_terraform_image.null_resource.run_command[0] (local-exec): metadata:
module.build_terraform_image.null_resource.run_command[0] (local-exec):   '@type': type.googleapis.com/google.devtools.cloudbuild.v1.BuildOperationMetadata
module.build_terraform_image.null_resource.run_command[0] (local-exec):   build:
module.build_terraform_image.null_resource.run_command[0] (local-exec):     - northamerica-northeast1-docker.pkg.dev/prj-b-cicd-anjl/tf-runners/terraform:v1.3.0
module.build_terraform_image.null_resource.run_command[0] (local-exec):     - northamerica-northeast1-docker.pkg.dev/prj-b-cicd-anjl/tf-runners/terraform:v1
module.build_terraform_image.null_resource.run_command[0] (local-exec):     source:
module.build_terraform_image.null_resource.run_command[0] (local-exec):       repoSource:
module.build_terraform_image.null_resource.run_command[0] (local-exec):     sourceProvenance:
module.build_terraform_image.null_resource.run_command[0] (local-exec):         projectId: prj-b-cicd-anjl
module.build_terraform_image.null_resource.run_command[0] (local-exec):       - --tag=northamerica-northeast1-docker.pkg.dev/prj-b-cicd-anjl/tf-runners/terraform:v1
module.build_terraform_image.null_resource.run_command[0] (local-exec):       - .
module.build_terraform_image.null_resource.run_command[0] (local-exec):       name: northamerica-northeast1-docker.pkg.dev/prj-b-cicd-anjl/tf-runners/terraform:v1.3.0
  "group_billing_admins" = {
    "name" = "group_billing_admins3"
    "resource_name" = "groups/01x0gk370zmofdp"
  }
  "group_org_admins" = {
    "resource_name" = "groups/02afmg2816usjpz"
  }
  "monitoring_workspace_users" = {
    "id" = "monitoring_workspace_users3@pbmm.landing.systems"
    "name" = "monitoring_workspace_users3"
    "resource_name" = "groups/01302m920kt9k2j"
  }
})
seed_project_id = "prj-b-seed-06dd"

obriensystems commented 1 year ago

add acm role

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ export network_step_sa=$(terraform output -raw networks_step_terraform_service_account_email)
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ echo $network_step_sa
sa-terraform-net@prj-b-seed-06dd.iam.gserviceaccount.com
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ export projects_step_sa=$(terraform output -raw projects_step_terraform_service_account_email)
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ export projects_gcs_bucket_tfstate=$(terraform output -raw projects_gcs_bucket_tfstate)
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ echo "network step service account = ${network_step_sa}"
echo "projects step service account = ${projects_step_sa}"
echo "projects gcs bucket tfstate = ${projects_gcs_bucket_tfstate}"
network step service account = sa-terraform-net@prj-b-seed-06dd.iam.gserviceaccount.com
projects step service account = sa-terraform-proj@prj-b-seed-06dd.iam.gserviceaccount.com
projects gcs bucket tfstate = bkt-prj-b-seed-06dd-gcp-projects-tfstate

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ export backend_bucket=$(terraform output -raw gcs_bucket_tfstate)
echo "backend_bucket = ${backend_bucket}"
backend_bucket = bkt-prj-b-seed-tfstate-4256
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ export backend_bucket_projects=$(terraform output -raw projects_gcs_bucket_tfstate)
echo "backend_bucket_projects = ${backend_bucket_projects}"
backend_bucket_projects = bkt-prj-b-seed-06dd-gcp-projects-tfstate

cp backend.tf.example backend.tf

edit with tfstate bucket

terraform {
  backend "gcs" {
    bucket = "bkt-prj-b-seed-tfstate-4256" #"UPDATE_ME"
    prefix = "terraform/bootstrap/state"
  }
}

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation (lz-tef-pls3)$ for i in `find -name 'backend.tf'`; do sed -i "s/UPDATE_ME/${backend_bucket}/" $i; done
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation (lz-tef-pls3)$ for i in `find -name 'backend.tf'`; do sed -i "s/UPDATE_PROJECTS_BACKEND/${backend_bucket_projects}/" $i; done

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation (lz-tef-pls3)$ cd 0-bootstrap/
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ 

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ terraform init

Initializing the backend...
Acquiring state lock. This may take a few moments...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "gcs" backend. No existing state was found in the newly
  configured "gcs" backend. Do you want to copy this state to the new "gcs"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: 

yes

- Reusing previous version of hashicorp/null from the dependency lock file
- Using previously-installed hashicorp/google v4.63.1
- Using previously-installed hashicorp/time v0.9.1
- Using previously-installed hashicorp/random v3.5.1
- Using previously-installed hashicorp/google-beta v4.63.1
- Using previously-installed hashicorp/external v2.3.1
- Using previously-installed hashicorp/null v3.2.1

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

section 12
 terraform plan -input=false -out bootstrap.tfplan

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

step 13
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation/0-bootstrap (lz-tef-pls3)$ cd ../..
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone (lz-tef-pls3)$ ls
terraform-example-foundation

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone (lz-tef-pls3)$ gcloud source repos clone gcp-policies --project=${cloudbuild_project_id}
Cloning into '/home/root_/lz-tef-pls3/CloudLandingZone/gcp-policies'...
warning: You appear to have cloned an empty repository.
Project [prj-b-cicd-anjl] repository [gcp-policies] was cloned to [/home/root_/lz-tef-pls3/CloudLandingZone/gcp-policies].

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone (lz-tef-pls3)$ cd gcp-policies
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-policies (lz-tef-pls3)$ git checkout -b main
Switched to a new branch 'main'

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-policies (lz-tef-pls3)$ ls
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-policies (lz-tef-pls3)$ cp -RT ../terraform-example-foundation/policy-library/ .
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-policies (lz-tef-pls3)$ ls
lib  policies

step 14

git add .
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-policies (lz-tef-pls3)$ git commit -m 'Initialize policy library repo'
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-policies (lz-tef-pls3)$ git push --set-upstream origin main
Enumerating objects: 118, done.
Counting objects: 100% (118/118), done.
Delta compression using up to 4 threads
Compressing objects: 100% (118/118), done.
Writing objects: 100% (118/118), 72.54 KiB | 2.79 MiB/s, done.
Total 118 (delta 87), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (87/87)
To https://source.developers.google.com/p/prj-b-cicd-anjl/r/gcp-policies
 * [new branch]      main -> main
Branch 'main' set up to track remote branch 'main' from 'origin'.

check CSR

section 16
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-policies (lz-tef-pls3)$ cd ..
root_@cloudshell:~/lz-tef-pls3/CloudLandingZone (lz-tef-pls3)$ gcloud source repos clone gcp-bootstrap --project=${cloudbuild_project_id}
Cloning into '/home/root_/lz-tef-pls3/CloudLandingZone/gcp-bootstrap'...
warning: You appear to have cloned an empty repository.
Project [prj-b-cicd-anjl] repository [gcp-bootstrap] was cloned to [/home/root_/lz-tef-pls3/CloudLandingZone/gcp-bootstrap].

  281  cd ..
  282  gcloud source repos clone gcp-bootstrap --project=${cloudbuild_project_id}
  283  cd gcp-bootstrap
  284  git checkout -b plan
  285  mkdir -p envs/shared
  286  cp -RT ../terraform-example-foundation/0-bootstrap/ ./envs/shared
  287  cp ../terraform-example-foundation/build/cloudbuild-tf-* .
  288  cp ../terraform-example-foundation/build/tf-wrapper.sh .
  289  chmod 755 ./tf-wrapper.sh
  290  git add .
  291  git commit -m 'Initialize bootstrap repo'

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/gcp-bootstrap (lz-tef-pls3)$ git push --set-upstream origin plan
Enumerating objects: 54, done.
Counting objects: 100% (54/54), done.
Delta compression using up to 4 threads
Compressing objects: 100% (52/52), done.
Writing objects: 100% (54/54), 399.79 KiB | 10.25 MiB/s, done.
Total 54 (delta 14), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (14/14)
To https://source.developers.google.com/p/prj-b-cicd-anjl/r/gcp-bootstrap
 * [new branch]      plan -> plan
Branch 'plan' set up to track remote branch 'plan' from 'origin'.

next step 17 of 1-org
https://github.com/CloudLandingZone/terraform-example-foundation/blob/master/1-org/README.md

obriensystems commented 1 year ago

1-org pending

root_@cloudshell:~/lz-tef-pls3/CloudLandingZone/terraform-example-foundation (lz-tef-pls3)$ git status
On branch pbmm
Your branch is up to date with 'origin/pbmm'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    0-bootstrap/terraform.example.tfvars
        modified:   1-org/envs/shared/backend.tf
        modified:   2-environments/envs/development/backend.tf
        modified:   2-environments/envs/non-production/backend.tf
        modified:   2-environments/envs/production/backend.tf
        modified:   3-networks-dual-svpc/envs/development/backend.tf
        modified:   3-networks-dual-svpc/envs/non-production/backend.tf
        modified:   3-networks-dual-svpc/envs/production/backend.tf
        modified:   3-networks-dual-svpc/envs/shared/backend.tf
        modified:   3-networks-hub-and-spoke/envs/development/backend.tf
        modified:   3-networks-hub-and-spoke/envs/non-production/backend.tf
        modified:   3-networks-hub-and-spoke/envs/production/backend.tf
        modified:   3-networks-hub-and-spoke/envs/shared/backend.tf
        modified:   4-projects/business_unit_1/development/backend.tf
        modified:   4-projects/business_unit_1/non-production/backend.tf
        modified:   4-projects/business_unit_1/production/backend.tf
        modified:   4-projects/business_unit_1/shared/backend.tf
        modified:   4-projects/business_unit_2/development/backend.tf
        modified:   4-projects/business_unit_2/non-production/backend.tf
        modified:   4-projects/business_unit_2/production/backend.tf
        modified:   4-projects/business_unit_2/shared/backend.tf

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        0-bootstrap/backend.tf
        0-bootstrap/bootstrap.tfplan

fmichaelobrien commented 1 year ago

Assigned to fmichaelobrien

github-actions[bot] commented 11 months ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

fmichaelobrien commented 6 months ago

restarting for TEF V4

obriensystems commented 5 months ago

Got past 0-bootstrap - working 1-org below

terraform-google-modules / terraform-example-foundation

Document example deployment of the TEF with optional PubSec overlay via PBMM specific Landing Zone #940

1133

TL;DR

Work Items

WI0: deploy the TEF (Selectively comment business user 2 projects - TB under quota)

WI1: Diff between TEF and PBMM repos - (code/deployment) using both deployments

WI2: Selective rebase

Terraform Resources

Detailed design

Additional information

1133