20240306: - moving to the TEF https://github.com/terraform-google-modules/terraform-example-foundation under https://github.com/terraform-google-modules/terraform-example-foundation/issues/1133 review previous: https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/243 review previous issues: https://github.com/terraform-google-modules/terraform-example-foundation/issues/940

see ongoing list of so far minor issues we can move on from

20240304:1200 - replaced by https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/351

Introduce a 2024 light LZ - an IaaS NGFW based landing zone based on TEF V4 reuse the kcc repo in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/870

Same as https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/824 A terraform version of the landing zone Introduce a 2024 light LZ - an IaaS NGFW based landing zone based on TEF V4

branch https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/tree/345-landing-zone-fortigate-light org: olapp

There is a requirement for a mixed day 2 deployment strategy around kubernetes and gcloud or terraform. The following packages will shadow the existing kubernetes yaml config - but in terraform

core-landing-zone - inventory
client-setup - inventory
client-landing-zone - inventory
client-project-setup
gatekeeper-policies - optional https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/gatekeeper-policies
project/hub-env - optional for non-firewall-plus https://cloud.google.com/blog/products/identity-security/introducing-google-cloud-firewall-plus-with-intrusion-prevention IaaS security via Fortinet https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/project/hub-env/README.md#resources

see

graph LR;
    style LZV2 fill:#44f,stroke:#f66,stroke-width:2px,color:#fff,stroke-dasharray: 5 5
    %% mapped and documented
    project/hub-env-->core-landing-zone;
    client-setup;
    client-setup-->dns-project;
    client-setup-->kcc-management-project;
    client-landing-zone-->client-setup;
    client-project-setup-->client-landing-zone;
    client-project-setup-->client-management-project;
    gatekeeper-policies;

    kcc-management-project;
    core-landing-zone-->kcc-management-project;
    dns-project-->core-landing-zone;
    logging-project-->core-landing-zone;
    client-management-project-->client-setup;
    host-project-->client-landing-zone;

mermaid - diagrams as code

Structure

Based on parts of the following repositories

some of this one but these models are customized TEF V1 modules
TEF V4
https://github.com/terraform-google-modules/terraform-example-foundation
Fortinet
https://github.com/40net-cloud/fortinet-gcp-solutions/tree/master/FortiGate
https://github.com/fortinet/fortigate-tutorial-gcp/tree/main/terraform
we have parts of our base and client architecture as a reference here

Architecture

https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Architecture
Base Landing Zone

Merged with Fortigate LB sandwich cluster - re-peer with above

see existing LZ V2 architecture that we will generally model off as a defacto port from KCC LZ V2
core-landing-zone - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/core-landing-zone
client-setup - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/client-setup
client-landing-zone - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/client-landing-zone
client-project-setup - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/client-project-setup
guardrails - https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/guardrails

Work Items

Fortigate based HA cluster with 4 VPCs
client VPC that is peered with the NGFW southbound VPC
gitops deploy pipeline based on github or ADO with terraform provider token based authentication to ADO for example

Updates Requested

git symlinks removed
extra projects like kms optional - see https://github.com/terraform-google-modules/terraform-example-foundation/issues/1142
project factory creates gsuite group

Critical -1.3 terraform needs to goto 1.7 for PBR (link) - without PBR we dont have PBMM microsegmenation https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/Dockerfile#L18 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/issues/854 https://cloud.google.com/vpc/docs/policy-based-routes https://medium.com/google-cloud/why-policy-based-routing-is-a-game-changer-f4c6a7badccb https://codelabs.developers.google.com/codelabs/cloudnet-pbr#0

options upgrade to 1.7 (need retrofit for example on OPTIONAL), run pbr or the entire tef as local (no docker container), gcloud of pbr is problematic

4 types of kb

unknown unknowns

plan up/clean/modify

1 - bring up the full system - as is
2 - remove unused services/costs - global search on each dependency tree - or try/flag wrapper
typescript like issues with webstream? for tf - or like reference calls in java
vpc flow lots off - save $x x 100/m - a lot of discussion
3 - attach ado and fg
4 - check finops
5 - simplify deployment local - to prep for ADO

Iterations

0 - validate: get untouched TEF up (default CB running TF 1.3) - nprod/prod-aka-restricted) 1 - try 1.5.6 docker change - hopefully no deprecation issues like https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/224

check TF version checks >= 1.3 - see monitoring in 0-bootstrap
is module version 1.14 (sept 2022) tied to a terraform release like 1.3.0? need to check 2 - check/remove symlinks if they are not auto-imported in ADO https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-hub-and-spoke/envs/development/common.auto.tfvars as we did in PBMM https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/environments/bootstrap/bootstrap.sh#L453

3 - comment out modules in each section that are not in use (dedicated interconnect ie:) - to avoid TF 1.3+ fix on sections we will remove 4 - localizing: deploy via local TF (1.5.6 min) no CB docker container deploy output is stripped down working local under TF 1.6+ 5 - refactoring of hub-spoke network - prep for fg 6 - fortinet integration 7 - prep/modify for security review GCP local and FG - prep for sec team review

review modules to comment/remove

all interconnect, dns policies related to interconnect

later review managed Terraform https://cloud.google.com/infrastructure-manager/docs/overview

integrating as a start the original tutorial (not yet the larger one in 40net) - to start as the simplest base for the client

https://github.com/fortinet/fortigate-tutorial-gcp/tree/main/terraform later with back references https://github.com/40net-cloud/fortinet-gcp-solutions/tree/master/FortiGate

in branch

git@github.com:fortinet/fortigate-tutorial-gcp.git

under https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/pull/347

Procedure for https://github.com/fortinet/fortigate-tutorial-gcp/tree/main/terraform/day0

create project, enable services

michael@cloudshell:~/fortigate-terraform-olapp (fortigate-terraform-olapp)$ cd fortigate-tutorial-gcp/
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp (fortigate-terraform-olapp)$ ls
deployment-manager  docs  gcloud  README.md  service_account_create.sh  terraform
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp (fortigate-terraform-olapp)$ gcloud services enable compute.googleapis.com
gcloud services enable container.googleapis.com
Operation "operations/acf.p2-460528556276-ff26730d-c0d2-41a0-a0fb-f5d5f072a8da" finished successfully.
Operation "operations/acf.p2-460528556276-f9e45108-3ba8-4c93-94c2-6bb6fcd7733f" finished successfully.

create service account

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ chmod 777 ../../service_account_create.sh 
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ ../../service_account_create.sh 
Your active configuration is: [cloudshell-31611]
Creating FortigateSdnReader role in project fortigate-terraform-olapp...
Created role [FortigateSdnReader].
etag: BwYSwu_xMUk=
includedPermissions:
- compute.instances.list
- compute.zones.list
- container.clusters.list
- container.nodes.list
- container.pods.list
- container.services.list
name: projects/fortigate-terraform-olapp/roles/FortigateSdnReader
stage: ALPHA
title: FortiGate SDN Connector Role (read-only)
Creating new service account (FortiGate SDN Connector)...
Created service account [fortigatesdn-ro].
Granting fortigatesdn-ro service account access to project fortigate-terraform-olapp...
Updated IAM policy for project [fortigate-terraform-olapp].
bindings:
- members:
  - serviceAccount:fortigatesdn-ro@fortigate-terraform-olapp.iam.gserviceaccount.com
  role: projects/fortigate-terraform-olapp/roles/FortigateSdnReader
- members:
  - serviceAccount:service-460528556276@compute-system.iam.gserviceaccount.com
  role: roles/compute.serviceAgent
- members:
  - serviceAccount:service-460528556276@container-engine-robot.iam.gserviceaccount.com
  role: roles/container.serviceAgent
- members:
  - serviceAccount:service-460528556276@containerregistry.iam.gserviceaccount.com
  role: roles/containerregistry.ServiceAgent
- members:
  - serviceAccount:460528556276-compute@developer.gserviceaccount.com
  - serviceAccount:460528556276@cloudservices.gserviceaccount.com
  role: roles/editor
- members:
  - user:michael@obrienlabs.app
  role: roles/owner
- members:
  - serviceAccount:service-460528556276@gcp-sa-pubsub.iam.gserviceaccount.com
  role: roles/pubsub.serviceAgent
etag: BwYSwvAqT8I=
version: 1
serviceAccount:fortigatesdn-ro@fortigate-terraform-olapp.iam.gserviceaccount.com
Service account created succesfully

move license files (make sure VMs are not up with them)

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ cp ../../../../FGVM8VTM2400018*.lic .
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ ls 
FGVM8VTM24000185.lic  FGVM8VTM24000186.lic  main.tf  outputs.tf  README.md  terraform.tfvars  variables.tf

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ mv FGVM8VTM24000186.lic lic2.lic
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ mv FGVM8VTM24000185.lic lic1.lic

adjust terraform.tfvars for your project, region

GCP_PROJECT = "fortigate-terraform-olapp"
GCE_REGION = "northamerica-northeast1"
prefix = "fgt-"

change fortigate image from 7.0 to 7.4 in line 5 of main.tf in the fgcp-ha-ap-lb module

  family          = "fortigate-74-byol"

Run

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ terraform --version
Terraform v1.7.2
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.7.4. You can update by downloading from https://www.terraform.io/downloads.html

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ terraform init

Initializing the backend...
Initializing modules...
- fortigates in ../modules/fgcp-ha-ap-lb
- sample_networks in ../modules/sample-networks

Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Finding latest version of hashicorp/http...
- Finding latest version of hashicorp/google-beta...
- Finding latest version of fortinetdev/fortios...
- Finding latest version of hashicorp/random...
- Installing hashicorp/google v5.18.0...
- Installed hashicorp/google v5.18.0 (signed by HashiCorp)
- Installing hashicorp/http v3.4.2...
- Installed hashicorp/http v3.4.2 (signed by HashiCorp)
- Installing hashicorp/google-beta v5.18.0...
- Installed hashicorp/google-beta v5.18.0 (signed by HashiCorp)
- Installing fortinetdev/fortios v1.19.0...
- Installed fortinetdev/fortios v1.19.0 (signed by a HashiCorp partner, key ID 31ECDEBCB7DAB5F0)
- Installing hashicorp/random v3.6.0...
- Installed hashicorp/random v3.6.0 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ terraform plan --out day0.plan

Plan: 35 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + api-key          = (known after apply)
  + default_password = (known after apply)
  + fgt-mgmt-eips    = [
      + (known after apply),
      + (known after apply),
    ]
  + fgt_umigs        = [
      + (known after apply),
      + (known after apply),
    ]
  + health_check     = (known after apply)
  + ilb              = (known after apply)
  + internal_subnet  = (known after apply)
  + internal_vpc     = (known after apply)
  + prefix           = "fgt-"
  + project          = "fortigate-terraform-olapp"
  + region           = "northamerica-northeast1"
╷
│ Warning: Deprecated attribute
│ 
│   on main.tf line 19, in module "fortigates":
│   19:   admin_acl       = ["${data.http.my_ip.body}/32"]

ichael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ terraform apply day0.plan
module.sample_networks.google_compute_network.vpcs[3]: Creating...
module.sample_networks.google_compute_network.vpcs[0]: Creating...
module.sample_networks.google_compute_network.vpcs[1]: Creating...
module.sample_networks.google_compute_network.vpcs[2]: Creating...

1017

module.sample_networks.google_compute_network.vpcs[2]: Still creating... [10s elapsed]
module.sample_networks.google_compute_network.vpcs[0]: Still creating... [10s elapsed]
module.sample_networks.google_compute_network.vpcs[1]: Still creating... [10s elapsed]
module.sample_networks.google_compute_network.vpcs[3]: Still creating... [10s elapsed]
module.sample_networks.google_compute_network.vpcs[2]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-hasync]
module.sample_networks.google_compute_network.vpcs[1]: Still creating... [20s elapsed]
module.sample_networks.google_compute_network.vpcs[0]: Still creating... [20s elapsed]
module.sample_networks.google_compute_network.vpcs[3]: Still creating... [20s elapsed]
module.sample_networks.google_compute_network.vpcs[0]: Creation complete after 22s [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-external]
module.sample_networks.google_compute_network.vpcs[1]: Creation complete after 22s [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-internal]
module.sample_networks.google_compute_network.vpcs[3]: Creation complete after 22s [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-mgmt]
module.sample_networks.google_compute_subnetwork.subnets[2]: Creating...
module.sample_networks.google_compute_subnetwork.subnets[0]: Creating...
module.sample_networks.google_compute_subnetwork.subnets[1]: Creating...
module.sample_networks.google_compute_subnetwork.subnets[3]: Creating...
module.sample_networks.google_compute_subnetwork.subnets[0]: Still creating... [10s elapsed]
module.sample_networks.google_compute_subnetwork.subnets[2]: Still creating... [10s elapsed]
module.sample_networks.google_compute_subnetwork.subnets[1]: Still creating... [10s elapsed]
module.sample_networks.google_compute_subnetwork.subnets[3]: Still creating... [10s elapsed]
module.sample_networks.google_compute_subnetwork.subnets[3]: Creation complete after 13s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-mgmt]
module.sample_networks.google_compute_subnetwork.subnets[1]: Creation complete after 14s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal]
module.sample_networks.google_compute_subnetwork.subnets[2]: Still creating... [20s elapsed]
module.sample_networks.google_compute_subnetwork.subnets[0]: Still creating... [20s elapsed]
module.sample_networks.google_compute_subnetwork.subnets[2]: Creation complete after 24s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-hasync]
module.sample_networks.google_compute_subnetwork.subnets[0]: Creation complete after 25s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-external]
module.fortigates.data.google_compute_image.fgt_image: Reading...
module.fortigates.data.google_compute_zones.zones_in_region: Reading...
module.fortigates.data.google_compute_default_service_account.default: Reading...
module.fortigates.data.google_compute_subnetwork.subnets[3]: Reading...
module.fortigates.data.google_compute_subnetwork.subnets[1]: Reading...
module.fortigates.data.google_compute_subnetwork.subnets[2]: Reading...
module.fortigates.google_compute_address.mgmt_pub[1]: Creating...
module.fortigates.google_compute_address.mgmt_pub[0]: Creating...
module.fortigates.random_string.api_key: Creating...
module.fortigates.google_compute_region_health_check.health_check: Creating...
module.fortigates.random_string.api_key: Creation complete after 0s [id=YAZEDTw2B8fQEw5VoOdS1KNVjjTwDH]
module.fortigates.data.google_compute_subnetwork.subnets[0]: Reading...
module.fortigates.data.google_compute_subnetwork.subnets[3]: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-mgmt]
module.fortigates.google_compute_firewall.allow-mgmt: Creating...
module.fortigates.data.google_compute_subnetwork.subnets[1]: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal]
module.fortigates.data.google_compute_subnetwork.subnets[2]: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-hasync]
module.fortigates.data.google_compute_zones.zones_in_region: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1]
module.fortigates.data.google_compute_default_service_account.default: Read complete after 0s [id=projects/fortigate-terraform-olapp/serviceAccounts/460528556276-compute@developer.gserviceaccount.com]
module.fortigates.data.google_compute_image.fgt_image: Read complete after 0s [id=projects/fortigcp-project-001/global/images/fortinet-fgt-743-20240208-001-w-license]
module.fortigates.google_compute_address.mgmt_priv[1]: Creating...
module.fortigates.google_compute_address.mgmt_priv[0]: Creating...
module.fortigates.data.google_compute_subnetwork.subnets[0]: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-external]
module.fortigates.google_compute_firewall.allow-hasync: Creating...
module.fortigates.google_compute_firewall.allow-port2: Creating...
module.fortigates.google_compute_address.int_priv[0]: Creating...
module.fortigates.google_compute_address.int_priv[1]: Creating...
module.fortigates.google_compute_address.mgmt_pub[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_region_health_check.health_check: Still creating... [10s elapsed]
module.fortigates.google_compute_firewall.allow-mgmt: Still creating... [10s elapsed]
module.fortigates.google_compute_address.mgmt_priv[1]: Still creating... [10s elapsed]
module.fortigates.google_compute_address.mgmt_priv[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_firewall.allow-hasync: Still creating... [10s elapsed]
module.fortigates.google_compute_firewall.allow-port2: Still creating... [10s elapsed]
module.fortigates.google_compute_address.int_priv[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_address.int_priv[1]: Still creating... [10s elapsed]
module.fortigates.google_compute_region_health_check.health_check: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/healthChecks/fgthealthcheck-http8008-na-northeast1]
module.fortigates.google_compute_address.ilb: Creating...
module.fortigates.google_compute_address.mgmt_pub[1]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgteip2-mgmt-na-northeast1]
module.fortigates.google_compute_address.hasync_priv[1]: Creating...
module.fortigates.google_compute_firewall.allow-mgmt: Creation complete after 11s [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-mgmt-allow-admin]
module.fortigates.google_compute_address.hasync_priv[0]: Creating...
module.fortigates.google_compute_address.mgmt_pub[0]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgteip1-mgmt-na-northeast1]
module.fortigates.google_compute_router.nat_router: Creating...
module.fortigates.google_compute_address.int_priv[1]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-trust-na-northeast1]
module.fortigates.google_compute_address.ext_priv[0]: Creating...
module.fortigates.google_compute_address.int_priv[0]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-trust-na-northeast1]
module.fortigates.google_compute_firewall.allow-port1: Creating...
module.fortigates.google_compute_address.mgmt_priv[1]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-mgmt-na-northeast1]
module.fortigates.google_compute_address.ext_priv[1]: Creating...
module.fortigates.google_compute_address.mgmt_priv[0]: Creation complete after 12s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-mgmt-na-northeast1]
module.fortigates.google_compute_disk.logdisk[0]: Creating...
module.fortigates.google_compute_firewall.allow-port2: Creation complete after 12s [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-trust-allowall]
module.fortigates.google_compute_disk.logdisk[1]: Creating...
module.fortigates.google_compute_firewall.allow-hasync: Creation complete after 12s [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-hasync-allow-fgt]
module.fortigates.google_compute_address.hasync_priv[1]: Still creating... [10s elapsed]
module.fortigates.google_compute_address.hasync_priv[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_router.nat_router: Still creating... [10s elapsed]
module.fortigates.google_compute_address.ext_priv[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_firewall.allow-port1: Still creating... [10s elapsed]
module.fortigates.google_compute_address.ext_priv[1]: Still creating... [10s elapsed]
module.fortigates.google_compute_disk.logdisk[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_disk.logdisk[1]: Still creating... [10s elapsed]
module.fortigates.google_compute_address.hasync_priv[1]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-hasync-na-northeast1]
module.fortigates.google_compute_router.nat_router: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/routers/fgtcr-cloudnat-na-northeast1]
module.fortigates.google_compute_address.ilb: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip-ilb-trust-na-northeast1]
module.fortigates.google_compute_address.hasync_priv[0]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-hasync-na-northeast1]
module.fortigates.google_compute_router_nat.cloud_nat: Creating...
module.fortigates.google_compute_address.ext_priv[1]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-untrust-na-northeast1]
module.fortigates.google_compute_address.ext_priv[0]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-untrust-na-northeast1]
module.fortigates.google_compute_firewall.allow-port1: Creation complete after 12s [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-untrust-allowall]
module.fortigates.google_compute_disk.logdisk[0]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/disks/fgtdisk-logdisk1-na-northeast1-a]
module.fortigates.google_compute_disk.logdisk[1]: Creation complete after 11s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/disks/fgtdisk-logdisk2-na-northeast1-b]
module.fortigates.google_compute_instance.fgt-vm[0]: Creating...
module.fortigates.google_compute_instance.fgt-vm[1]: Creating...
odule.fortigates.google_compute_router_nat.cloud_nat: Still creating... [10s elapsed]
module.fortigates.google_compute_instance.fgt-vm[1]: Still creating... [10s elapsed]
module.fortigates.google_compute_instance.fgt-vm[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_router_nat.cloud_nat: Creation complete after 11s [id=fortigate-terraform-olapp/northamerica-northeast1/fgtcr-cloudnat-na-northeast1/fgtnat-cloudnat-na-northeast1]
module.fortigates.google_compute_instance.fgt-vm[1]: Creation complete after 14s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instances/fgtvm2-na-northeast1-b]
module.fortigates.google_compute_instance.fgt-vm[0]: Still creating... [20s elapsed]

odule.fortigates.google_compute_instance.fgt-vm[0]: Creation complete after 23s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instances/fgtvm1-na-northeast1-a]
module.fortigates.google_compute_instance_group.fgt-umigs[0]: Creating...
module.fortigates.google_compute_instance_group.fgt-umigs[1]: Creating...
module.fortigates.google_compute_instance_group.fgt-umigs[0]: Still creating... [10s elapsed]
module.fortigates.google_compute_instance_group.fgt-umigs[1]: Still creating... [10s elapsed]
module.fortigates.google_compute_instance_group.fgt-umigs[1]: Creation complete after 12s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instanceGroups/fgtumig1-na-northeast1-b]
module.fortigates.google_compute_instance_group.fgt-umigs[0]: Creation complete after 12s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instanceGroups/fgtumig0-na-northeast1-a]
module.fortigates.google_compute_region_backend_service.ilb_bes: Creating...

module.fortigates.google_compute_region_backend_service.ilb_bes: Still creating... [20s elapsed]
module.fortigates.google_compute_region_backend_service.ilb_bes: Creation complete after 22s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/backendServices/fgtbes-ilb-trust-na-northeast1]
module.fortigates.google_compute_forwarding_rule.ilb_fwd_rule: Creating...
module.fortigates.google_compute_forwarding_rule.ilb_fwd_rule: Still creating... [10s elapsed]
module.fortigates.google_compute_forwarding_rule.ilb_fwd_rule: Still creating... [20s elapsed]
module.fortigates.google_compute_forwarding_rule.ilb_fwd_rule: Creation complete after 21s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/fgtfwdrule-ilb-trust-na-northeast1]
╷
│ Warning: Deprecated attribute
│ 
│   on main.tf line 19, in module "fortigates":
│   19:   admin_acl       = ["${data.http.my_ip.body}/32"]
│ 
│ The attribute "body" is deprecated. Refer to the provider documentation for details.
│ 
│ (and one more similar warning elsewhere)
╵

Apply complete! Resources: 35 added, 0 changed, 0 destroyed.

Outputs:

api-key = "YAZEDTw2B8fQEw5VoOdS1KNVjjTwDH"
default_password = "348639094312086080"
fgt-mgmt-eips = [
  "34.152.20.239",
  "34.95.52.25",
]
fgt_umigs = [
  "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instanceGroups/fgtumig0-na-northeast1-a",
  "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instanceGroups/fgtumig1-na-northeast1-b",
]
health_check = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/healthChecks/fgthealthcheck-http8008-na-northeast1"
ilb = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/fgtfwdrule-ilb-trust-na-northeast1"
internal_subnet = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal"
internal_vpc = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/global/networks/fgt-vpc-internal"
prefix = "fgt-"
project = "fortigate-terraform-olapp"
region = "northamerica-northeast1"

https://34.152.20.239/

triaging against my working gcloud implementation instance group not set

actually took 10 min for MIG to come up

connectivity test on gcloud reference

gcloud tests OK

terraform does not

Issue is no external load balancer in terraform like there is in gcloud

before Destroying terraform resources - run day1 as instructions for fortigate access are after this

https://github.com/fortinet/fortigate-tutorial-gcp/tree/main/terraform/day1

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ cd ../day1/
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform init

Initializing the backend...
Initializing modules...
- inbound in ../modules/usecases/inbound-ns
- outbound in ../modules/usecases/outbound-ns
- peer1 in ../modules/usecases/spoke-vpc
- peer2 in ../modules/usecases/spoke-vpc

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform
- Finding latest version of hashicorp/google-beta...
- Finding latest version of fortinetdev/fortios...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google-beta v5.18.0...
- Installed hashicorp/google-beta v5.18.0 (signed by HashiCorp)
- Installing fortinetdev/fortios v1.19.0...
- Installed fortinetdev/fortios v1.19.0 (signed by a HashiCorp partner, key ID 31ECDEBCB7DAB5F0)
- Installing hashicorp/google v5.18.0...
- Installed hashicorp/google v5.18.0 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

plan 1813:

ichael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform plan -out tf.plan
data.terraform_remote_state.base: Reading...
data.terraform_remote_state.base: Read complete after 0s
module.peer1.data.google_compute_subnetwork.hub: Reading...
data.google_compute_zones.local: Reading...
module.peer1.data.google_compute_subnetwork.hub: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal]
data.google_compute_zones.local: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1]
1814

terraform plan is hanging

had one of the VMs down on purpose due to shared license - restarting after stopping one of the gcloud ones

1817 both VMs up

stopping hung plan
Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...

Stopping operation...

ctrl-c again
C
Two interrupts received. Exiting immediately. Note that data loss may have occurred.

╷
│ Error: operation canceled
│ 
│ 

1821

rerun terraform plan on day0 - 3 timestamp updates

ichael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform (fortigate-terraform-olapp)$ cd day0
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ terraform plan -out tf.plan
data.http.my_ip: Reading...
data.http.my_ip: Read complete after 0s [id=http://api.ipify.org]
data.google_service_account.fgt: Reading...
module.sample_networks.google_compute_network.vpcs[3]: Refreshing state... [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-mgmt]
module.sample_networks.google_compute_network.vpcs[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-external]
module.sample_networks.google_compute_network.vpcs[2]: Refreshing state... [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-hasync]
module.sample_networks.google_compute_network.vpcs[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/global/networks/fgt-vpc-internal]
data.google_service_account.fgt: Read complete after 0s [id=projects/fortigate-terraform-olapp/serviceAccounts/fortigatesdn-ro@fortigate-terraform-olapp.iam.gserviceaccount.com]
module.sample_networks.google_compute_subnetwork.subnets[3]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-mgmt]
module.sample_networks.google_compute_subnetwork.subnets[2]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-hasync]
module.sample_networks.google_compute_subnetwork.subnets[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal]
module.sample_networks.google_compute_subnetwork.subnets[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-external]
module.fortigates.random_string.api_key: Refreshing state... [id=YAZEDTw2B8fQEw5VoOdS1KNVjjTwDH]
module.fortigates.data.google_compute_default_service_account.default: Reading...
module.fortigates.data.google_compute_subnetwork.subnets[0]: Reading...
module.fortigates.data.google_compute_zones.zones_in_region: Reading...
module.fortigates.google_compute_address.mgmt_pub[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgteip2-mgmt-na-northeast1]
module.fortigates.data.google_compute_subnetwork.subnets[3]: Reading...
module.fortigates.data.google_compute_subnetwork.subnets[1]: Reading...
module.fortigates.data.google_compute_subnetwork.subnets[2]: Reading...
module.fortigates.data.google_compute_image.fgt_image: Reading...
module.fortigates.google_compute_region_health_check.health_check: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/healthChecks/fgthealthcheck-http8008-na-northeast1]
module.fortigates.google_compute_address.mgmt_pub[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgteip1-mgmt-na-northeast1]
module.fortigates.data.google_compute_zones.zones_in_region: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1]
module.fortigates.google_compute_disk.logdisk[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/disks/fgtdisk-logdisk2-na-northeast1-b]
module.fortigates.data.google_compute_subnetwork.subnets[2]: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-hasync]
module.fortigates.google_compute_disk.logdisk[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/disks/fgtdisk-logdisk1-na-northeast1-a]
module.fortigates.data.google_compute_default_service_account.default: Read complete after 1s [id=projects/fortigate-terraform-olapp/serviceAccounts/460528556276-compute@developer.gserviceaccount.com]
module.fortigates.data.google_compute_subnetwork.subnets[1]: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal]
module.fortigates.data.google_compute_subnetwork.subnets[3]: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-mgmt]
module.fortigates.data.google_compute_subnetwork.subnets[0]: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-external]
module.fortigates.data.google_compute_image.fgt_image: Read complete after 1s [id=projects/fortigcp-project-001/global/images/fortinet-fgt-743-20240208-001-w-license]
module.fortigates.google_compute_router.nat_router: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/routers/fgtcr-cloudnat-na-northeast1]
module.fortigates.google_compute_address.hasync_priv[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-hasync-na-northeast1]
module.fortigates.google_compute_address.hasync_priv[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-hasync-na-northeast1]
module.fortigates.google_compute_firewall.allow-port2: Refreshing state... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-trust-allowall]
module.fortigates.google_compute_address.int_priv[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-trust-na-northeast1]
module.fortigates.google_compute_address.int_priv[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-trust-na-northeast1]
module.fortigates.google_compute_firewall.allow-port1: Refreshing state... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-untrust-allowall]
module.fortigates.google_compute_address.ilb: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip-ilb-trust-na-northeast1]
module.fortigates.google_compute_address.ext_priv[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-untrust-na-northeast1]
module.fortigates.google_compute_address.ext_priv[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-untrust-na-northeast1]
module.fortigates.google_compute_firewall.allow-hasync: Refreshing state... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-hasync-allow-fgt]
module.fortigates.google_compute_firewall.allow-mgmt: Refreshing state... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-mgmt-allow-admin]
module.fortigates.google_compute_address.mgmt_priv[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip1-mgmt-na-northeast1]
module.fortigates.google_compute_address.mgmt_priv[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgtip2-mgmt-na-northeast1]
module.fortigates.google_compute_router_nat.cloud_nat: Refreshing state... [id=fortigate-terraform-olapp/northamerica-northeast1/fgtcr-cloudnat-na-northeast1/fgtnat-cloudnat-na-northeast1]
module.fortigates.google_compute_instance.fgt-vm[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instances/fgtvm2-na-northeast1-b]
module.fortigates.google_compute_instance.fgt-vm[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instances/fgtvm1-na-northeast1-a]
module.fortigates.google_compute_instance_group.fgt-umigs[0]: Refreshing state... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instanceGroups/fgtumig0-na-northeast1-a]
module.fortigates.google_compute_instance_group.fgt-umigs[1]: Refreshing state... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instanceGroups/fgtumig1-na-northeast1-b]
module.fortigates.google_compute_region_backend_service.ilb_bes: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/backendServices/fgtbes-ilb-trust-na-northeast1]
module.fortigates.google_compute_forwarding_rule.ilb_fwd_rule: Refreshing state... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/fgtfwdrule-ilb-trust-na-northeast1]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.fortigates.google_compute_firewall.allow-mgmt will be updated in-place
  ~ resource "google_compute_firewall" "allow-mgmt" {
        id                      = "projects/fortigate-terraform-olapp/global/firewalls/fgtfw-mgmt-allow-admin"
        name                    = "fgtfw-mgmt-allow-admin"
      ~ source_ranges           = [
          - "35.237.152.177/32",
          + "34.74.37.83/32",
        ]
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.fortigates.google_compute_instance.fgt-vm[0] will be updated in-place
  ~ resource "google_compute_instance" "fgt-vm" {
        id                   = "projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instances/fgtvm1-na-northeast1-a"
      ~ metadata             = {
          - "ssh-keys"  = <<-EOT
                michael:ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMzgqUbuoTGurR7Ze7ShZnxvsW+ZXWnD/PVZOmocDd5or0C2kWYge2rTPaIi9J2dc0hFqLn6vVs47in6pIpoGvo= google-ssh {"userName":"michael@obrienlabs.app","expireOn":"2024-03-03T15:31:27+0000"}
                michael:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAHsrk48K2tnrLp3JdtFCExE/Xp8JCKIXhGX4t8z7b7iZtrjYZ8yYGLfGhhmbm8Rm+C1vhZQYzesg3L0XOUrcTV6lnICgoIn1UwIBRoX3BA7+BQVS2g/FDvrwH1wG3dvePP5mL8SM2ZDET184CblypWSqIQjHAqUGWRon28CIpmByG/zWas7J/7RdPZdMk5Y95FKW//YnQPo95tAynUYe70T6avRUOFooP12dmpSvRptsxmXiEZ6ZO0o+ck7P4yaibh1HrYhBK3/1uw5npPzUmD06cHqMhdcywsBsky/e4RkXs3pDLsUXQVc/fhJm8xse30SvcaU8ei2wKtaoh2neWfc= google-ssh {"userName":"michael@obrienlabs.app","expireOn":"2024-03-03T15:31:41+0000"}
                michael:ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGZerclqaX4Umr1bd92pMztzX32l1Jvuo+CiuOwn5oqomkFis/+EQMn6D930GnV+ZrCjX0mm+KvJTujvmTbFkhs= google-ssh {"userName":"michael@obrienlabs.app","expireOn":"2024-03-03T15:31:53+0000"}
                michael:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbg7whJzUIiH6nelcmjFCXcVE9PL4q/7g+uTG7CXdn8eq16d7emjyWeE/4Ogbkr9LJMwae9CXROMilr7XY3XP2RuNeVV1gE6LsF4NkHhqx2VWHf6+B09/7QO+I8J/bQW6eRP+sqSJwJoU4hWx7F6H6rmumHgoH52mIVw6cI7GCbnaN9IhonlvDQWNp9w5LTzrE8eo6kJHXUQ9uxajQmS7je0Oyh2wV4AJFCtQlp53/2GwDIU0PipODn8W2YM8lpOqaXWDbcn4kgI2y+/cNrBVjfxwfluhWF5aoz+ZQBiPI3ZqJ4QheHMDshVVleIxd9cMtgSI5B2xAzcaPWeYOlxEx google-ssh {"userName":"michael@obrienlabs.app","expireOn":"2024-03-03T15:32:08+0000"}
            EOT -> null
          ~ "user-data" = <<-EOT
                config system global
                  set hostname fgtvm-na-northeast1-a
                end
                config system probe-response
                    set mode http-probe
                    set http-probe-value OK
                    set port 8008
                end
                config system api-user
                  edit terraform
                    set api-key YAZEDTw2B8fQEw5VoOdS1KNVjjTwDH
                    set accprofile "prof_admin"
                    config trusthost
                          edit 0
              -         set ipv4-trusthost 35.237.152.177/32
              +         set ipv4-trusthost 34.74.37.83/32
                      next
                        end
                  next
                end
                config system sdn-connector
                    edit "gcp"
                        set type gcp
                        set ha-status enable
                    next
                end
                config system dns
                  set primary 169.254.169.254
                  set protocol cleartext
                  unset secondary
                end
                config system ha
                    set group-name "gcp-group"
                    set mode a-p
                    set hbdev "port3" 50
                    set session-pickup enable
                    set ha-mgmt-status enable
                    config ha-mgmt-interfaces
                        edit 1
                            set interface "port4"
                            set gateway 172.20.3.1
                        next
                    end
                    set override disable
                    set priority 1
                    set unicast-hb enable
                    set unicast-hb-peerip 172.20.2.2
                    set unicast-hb-netmask 255.255.255.0
                end
                config system interface
                  edit port1
                    set mode static
                    set ip 172.20.0.3/32
                  next
                  edit port2
                    set mode static
                    set allowaccess ping
                    set ip 172.20.1.3/32
                    set secondary-IP enable
                    config secondaryip
                      edit 0
                      set ip 172.20.1.4/32
                      set allowaccess probe-response
                      next
                    end
                  next
                  edit port3
                    set mode static
                    set allowaccess ping
                    set ip 172.20.2.3/32
                  next
                  edit port4
                    set mode static
                    set ip 172.20.3.2/32
                    set allowaccess ping https ssh fgfm
                  next
                  edit "probe"
                    set vdom "root"
                    set ip 169.254.255.100 255.255.255.255
                    set allowaccess probe-response
                    set type loopback
                next
                end
                config router static
                  edit 0
                    set device port1
                    set gateway 172.20.0.1
                  next
                  edit 0
                    set device port2
                    set dst 172.20.0.0/24
                    set gateway 172.20.1.1
                  next
                  edit 0
                    set device port2
                    set dst 35.191.0.0/16
                    set gateway 172.20.1.1
                  next
                  edit 0
                    set device port2
                    set dst 130.211.0.0/22
                    set gateway 172.20.1.1
                  next
                end
            EOT
            # (1 unchanged element hidden)
        }
        name                 = "fgtvm1-na-northeast1-a"
        tags                 = [
            "fgt",
        ]
        # (18 unchanged attributes hidden)

        # (9 unchanged blocks hidden)
    }

  # module.fortigates.google_compute_instance.fgt-vm[1] will be updated in-place
  ~ resource "google_compute_instance" "fgt-vm" {
        id                   = "projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instances/fgtvm2-na-northeast1-b"
      ~ metadata             = {
          ~ "user-data" = <<-EOT
                config system global
                  set hostname fgtvm-na-northeast1-b
                end
                config system probe-response
                    set mode http-probe
                    set http-probe-value OK
                    set port 8008
                end
                config system api-user
                  edit terraform
                    set api-key YAZEDTw2B8fQEw5VoOdS1KNVjjTwDH
                    set accprofile "prof_admin"
                    config trusthost
                          edit 0
              -         set ipv4-trusthost 35.237.152.177/32
              +         set ipv4-trusthost 34.74.37.83/32
                      next
                        end
                  next
                end
                config system sdn-connector
                    edit "gcp"
                        set type gcp
                        set ha-status enable
                    next
                end
                config system dns
                  set primary 169.254.169.254
                  set protocol cleartext
                  unset secondary
                end
                config system ha
                    set group-name "gcp-group"
                    set mode a-p
                    set hbdev "port3" 50
                    set session-pickup enable
                    set ha-mgmt-status enable
                    config ha-mgmt-interfaces
                        edit 1
                            set interface "port4"
                            set gateway 172.20.3.1
                        next
                    end
                    set override disable
                    set priority 0
                    set unicast-hb enable
                    set unicast-hb-peerip 172.20.2.3
                    set unicast-hb-netmask 255.255.255.0
                end
                config system interface
                  edit port1
                    set mode static
                    set ip 172.20.0.2/32
                  next
                  edit port2
                    set mode static
                    set allowaccess ping
                    set ip 172.20.1.2/32
                    set secondary-IP enable
                    config secondaryip
                      edit 0
                      set ip 172.20.1.4/32
                      set allowaccess probe-response
                      next
                    end
                  next
                  edit port3
                    set mode static
                    set allowaccess ping
                    set ip 172.20.2.2/32
                  next
                  edit port4
                    set mode static
                    set ip 172.20.3.3/32
                    set allowaccess ping https ssh fgfm
                  next
                  edit "probe"
                    set vdom "root"
                    set ip 169.254.255.100 255.255.255.255
                    set allowaccess probe-response
                    set type loopback
                next
                end
                config router static
                  edit 0
                    set device port1
                    set gateway 172.20.0.1
                  next
                  edit 0
                    set device port2
                    set dst 172.20.0.0/24
                    set gateway 172.20.1.1
                  next
                  edit 0
                    set device port2
                    set dst 35.191.0.0/16
                    set gateway 172.20.1.1
                  next
                  edit 0
                    set device port2
                    set dst 130.211.0.0/22
                    set gateway 172.20.1.1
                  next
                end
            EOT
            # (1 unchanged element hidden)
        }
        name                 = "fgtvm2-na-northeast1-b"
        tags                 = [
            "fgt",
        ]
        # (18 unchanged attributes hidden)

        # (9 unchanged blocks hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.
╷
│ Warning: Deprecated attribute
│ 
│   on main.tf line 19, in module "fortigates":
│   19:   admin_acl       = ["${data.http.my_ip.body}/32"]
│ 
│ The attribute "body" is deprecated. Refer to the provider documentation for details.
│ 
│ (and 3 more similar warnings elsewhere)
╵

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: tf.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "tf.plan"

terraform apply --parallelism=1 tf.plan

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ terraform apply --parallelism=1 tf.plan
module.fortigates.google_compute_firewall.allow-mgmt: Modifying... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-mgmt-allow-admin]
module.fortigates.google_compute_firewall.allow-mgmt: Still modifying... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-mgmt-allow-admin, 10s elapsed]
module.fortigates.google_compute_firewall.allow-mgmt: Modifications complete after 12s [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-mgmt-allow-admin]
module.fortigates.google_compute_instance.fgt-vm[1]: Modifying... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instances/fgtvm2-na-northeast1-b]
module.fortigates.google_compute_instance.fgt-vm[1]: Still modifying... [id=projects/fortigate-terraform-olapp/zone...st1-b/instances/fgtvm2-na-northeast1-b, 10s elapsed]
module.fortigates.google_compute_instance.fgt-vm[1]: Modifications complete after 12s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instances/fgtvm2-na-northeast1-b]
module.fortigates.google_compute_instance.fgt-vm[0]: Modifying... [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instances/fgtvm1-na-northeast1-a]
module.fortigates.google_compute_instance.fgt-vm[0]: Still modifying... [id=projects/fortigate-terraform-olapp/zone...st1-a/instances/fgtvm1-na-northeast1-a, 10s elapsed]
module.fortigates.google_compute_instance.fgt-vm[0]: Modifications complete after 12s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instances/fgtvm1-na-northeast1-a]
╷
│ Warning: Deprecated attribute
│ 
│   on main.tf line 19, in module "fortigates":
│   19:   admin_acl       = ["${data.http.my_ip.body}/32"]
│ 
│ The attribute "body" is deprecated. Refer to the provider documentation for details.
│ 
│ (and one more similar warning elsewhere)
╵

Apply complete! Resources: 0 added, 3 changed, 0 destroyed.

Outputs:

api-key = "YAZEDTw2B8fQEw5VoOdS1KNVjjTwDH"
default_password = "348639094312086080"
fgt-mgmt-eips = [
  "34.152.20.239",
  "34.95.52.25",
]
fgt_umigs = [
  "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instanceGroups/fgtumig0-na-northeast1-a",
  "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instanceGroups/fgtumig1-na-northeast1-b",
]
health_check = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/healthChecks/fgthealthcheck-http8008-na-northeast1"
ilb = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/fgtfwdrule-ilb-trust-na-northeast1"
internal_subnet = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal"
internal_vpc = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/global/networks/fgt-vpc-internal"
prefix = "fgt-"
project = "fortigate-terraform-olapp"
region = "northamerica-northeast1"

reapply day1

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform plan -out tf.plan
data.terraform_remote_state.base: Reading...
data.terraform_remote_state.base: Read complete after 0s
data.google_compute_zones.local: Reading...
module.peer1.data.google_compute_subnetwork.hub: Reading...
module.peer1.data.google_compute_subnetwork.hub: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal]
data.google_compute_zones.local: Read complete after 1s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform planned the following actions, but then encountered a problem:

  # google_compute_address.wrkld_tier1 will be created
  + resource "google_compute_address" "wrkld_tier1" {
      + address            = (known after apply)
      + address_type       = "INTERNAL"
      + creation_timestamp = (known after apply)
      + effective_labels   = (known after apply)
      + id                 = (known after apply)
      + label_fingerprint  = (known after apply)
      + name               = "fgt-ip-wrkld-tier1"
      + network_tier       = (known after apply)
      + prefix_length      = (known after apply)
      + project            = "fortigate-terraform-olapp"
      + purpose            = (known after apply)
      + region             = "northamerica-northeast1"
      + self_link          = (known after apply)
      + subnetwork         = (known after apply)
      + terraform_labels   = (known after apply)
      + users              = (known after apply)
    }

  # google_compute_address.wrkld_tier2 will be created
  + resource "google_compute_address" "wrkld_tier2" {
      + address            = (known after apply)
      + address_type       = "INTERNAL"
      + creation_timestamp = (known after apply)
      + effective_labels   = (known after apply)
      + id                 = (known after apply)
      + label_fingerprint  = (known after apply)
      + name               = "fgt-ip-wrkld-tier2"
      + network_tier       = (known after apply)
      + prefix_length      = (known after apply)
      + project            = "fortigate-terraform-olapp"
      + purpose            = (known after apply)
      + region             = "northamerica-northeast1"
      + self_link          = (known after apply)
      + subnetwork         = (known after apply)
      + terraform_labels   = (known after apply)
      + users              = (known after apply)
    }

  # google_compute_firewall.tier1 will be created
  + resource "google_compute_firewall" "tier1" {
      + creation_timestamp = (known after apply)
      + destination_ranges = (known after apply)
      + direction          = (known after apply)
      + enable_logging     = (known after apply)
      + id                 = (known after apply)
      + name               = "fgt-wrkld-fw-tier1-allowall"
      + network            = (known after apply)
      + priority           = 1000
      + project            = "fortigate-terraform-olapp"
      + self_link          = (known after apply)
      + source_ranges      = [
          + "0.0.0.0/0",
        ]

      + allow {
          + ports    = []
          + protocol = "all"
        }
    }

  # google_compute_firewall.tier2 will be created
  + resource "google_compute_firewall" "tier2" {
      + creation_timestamp = (known after apply)
      + destination_ranges = (known after apply)
      + direction          = (known after apply)
      + enable_logging     = (known after apply)
      + id                 = (known after apply)
      + name               = "fgt-wrkld-fw-tier2-allowall"
      + network            = (known after apply)
      + priority           = 1000
      + project            = "fortigate-terraform-olapp"
      + self_link          = (known after apply)
      + source_ranges      = [
          + "0.0.0.0/0",
        ]

      + allow {
          + ports    = []
          + protocol = "all"
        }
    }

  # google_compute_network.tier1 will be created
  + resource "google_compute_network" "tier1" {
      + auto_create_subnetworks                   = false
      + delete_default_routes_on_create           = true
      + gateway_ipv4                              = (known after apply)
      + id                                        = (known after apply)
      + internal_ipv6_range                       = (known after apply)
      + mtu                                       = (known after apply)
      + name                                      = "fgt-wrkld-vpc-tier1"
      + network_firewall_policy_enforcement_order = "AFTER_CLASSIC_FIREWALL"
      + numeric_id                                = (known after apply)
      + project                                   = "fortigate-terraform-olapp"
      + routing_mode                              = (known after apply)
      + self_link                                 = (known after apply)
    }

  # google_compute_network.tier2 will be created
  + resource "google_compute_network" "tier2" {
      + auto_create_subnetworks                   = false
      + delete_default_routes_on_create           = true
      + gateway_ipv4                              = (known after apply)
      + id                                        = (known after apply)
      + internal_ipv6_range                       = (known after apply)
      + mtu                                       = (known after apply)
      + name                                      = "fgt-wrkld-vpc-tier2"
      + network_firewall_policy_enforcement_order = "AFTER_CLASSIC_FIREWALL"
      + numeric_id                                = (known after apply)
      + project                                   = "fortigate-terraform-olapp"
      + routing_mode                              = (known after apply)
      + self_link                                 = (known after apply)
    }

  # google_compute_subnetwork.tier1 will be created
  + resource "google_compute_subnetwork" "tier1" {
      + creation_timestamp         = (known after apply)
      + external_ipv6_prefix       = (known after apply)
      + fingerprint                = (known after apply)
      + gateway_address            = (known after apply)
      + id                         = (known after apply)
      + internal_ipv6_prefix       = (known after apply)
      + ip_cidr_range              = "10.0.0.0/16"
      + ipv6_cidr_range            = (known after apply)
      + name                       = "fgt-wrkld-sb-tier1"
      + network                    = (known after apply)
      + private_ip_google_access   = (known after apply)
      + private_ipv6_google_access = (known after apply)
      + project                    = "fortigate-terraform-olapp"
      + purpose                    = (known after apply)
      + region                     = "northamerica-northeast1"
      + secondary_ip_range         = (known after apply)
      + self_link                  = (known after apply)
      + stack_type                 = (known after apply)
    }

  # google_compute_subnetwork.tier2 will be created
  + resource "google_compute_subnetwork" "tier2" {
      + creation_timestamp         = (known after apply)
      + external_ipv6_prefix       = (known after apply)
      + fingerprint                = (known after apply)
      + gateway_address            = (known after apply)
      + id                         = (known after apply)
      + internal_ipv6_prefix       = (known after apply)
      + ip_cidr_range              = "10.1.0.0/16"
      + ipv6_cidr_range            = (known after apply)
      + name                       = "fgt-wrkld-sb-tier2"
      + network                    = (known after apply)
      + private_ip_google_access   = (known after apply)
      + private_ipv6_google_access = (known after apply)
      + project                    = "fortigate-terraform-olapp"
      + purpose                    = (known after apply)
      + region                     = "northamerica-northeast1"
      + secondary_ip_range         = (known after apply)
      + self_link                  = (known after apply)
      + stack_type                 = (known after apply)
    }

  # module.peer1.google_compute_network_peering.hub_to_spoke will be created
  + resource "google_compute_network_peering" "hub_to_spoke" {
      + export_custom_routes                = true
      + export_subnet_routes_with_public_ip = true
      + id                                  = (known after apply)
      + import_custom_routes                = false
      + name                                = "peer-fgthub-to-fgt-wrkld-vpc-tier1-fortigate-terraform-olapp"
      + network                             = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/global/networks/fgt-vpc-internal"
      + peer_network                        = (known after apply)
      + stack_type                          = "IPV4_ONLY"
      + state                               = (known after apply)
      + state_details                       = (known after apply)
    }

  # module.peer1.google_compute_network_peering.spoke_to_hub will be created
  + resource "google_compute_network_peering" "spoke_to_hub" {
      + export_custom_routes                = false
      + export_subnet_routes_with_public_ip = true
      + id                                  = (known after apply)
      + import_custom_routes                = true
      + name                                = "peer-fgt-wrkld-vpc-tier1-fortigate-terraform-olapp-to-fgthub"
      + network                             = (known after apply)
      + peer_network                        = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/global/networks/fgt-vpc-internal"
      + stack_type                          = "IPV4_ONLY"
      + state                               = (known after apply)
      + state_details                       = (known after apply)
    }

Plan: 10 to add, 0 to change, 0 to destroy.
╷
│ Error: Error create fortios client: Error using Token to login: 
│ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
│ <html><head>
│ <title>403 Forbidden</title>
│ </head><body>
│ <h1>Forbidden</h1>
│ <p>You don't have permission to access this resource.</p>
│ <p>Additionally, a 403 Forbidden
│ error was encountered while trying to use an ErrorDocument to handle the request.</p>
│ </body></html>
│ 
│ 
│   with provider["registry.terraform.io/fortinetdev/fortios"],
│   on versions.tf line 21, in provider "fortios":
│   21: provider "fortios" {
│ 
╵

destroy day0 and restart with byol 70 to test fortios authentication

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform show
No state.

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day0 (fortigate-terraform-olapp)$ terraform destroy
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

module.fortigates.google_compute_firewall.allow-port2: Destroying... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-trust-allowall]
module.fortigates.google_compute_firewall.allow-mgmt: Destroying... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-mgmt-allow-admin]
module.fortigates.google_compute_firewall.allow-port1: Destroying... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-untrust-allowall]
module.fortigates.google_compute_forwarding_rule.ilb_fwd_rule: Destroying... [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/fgtfwdrule-ilb-trust-na-northeast1]
module.fortigates.google_compute_firewall.allow-hasync: Destroying... [id=projects/fortigate-terraform-olapp/global/firewalls/fgtfw-hasync-allow-fgt]
module.fortigates.google_compute_router_nat.cloud_nat: Destroying... [id=fortigate-terraform-olapp/northamerica-northeast1/fgtcr-cloudnat-na-northeast1/fgtnat-cloudnat-na-northeast1]

Destroy complete! Resources: 35 destroyed.

just noticed day0 - missed api trusthost in main.tf:10

  url             = "http://api.ipify.org"
  url             = "http://172...160"

1838: running with ip change terraform init and plan

Issue with fortios authentication in day1 and an expected config for https://github.com/fortinet/fortigate-tutorial-gcp/blob/main/terraform/day0/main.tf#L9

# Auto-detect your own IP address to add it to the API trusthost list in FortiGate configuration
data "http" "my_ip" {
  url             = "http://api.ipify.org"
}
never mind - that address gets your IP - leave as is - but the issue may be IPV6 related

testing a hardcoded all-internet range - no need for ipv4 discovery
module "fortigates" {
  source          = "../modules/fgcp-ha-ap-lb"
  region          = var.GCE_REGION
  service_account = data.google_service_account.fgt.email != null ? data.google_service_account.fgt.email : ""
 # admin_acl       = ["${data.http.my_ip.body}/32"]
  admin_acl       = ["0.0.0.0/0"]
 # api_acl         = ["${data.http.my_ip.body}/32"]
  api_acl         = ["0.0.0.0/0"]

terraform apply with the 0.0.0.0/0 acl

Apply complete! Resources: 35 added, 0 changed, 0 destroyed.

Outputs:

api-key = "ow9vapFBmjQfByIVNZBpXlM3Zziw5Z"
default_password = "3205180671325155452"
fgt-mgmt-eips = [
  "34.95.52.25",
  "34.152.20.239",
]
fgt_umigs = [
  "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instanceGroups/fgtumig0-na-northeast1-a",
  "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/zones/northamerica-northeast1-b/instanceGroups/fgtumig1-na-northeast1-b",
]
health_check = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/healthChecks/fgthealthcheck-http8008-na-northeast1"
ilb = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/fgtfwdrule-ilb-trust-na-northeast1"
internal_subnet = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal"
internal_vpc = "https://www.googleapis.com/compute/v1/projects/fortigate-terraform-olapp/global/networks/fgt-vpc-internal"
prefix = "fgt-"
project = "fortigate-terraform-olapp"
region = "northamerica-northeast1"

working now with the 0.0.0.0/0 incoming change on the VMs

issue was fortigate incoming allow on /32 address did not handle IPV6 - not the firewall rule allow which was already 0.0.0.0/0

In some clients - "what is my ip" will return an IPV6 address instead of a IPV4 address The fix for this is to ignore the lookup of your /32 ip and just open it to 0.0.0.0/0 without the fix - we hang on admin access

change

in https://github.com/fortinet/fortigate-tutorial-gcp/blob/main/terraform/day0/main.tf#L9

 data "http" "my_ip" {
   url             = "http://api.ipify.org"
 }

 # Create base deployment of FortiGate HA cluster
@@ -16,8 +17,10 @@ module "fortigates" {
   region          = var.GCE_REGION
   service_account = data.google_service_account.fgt.email != null ? data.google_service_account.fgt.email : ""
-  admin_acl       = ["${data.http.my_ip.body}/32"]
-  api_acl         = ["${data.http.my_ip.body}/32"]
+ # admin_acl       = ["${data.http.my_ip.body}/32"]
+  admin_acl       = ["0.0.0.0/0"]
+ # api_acl         = ["${data.http.my_ip.body}/32"]
+  api_acl         = ["0.0.0.0/0"]

day1 issue on fortios during plan https://github.com/fortinet/fortigate-tutorial-gcp/blob/main/terraform/day1/versions.tf#L21

Plan: 10 to add, 0 to change, 0 to destroy.
╷
│ Error: Error create fortios client: Error using Token to login: 
│ <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
│ <html><head>
│ <title>403 Forbidden</title>
│ </head><body>
│ <h1>Forbidden</h1>
│ <p>You don't have permission to access this resource.</p>
│ <p>Additionally, a 403 Forbidden
│ error was encountered while trying to use an ErrorDocument to handle the request.</p>
│ </body></html>
│ 
│ 
│   with provider["registry.terraform.io/fortinetdev/fortios"],
│   on versions.tf line 21, in provider "fortios":
│   21: provider "fortios" {
│ 
╵

fixing by using the username password option of the fortios provider instead of the token - and switching to the last VM I authenticated on - index 1


day1/versions.tf
provider "fortios" {
# TODO: automatically find which peer is primary at the moment of deployment
#       for now we just go to the first instance

  hostname  = data.terraform_remote_state.base.outputs.fgt-mgmt-eips[1]
  username  = "admin" 
  password  = "m...1"
  #token     = data.terraform_remote_state.base.outputs.api-key
  insecure  = "true"

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform plan -out tf.plan

Plan: 31 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + public_ip = (known after apply)

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: tf.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "tf.plan"

1920
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform apply --parallelism=1 tf.plan
fortios_firewall_address.tier2: Creating...
fortios_firewall_address.tier2: Creation complete after 1s [id=gcp-tier2]
fortios_firewall_address.tier1: Creating...
fortios_firewall_address.tier1: Creation complete after 0s [id=gcp-tier1]
google_compute_network.tier2: Creating...

getting quota error
fortios_firewall_policy.tier1-to-tier2: Creation complete after 0s [id=1]
╷
│ Error: Error waiting to create Network: Error waiting for Creating Network: Quota 'NETWORKS' exceeded.  Limit: 5.0 globally.
│       metric name = compute.googleapis.com/networks
│       limit name = NETWORKS-per-project
│       limit = 5
│       dimensions = map[global:global]
│ 
│ 
│   with google_compute_network.tier1,
│   on workloads.tf line 6, in resource "google_compute_network" "tier1":
│    6: resource "google_compute_network" "tier1" {
│

Thank you for submitting Case # (ID:19e7ad152b654b908c) to Google Cloud Platform support for the following quota:
Change Networks from 5 to 10

1 min
Your quota request for fortigate-terraform-olapp has been approved and your project quota has been adjusted according to the following requested limits:

+----------+------------+--------+-----------------+----------------+
| NAME     | DIMENSIONS | REGION | REQUESTED LIMIT | APPROVED LIMIT |
+----------+------------+--------+-----------------+----------------+
| NETWORKS |            | GLOBAL |              10 |             10 |
+----------+------------+--------+-----------------+----------------+

After approval, Quotas can take up to 15 min to be fully visible in the Cloud Console and available to you.

hold for now https://github.com/40net-cloud/fortinet-gcp-solutions/issues/11

rerun

michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform plan -out tf.plan
Plan: 27 to add, 0 to change, 0 to destroy.

1928
michael@cloudshell:~/fortigate-terraform-olapp/fortigate-tutorial-gcp/terraform/day1 (fortigate-terraform-olapp)$ terraform apply --parallelism=1 tf.plan
google_compute_network.tier1: Creating...
google_compute_network.tier1: Still creating... [10s elapsed]
google_compute_network.tier1: Still creating... [20s elapsed]
google_compute_network.tier1: Still creating... [30s elapsed]
google_compute_network.tier1: Creation complete after 33s [id=projects/fortigate-terraform-olapp/global/networks/fgt-wrkld-vpc-tier1]
google_compute_network.tier2: Creating...
google_compute_network.tier2: Still creating... [10s elapsed]
google_compute_network.tier2: Still creating... [20s elapsed]
google_compute_network.tier2: Creation complete after 23s [id=projects/fortigate-terraform-olapp/global/networks/fgt-wrkld-vpc-tier2]
google_compute_firewall.tier1: Creating...
google_compute_firewall.tier1: Still creating... [10s elapsed]
google_compute_firewall.tier1: Creation complete after 12s [id=projects/fortigate-terraform-olapp/global/firewalls/fgt-wrkld-fw-tier1-allowall]
google_compute_subnetwork.tier1: Creating...
google_compute_subnetwork.tier1: Still creating... [10s elapsed]
google_compute_subnetwork.tier1: Creation complete after 12s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-wrkld-sb-tier1]
google_compute_subnetwork.tier2: Creating...
google_compute_subnetwork.tier2: Still creating... [10s elapsed]
oogle_compute_subnetwork.tier2: Still creating... [20s elapsed]
google_compute_subnetwork.tier2: Creation complete after 22s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-wrkld-sb-tier2]
google_compute_firewall.tier2: Creating...
google_compute_firewall.tier2: Still creating... [10s elapsed]
google_compute_firewall.tier2: Creation complete after 12s [id=projects/fortigate-terraform-olapp/global/firewalls/fgt-wrkld-fw-tier2-allowall]
google_compute_address.wrkld_tier1: Creating...
google_compute_address.wrkld_tier1: Still creating... [10s elapsed]
google_compute_address.wrkld_tier1: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgt-ip-wrkld-tier1]
module.peer1.google_compute_network_peering.hub_to_spoke: Creating...
module.peer1.google_compute_network_peering.hub_to_spoke: Still creating... [10s elapsed]
module.peer1.google_compute_network_peering.hub_to_spoke: Creation complete after 11s [id=fgt-vpc-internal/peer-fgthub-to-fgt-wrkld-vpc-tier1-fortigate-terraform-olapp]
google_compute_address.wrkld_tier2: Creating...
google_compute_address.wrkld_tier2: Still creating... [10s elapsed]
google_compute_address.wrkld_tier2: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/fgt-ip-wrkld-tier2]
module.peer1.google_compute_network_peering.spoke_to_hub: Creating...
1930
module.peer1.google_compute_network_peering.spoke_to_hub: Still creating... [10s elapsed]
module.peer1.google_compute_network_peering.spoke_to_hub: Still creating... [20s elapsed]
module.peer1.google_compute_network_peering.spoke_to_hub: Creation complete after 22s [id=fgt-wrkld-vpc-tier1/peer-fgt-wrkld-vpc-tier1-fortigate-terraform-olapp-to-fgthub]
module.inbound.data.fortios_system_proberesponse.probe: Reading...
module.inbound.data.fortios_system_proberesponse.probe: Read complete after 0s [id=SystemProbeResponse]
module.inbound.data.fortios_system_interface.probe: Reading...
module.inbound.data.fortios_system_interface.probe: Read complete after 0s [id=probe]
module.peer2.data.google_compute_subnetwork.hub: Reading...
module.peer2.data.google_compute_subnetwork.hub: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/subnetworks/fgt-sb-internal]
module.inbound.google_compute_route.default_route: Creating...
module.inbound.google_compute_route.default_route: Still creating... [10s elapsed]
module.inbound.google_compute_route.default_route: Still creating... [20s elapsed]
module.inbound.google_compute_route.default_route: Creation complete after 22s [id=projects/fortigate-terraform-olapp/global/routes/fgt-rt-default-via-fgt]
module.inbound.fortios_firewallservice_custom.service[0]: Creating...
module.inbound.fortios_firewallservice_custom.service[0]: Creation complete after 0s [id=serv1-tcp8080]
module.inbound.fortios_firewallservice_custom.service_probe: Creating...
module.inbound.fortios_firewallservice_custom.service_probe: Creation complete after 1s [id=LB_Probe]
module.inbound.google_compute_region_backend_service.elb_bes: Creating...
module.inbound.google_compute_region_backend_service.elb_bes: Still creating... [10s elapsed]
module.inbound.google_compute_region_backend_service.elb_bes: Still creating... [20s elapsed]
module.inbound.google_compute_region_backend_service.elb_bes: Creation complete after 21s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/backendServices/fgt-bes-elb-na-northeast1]
module.inbound.google_compute_address.elb_eip: Creating...
1932
module.inbound.google_compute_address.elb_eip: Still creating... [10s elapsed]
module.inbound.google_compute_address.elb_eip: Creation complete after 11s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/addresses/serv1-eip-na-northeast1]
module.peer2.fortios_router_static.to_spoke_subnets["10.1.0.0/16"]: Creating...
module.peer2.fortios_router_static.to_spoke_subnets["10.1.0.0/16"]: Creation complete after 1s [id=6]
module.peer2.google_compute_network_peering.hub_to_spoke: Creating...
module.peer2.google_compute_network_peering.hub_to_spoke: Still creating... [10s elapsed]
module.peer2.google_compute_network_peering.hub_to_spoke: Creation complete after 11s [id=fgt-vpc-internal/peer-fgthub-to-fgt-wrkld-vpc-tier2-fortigate-terraform-olapp]
module.inbound.fortios_firewall_vip.vip[0]: Creating...
module.inbound.fortios_firewall_vip.vip[0]: Creation complete after 0s [id=serv1-tcp80]
module.inbound.google_compute_forwarding_rule.elb_frule: Creating...
module.inbound.google_compute_forwarding_rule.elb_frule: Still creating... [10s elapsed]
module.inbound.google_compute_forwarding_rule.elb_frule: Still creating... [20s elapsed]
module.inbound.google_compute_forwarding_rule.elb_frule: Creation complete after 21s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/serv1-fwdrule]
module.inbound.fortios_firewall_vip.vip_probe: Creating...
module.inbound.fortios_firewall_vip.vip_probe: Creation complete after 0s [id=serv1-probe]
module.peer2.google_compute_network_peering.spoke_to_hub: Creating...
module.peer2.google_compute_network_peering.spoke_to_hub: Still creating... [10s elapsed]
module.peer2.google_compute_network_peering.spoke_to_hub: Still creating... [20s elapsed]
module.peer2.google_compute_network_peering.spoke_to_hub: Creation complete after 21s [id=fgt-wrkld-vpc-tier2/peer-fgt-wrkld-vpc-tier2-fortigate-terraform-olapp-to-fgthub]
module.inbound.fortios_firewall_policy.vip_allow[0]: Creating...
module.inbound.fortios_firewall_policy.vip_allow[0]: Creation complete after 0s [id=2]
module.inbound.fortios_firewall_policy.probe_allow: Creating...
module.inbound.fortios_firewall_policy.probe_allow: Creation complete after 1s [id=3]
module.outbound.data.google_compute_forwarding_rule.elb: Reading...
module.outbound.data.google_compute_forwarding_rule.elb: Read complete after 0s [id=projects/fortigate-terraform-olapp/regions/northamerica-northeast1/forwardingRules/serv1-fwdrule]
module.outbound.fortios_firewall_ippool.this: Creating...
module.outbound.fortios_firewall_ippool.this: Creation complete after 0s [id=gcp-serv1-eip]
module.outbound.fortios_firewall_policy.allowout: Creating...
module.outbound.fortios_firewall_policy.allowout: Creation complete after 1s [id=4]
google_compute_instance.wrkld_websrv: Creating...
google_compute_instance.wrkld_websrv: Still creating... [10s elapsed]
google_compute_instance.wrkld_websrv: Creation complete after 12s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instances/fgt-wrkld-tier2-websrv]
google_compute_instance.wrkld_proxy: Creating...
google_compute_instance.wrkld_proxy: Still creating... [10s elapsed]
google_compute_instance.wrkld_proxy: Creation complete after 13s [id=projects/fortigate-terraform-olapp/zones/northamerica-northeast1-a/instances/fgt-wrkld-tier1-proxy]

Apply complete! Resources: 27 added, 0 changed, 0 destroyed.

Outputs:

public_ip = "34.118.190.20"

patching branch

diff --git a/2024_fortigate-accelerator/terraform/day0/main.tf b/2024_fortigate-accelerator/terraform/day0/main.tf
index 550804f..1b69e6d 100644
--- a/2024_fortigate-accelerator/terraform/day0/main.tf
+++ b/2024_fortigate-accelerator/terraform/day0/main.tf
@@ -6,6 +6,7 @@ data google_service_account fgt {
 }

 # Auto-detect your own IP address to add it to the API trusthost list in FortiGate configuration
+# ignore for now due to possible IPV6 issue - open up to 0.0.0.0/0 below in the 2 ACLs
 data "http" "my_ip" {
   url             = "http://api.ipify.org"
 }
@@ -16,8 +17,10 @@ module "fortigates" {

   region          = var.GCE_REGION
   service_account = data.google_service_account.fgt.email != null ? data.google_service_account.fgt.email : ""
-  admin_acl       = ["${data.http.my_ip.body}/32"]
-  api_acl         = ["${data.http.my_ip.body}/32"]
+  #admin_acl       = ["${data.http.my_ip.body}/32"]
+  admin_acl       = ["0.0.0.0/0"]
+  #api_acl         = ["${data.http.my_ip.body}/32"]
+  api_acl         = ["0.0.0.0/0"]

   # Use the below subnet names if you create new networks using sample_networks or update to your own
   # Remember to use subnet list as names, not selfLinks
diff --git a/2024_fortigate-accelerator/terraform/day1/versions.tf b/2024_fortigate-accelerator/terraform/day1/versions.tf
index 654e043..1b0a903 100644
--- a/2024_fortigate-accelerator/terraform/day1/versions.tf
+++ b/2024_fortigate-accelerator/terraform/day1/versions.tf
@@ -23,6 +23,9 @@ provider "fortios" {
 #       for now we just go to he first instance

   hostname  = data.terraform_remote_state.base.outputs.fgt-mgmt-eips[0]
-  token     = data.terraform_remote_state.base.outputs.api-key
+  username  = "admin" 
+  password  = "password"
+# use username and password or just token below
+  #token     = data.terraform_remote_state.base.outputs.api-key
   insecure  = "true"
 }

however, I think this authentication issue occurred because I changed the default password from the instance id between day0 and day1 - so on me, retesting without the pw change

TEF V4 is our focus now as the base LZ with Fortigate integration from the above repo https://github.com/terraform-google-modules/terraform-example-foundation

follow previous: https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/243 review previous issues: https://github.com/terraform-google-modules/terraform-example-foundation/issues/940

Org: olapp repo https://github.com/CloudLandingZone/terraform-example-foundation issue https://github.com/terraform-google-modules/terraform-example-foundation/issues/1133

michael@cloudshell:~$ ls
FGVM8VTM24000185.lic  FGVM8VTM24000186.lic  fortigate-terraform-olapp  fortinet-gcp-solutions-olapp  gcloud-ola  kcc-olapp  README-cloudshell.txt
michael@cloudshell:~$ mkdir tef-olapp
michael@cloudshell:~$ cd tef-olapp/
michael@cloudshell:~/tef-olapp$ mkdir github
michael@cloudshell:~/tef-olapp$ cd github/
michael@cloudshell:~/tef-olapp/github$ mkdir _CloudLandingZone-main
michael@cloudshell:~/tef-olapp/github$ git clone https://github.com/CloudLandingZone/terraform-example-foundation.git
Cloning into 'terraform-example-foundation'...

Prep

org is relatively clean with minimal IAM permissions
billing quota is 25 (asked for 20 in addition to default 5)
project quota is default - adding 40 to get 60 total https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/Onboarding#quota-increase

0-bootstrap

raised - will see if we can interleave the cloud-setup groups and the TEF groups later https://github.com/terraform-google-modules/terraform-example-foundation/issues/1135

following https://github.com/terraform-google-modules/terraform-example-foundation/tree/master/0-bootstrap#prerequisites

where is group_org_admins in the tfvars file - do CB first https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/terraform.example.tfvars#L44 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1136

copy tfvars

https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md#deploying-with-cloud-build

ichael@cloudshell:~/tef-olapp/github$ cd terraform-example-foundation/0-bootstrap/
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap$ mv terraform.example.tfvars terraform.tfvars
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap$

replace get org via https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh766-script/solutions/setup.sh#L101

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap$ gcloud config set project tef-olapp
BOOT_PROJECT_ID=tef-olapp
ORG_ID=$(gcloud projects get-ancestors $BOOT_PROJECT_ID --format='get(id)' | tail -1)
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ echo $ORG_ID
63025...

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ BILLING_FORMAT="--format=value(billingAccountName)"
BILLING_ID=$(gcloud billing projects describe $BOOT_PROJECT_ID $BILLING_FORMAT | sed 's/.*\///')
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ echo $BILLING_ID
012...

org_id = "REPLACE_ME" # format "000000000000"
billing_account = "REPLACE_ME" # format "000000-000000-000000"
group_org_admins = "REPLACE_ME"
group_billing_admins = "REPLACE_ME"
# group_org_admins = "gcp-organization-admins@example.com"
# group_billing_admins = "gcp-billing-admins@example.com"

default_region = "us-central1"
parent_folder = "01234567890"

#  Optional - for enabling the automatic groups creation, uncoment the groups
#  variable and update the values with the desired group names
# groups = {
#   create_groups = true,
#   billing_project = "billing-project",
#   required_groups = {
#     group_org_admins           = "group_org_admins_local_test@example.com"
#     group_billing_admins       = "group_billing_admins_local_test@example.com"
#     billing_data_users         = "billing_data_users_local_test@example.com"
#     audit_data_users           = "audit_data_users_local_test@example.com"
#     monitoring_workspace_users = "monitoring_workspace_users_local_test@example.com"
#   },
#   optional_groups = {
#     gcp_platform_viewer      = "gcp_platform_viewer_local_test@example.com"
#     gcp_security_reviewer    = "gcp_security_reviewer_local_test@example.com"
#     gcp_network_viewer       = "gcp_network_viewer_local_test@example.com"
#     gcp_scc_admin            = "gcp_scc_admin_local_test@example.com"
#     gcp_global_secrets_admin = "gcp_global_secrets_admin_local_test@example.com"
#     gcp_audit_viewer         = "gcp_audit_viewer_local_test@example.com"
#   }
# }

to
org_id = "63...53" # format "000000000000"
billing_account = "012...B" # format "000000-000000-000000"
group_org_admins = "gcp-organization-admins@o..p"
group_billing_admins = "gcp-billing-admins@ob..p"
 parent_folder = "10...6"

using the groups left over from the cloud setup for now

gcp-organization-admins
gcp-billing-admins

adding

audit_data_users
monitoring_workspace_users
billing_data_users

validating


michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ ../scripts/validate-requirements.sh -o 63...53 -b 01...B -u mi..pp
Validating required utility tools...
Validating Terraform installation...
Validating Google Cloud SDK installation...
Validating Git installation...
  git default branch must be configured as main.
  See the instructions at https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting .
Validating local gcloud configuration...
Validating roles assignment for current end user credential...
  The User must have the Organization Roles resourcemanager.folderCreator, resourcemanager.organizationAdmin and roles/orgpolicy.policyAdmin.
Validating 0-bootstrap configuration...
.......................................
Validation failed!
Errors found:
  git default branch must be configured as main.
  There are missing organization level roles on the Credential.

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    terraform.example.tfvars

no changes added to commit (use "git add" and/or "git commit -a")

raised - there is no main branch only master - adjusting script and moving on https://github.com/terraform-google-modules/terraform-example-foundation/issues/1137

in https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md#deploying-with-cloud-build

adjust https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/scripts/validate-requirements.sh#L127

    if ! git config init.defaultBranch | grep "main" >/dev/null ; then
        echo "  git default branch must be configured as main."
        echo "  See the instructions at https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting ."
        ERRORS+=$'  git default branch must be configured as main.\n'
    fi

Branch instructions are not accurate https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting

there is no main branch so we should use master - what if we are doing development on a ghNNN-name branch?

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git config init.defaultBranch
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git config --global init.defaultBranch master
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git config --global init.defaultBranch main
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    terraform.example.tfvars

no changes added to commit (use "git add" and/or "git commit -a")
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git config --global init.defaultBranch master

roles

Billing Account Administrator
Compute Shared VPC Admin
Folder Admin
Folder Creator
Organization Administrator
Organization Policy Administrator
Project Billing Manager
Project Creator
Project Deleter
Project IAM Admin
Service Account Token Creator

creating main branch for now instead of hoping "master" wont cause an issue later in cloud build

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git checkout -b main
Switched to a new branch 'main'
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git config init.defaultBranch
master
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git config --global init.defaultBranch main
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git config init.defaultBranch
main

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ ../scripts/validate-requirements.sh -o 63,,,
Validating required utility tools...
Validating Terraform installation...
Validating Google Cloud SDK installation...
Validating Git installation...
Validating local gcloud configuration...
Validating roles assignment for current end user credential...
Validating 0-bootstrap configuration...
.......................................
Validation successful!
No errors found.

Terraform version 1.7.2

little worried about pre 1.3 references - this would suggest OPTIONAL deprecation issues

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform --version
Terraform v1.7.2

0 - bootstrap - terraform init

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform init

Initializing the backend...
Initializing modules...
Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for bootstrap_csr_repo...
- bootstrap_csr_repo in .terraform/modules/bootstrap_csr_repo
- bootstrap_projects_remove_editor in modules/parent-iam-remove-role
Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for build_terraform_image...
- build_terraform_image in .terraform/modules/build_terraform_image
- cicd_project_iam_member in modules/parent-iam-member
Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 5.0.0 for gcp_projects_state_bucket...
- gcp_projects_state_bucket in .terraform/modules/gcp_projects_state_bucket/modules/simple_bucket
Downloading registry.terraform.io/terraform-google-modules/group/google 0.6.1 for optional_group...
- optional_group in .terraform/modules/optional_group
- org_iam_member in modules/parent-iam-member
- parent_iam_member in modules/parent-iam-member
Downloading registry.terraform.io/terraform-google-modules/group/google 0.6.1 for required_group...
- required_group in .terraform/modules/required_group
Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 7.0.0 for seed_bootstrap...
- seed_bootstrap in .terraform/modules/seed_bootstrap
Downloading registry.terraform.io/terraform-google-modules/org-policy/google 5.3.0 for seed_bootstrap.enable_cross_project_service_account_usage...
- seed_bootstrap.enable_cross_project_service_account_usage in .terraform/modules/seed_bootstrap.enable_cross_project_service_account_usage
Downloading registry.terraform.io/terraform-google-modules/kms/google 2.3.0 for seed_bootstrap.kms...
- seed_bootstrap.kms in .terraform/modules/seed_bootstrap.kms
Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for seed_bootstrap.seed_project...
- seed_bootstrap.seed_project in .terraform/modules/seed_bootstrap.seed_project
- seed_bootstrap.seed_project.budget in .terraform/modules/seed_bootstrap.seed_project/modules/budget
- seed_bootstrap.seed_project.essential_contacts in .terraform/modules/seed_bootstrap.seed_project/modules/essential_contacts
- seed_bootstrap.seed_project.gsuite_group in .terraform/modules/seed_bootstrap.seed_project/modules/gsuite_group
- seed_bootstrap.seed_project.project-factory in .terraform/modules/seed_bootstrap.seed_project/modules/core_project_factory
- seed_bootstrap.seed_project.project-factory.project_services in .terraform/modules/seed_bootstrap.seed_project/modules/project_services
- seed_bootstrap.seed_project.quotas in .terraform/modules/seed_bootstrap.seed_project/modules/quota_manager
- seed_bootstrap.seed_project.shared_vpc_access in .terraform/modules/seed_bootstrap.seed_project/modules/shared_vpc_access
- seed_project_iam_member in modules/parent-iam-member
Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 7.0.0 for tf_cloud_builder...
- tf_cloud_builder in .terraform/modules/tf_cloud_builder/modules/tf_cloudbuild_builder
Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 5.0.0 for tf_cloud_builder.bucket...
- tf_cloud_builder.bucket in .terraform/modules/tf_cloud_builder.bucket/modules/simple_bucket
- tf_private_pool in modules/cb-private-pool
Downloading registry.terraform.io/terraform-google-modules/network/google 9.0.0 for tf_private_pool.firewall_rules...
- tf_private_pool.firewall_rules in .terraform/modules/tf_private_pool.firewall_rules/modules/firewall-rules
Downloading registry.terraform.io/terraform-google-modules/network/google 9.0.0 for tf_private_pool.peered_network...
- tf_private_pool.peered_network in .terraform/modules/tf_private_pool.peered_network
- tf_private_pool.peered_network.firewall_rules in .terraform/modules/tf_private_pool.peered_network/modules/firewall-rules
- tf_private_pool.peered_network.routes in .terraform/modules/tf_private_pool.peered_network/modules/routes
- tf_private_pool.peered_network.subnets in .terraform/modules/tf_private_pool.peered_network/modules/subnets
- tf_private_pool.peered_network.vpc in .terraform/modules/tf_private_pool.peered_network/modules/vpc
Downloading registry.terraform.io/terraform-google-modules/vpn/google 4.0.0 for tf_private_pool.vpn_ha_cb_to_onprem...
- tf_private_pool.vpn_ha_cb_to_onprem in .terraform/modules/tf_private_pool.vpn_ha_cb_to_onprem/modules/vpn_ha
Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 7.0.0 for tf_source...
- tf_source in .terraform/modules/tf_source/modules/tf_cloudbuild_source
Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 5.0.0 for tf_source.cloudbuild_bucket...
- tf_source.cloudbuild_bucket in .terraform/modules/tf_source.cloudbuild_bucket/modules/simple_bucket
Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for tf_source.cloudbuild_project...
- tf_source.cloudbuild_project in .terraform/modules/tf_source.cloudbuild_project
- tf_source.cloudbuild_project.budget in .terraform/modules/tf_source.cloudbuild_project/modules/budget
- tf_source.cloudbuild_project.essential_contacts in .terraform/modules/tf_source.cloudbuild_project/modules/essential_contacts
- tf_source.cloudbuild_project.gsuite_group in .terraform/modules/tf_source.cloudbuild_project/modules/gsuite_group
- tf_source.cloudbuild_project.project-factory in .terraform/modules/tf_source.cloudbuild_project/modules/core_project_factory
- tf_source.cloudbuild_project.project-factory.project_services in .terraform/modules/tf_source.cloudbuild_project/modules/project_services
- tf_source.cloudbuild_project.quotas in .terraform/modules/tf_source.cloudbuild_project/modules/quota_manager
- tf_source.cloudbuild_project.shared_vpc_access in .terraform/modules/tf_source.cloudbuild_project/modules/shared_vpc_access
Downloading registry.terraform.io/terraform-google-modules/bootstrap/google 7.0.0 for tf_workspace...
- tf_workspace in .terraform/modules/tf_workspace/modules/tf_cloudbuild_workspace
Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 5.0.0 for tf_workspace.artifacts_bucket...
- tf_workspace.artifacts_bucket in .terraform/modules/tf_workspace.artifacts_bucket/modules/simple_bucket
Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 5.0.0 for tf_workspace.log_bucket...
- tf_workspace.log_bucket in .terraform/modules/tf_workspace.log_bucket/modules/simple_bucket
Downloading registry.terraform.io/terraform-google-modules/cloud-storage/google 5.0.0 for tf_workspace.state_bucket...
- tf_workspace.state_bucket in .terraform/modules/tf_workspace.state_bucket/modules/simple_bucket

Initializing provider plugins...
- Finding hashicorp/null versions matching ">= 2.1.0"...
- Finding hashicorp/external versions matching ">= 2.2.2"...
- Finding hashicorp/google versions matching ">= 3.33.0, >= 3.43.0, >= 3.50.0, >= 3.53.0, >= 3.64.0, >= 3.67.0, >= 3.77.0, >= 3.83.0, >= 4.17.0, >= 4.25.0, >= 4.28.0, != 4.31.0, >= 4.46.0, >= 4.64.0, >= 5.7.0, < 6.0.0"...
- Finding hashicorp/random versions matching ">= 2.1.0, >= 2.2.0, >= 3.1.0, ~> 3.4"...
- Finding hashicorp/time versions matching ">= 0.5.0"...
- Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 3.50.0, >= 3.64.0, >= 3.67.0, >= 3.77.0, >= 4.11.0, >= 4.17.0, >= 4.28.0, != 4.31.0, >= 4.64.0, >= 5.7.0, < 6.0.0"...
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)
- Installing hashicorp/external v2.3.3...
- Installed hashicorp/external v2.3.3 (signed by HashiCorp)
- Installing hashicorp/google v5.19.0...
- Installed hashicorp/google v5.19.0 (signed by HashiCorp)
- Installing hashicorp/random v3.6.0...
- Installed hashicorp/random v3.6.0 (signed by HashiCorp)
- Installing hashicorp/time v0.10.0...
- Installed hashicorp/time v0.10.0 (signed by HashiCorp)
- Installing hashicorp/google-beta v5.19.0...
- Installed hashicorp/google-beta v5.19.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Terraform plan - 260

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform plan -input=false -out bootstrap.tfplan
terraform plan -input=false -out bootstrap.tfplan

20240306: 1036

Terraform vet - need a local shell

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ export VET_PROJECT_ID=tef-olapp
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform show -json bootstrap.tfplan > bootstrap.json
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud beta terraform vet bootstrap.json --policy-library="../policy-library" --project ${VET_PROJECT_ID}
Pausing command execution:

This command requires the `terraform-tools` component to be installed. Would you like to install the `terraform-tools` component to continue command execution? (Y/n)?  

ERROR: (gcloud.beta.terraform.vet) 
You cannot perform this action because the Google Cloud CLI component manager 
is disabled for this installation. You can run the following command 
to achieve the same result for this installation: 

sudo apt-get install google-cloud-sdk-terraform-tools

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ sudo apt-get install google-cloud-sdk-terraform-tools
********************************************************************************
You are running apt-get inside of Cloud Shell. Note that your Cloud Shell  
machine is ephemeral and no system-wide change will persist beyond session end. 

To suppress this warning, create an empty ~/.cloudshell/no-apt-get-warning file.
The command will automatically proceed in 5 seconds or on any key. 

Visit https://cloud.google.com/shell/help for more information.                 
********************************************************************************
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following package was automatically installed and is no longer required:
  libpcre2-posix2
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  google-cloud-sdk-terraform-tools
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 24.9 MB of archives.
After this operation, 120 MB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt cloud-sdk-bullseye/main amd64 google-cloud-sdk-terraform-tools amd64 462.0.1-0 [24.9 MB]
Fetched 24.9 MB in 2s (12.8 MB/s)                            
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package google-cloud-sdk-terraform-tools.
(Reading database ... 151687 files and directories currently installed.)
Preparing to unpack .../google-cloud-sdk-terraform-tools_462.0.1-0_amd64.deb ...
Unpacking google-cloud-sdk-terraform-tools (462.0.1-0) ...
Setting up google-cloud-sdk-terraform-tools (462.0.1-0) ...

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud beta terraform vet bootstrap.json --policy-library="../policy-library" --project ${VET_PROJECT_ID}
Validating resources...done.

Terraform apply


michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ git status
On branch main
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    terraform.example.tfvars

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        bootstrap.json
        bootstrap.tfplan

terraform apply bootstrap.tfplan

1224 running

terraform apply

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-153288813308-adc4acf2-18f5-4617-bd64-7d5df77820f6" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable cloudidentity.googleapis.com
Operation "operations/acat.p2-153288813308-796324ee-c8f6-45f6-9c6b-79c27589f037" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services list | grep NAME
NAME: cloudidentity.googleapis.com
NAME: cloudresourcemanager.googleapis.com
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable cloudapis.googleapis.com
Operation "operations/acat.p2-153288813308-0b7d17c4-8781-4af3-9e61-ccececbb4119" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable servicemanagement.googleapis.com
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable serviceusage.googleapis.com
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable storage-api.googleapis.com
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable storage.googleapis.com
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services list | grep NAME
NAME: analyticshub.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigqueryconnection.googleapis.com
NAME: bigquerydatapolicy.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigqueryreservation.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: dataform.googleapis.com
NAME: dataplex.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

check roles
Billing Account Administrator
Compute Shared VPC Admin
Folder Admin
Folder Creator
Organization Administrator
Organization Policy Administrator
Project Billing Manager
Project Creator
Project Deleter
Project IAM Admin
Service Account Token Creator
Service Usage Consumer

check https://github.com/terraform-google-modules/terraform-example-foundation/issues/965

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform apply bootstrap.tfplan

module.seed_bootstrap.random_id.suffix: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Creating...
module.tf_private_pool.random_string.suffix: Creating...
random_string.suffix: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Creation complete after 0s [id=Mco]
module.seed_bootstrap.random_id.suffix: Creation complete after 0s [id=zKQ]
random_string.suffix: Creation complete after 0s [id=wm4z]
module.tf_private_pool.random_string.suffix: Creation complete after 0s [id=4ika]
google_folder.bootstrap: Creating...
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creating...
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creating...
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Creation complete after 5s [id=630259462753/roles/billing.creator]
module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Creation complete after 9s [id=folders/1078109772786/roles/iam.serviceAccountUser/group:gcp-organization-admins@obrienlabs.app]
google_folder.bootstrap: Still creating... [10s elapsed]
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Still creating... [10s elapsed]
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Still creating... [10s elapsed]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Still creating... [10s elapsed]
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Still creating... [10s elapsed]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
google_folder.bootstrap: Creation complete after 12s [id=folders/865611452734]
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Creation complete after 14s [id=folders/1078109772786/roles/serviceusage.serviceUsageConsumer/group:gcp-organization-admins@obrienlabs.app]
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creation complete after 14s [id=folders/1078109772786/roles/resourcemanager.projectCreator/group:gcp-organization-admins@obrienlabs.app]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creating...
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creation complete after 18s [id=630259462753/roles/billing.admin/group:gcp-billing-admins@obrienlabs.app]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creation complete after 19s [id=630259462753/roles/billing.user/group:gcp-organization-admins@obrienlabs.app]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creation complete after 19s [id=630259462753/roles/resourcemanager.organizationAdmin/group:gcp-organization-admins@obrienlabs.app]
╷
│ Error: failed pre-requisites: failed to check permissions on billing account "billingAccounts/012EDD-5AD5ED-ECFF0B": googleapi: Error 403: Cloud Billing API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "cloudbilling.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main,
│   on .terraform/modules/seed_bootstrap.seed_project/modules/core_project_factory/main.tf line 73, in resource "google_project" "main":
│   73: resource "google_project" "main" {
│ 
╵
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$

billing is enabled but not the api

enabling billing api
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable cloudbilling.googleapis.com
Operation "operations/acat.p2-153288813308-9c2dddaa-7b1d-4ac0-bd9c-3fe344d1e782" finished successfully.

raised https://github.com/terraform-google-modules/terraform-example-foundation/issues/1139

1232 terraform init

terraform plan -input=false -out bootstrap.tfplan
Plan: 248 to add, 0 to change, 0 to destroy.

terraform apply

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform apply bootstrap.tfplan
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [30s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [40s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [50s elapsed]

module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m0s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m30s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m40s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [1m50s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m0s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m30s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m40s elapsed]

module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [2m50s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [3m0s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [3m10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [3m20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [3m30s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creation complete after 3m34s [id=projects/prj-b-seed-31ca]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["monitoring.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.google_resource_manager_lien.lien[0]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["accesscontextmanager.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudasset.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["essentialcontacts.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.google_resource_manager_lien.lien[0]: Creation complete after 1s [id=p830013448499-ldf597632-f200-4bf9-8345-c7388b366ed8]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Creation complete after 3s [id=prj-b-seed-31ca/compute.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Creating...

module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["accesscontextmanager.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudasset.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["essentialcontacts.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudasset.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["essentialcontacts.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/cloudbilling.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/logging.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/serviceusage.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["monitoring.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/monitoring.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudkms.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/storage-api.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/iam.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["accesscontextmanager.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/accesscontextmanager.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["essentialcontacts.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/essentialcontacts.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudasset.googleapis.com"]: Creation complete after 21s [id=prj-b-seed-31ca/cloudasset.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["pubsub.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["assuredworkloads.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iamcredentials.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["securitycenter.googleapis.com"]: Creation complete after 20s [id=prj-b-seed-31ca/securitycenter.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["iamcredentials.googleapis.com"]: Creation complete after 3s [id=prj-b-seed-31ca/iamcredentials.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creating...

s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["pubsub.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["assuredworkloads.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Still creating... [10s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudkms.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["assuredworkloads.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["pubsub.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Still creating... [20s elapsed]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Creation complete after 19s [id=prj-b-seed-31ca/servicenetworking.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["assuredworkloads.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/assuredworkloads.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creation complete after 19s [id=prj-b-seed-31ca/appengine.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/admin.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/cloudbuild.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/cloudresourcemanager.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/bigquery.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["cloudkms.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/cloudkms.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/billingbudgets.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.module.project_services.google_project_service.project_services["pubsub.googleapis.com"]: Creation complete after 22s [id=prj-b-seed-31ca/pubsub.googleapis.com]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creating...
module.seed_bootstrap.data.google_storage_project_service_account.gcs_account: Reading...
module.seed_bootstrap.module.enable_cross_project_service_account_usage.google_project_organization_policy.project_policy_boolean[0]: Creating...
module.seed_bootstrap.module.kms[0].google_kms_key_ring.key_ring: Creating...
google_service_account.terraform-env-sa["bootstrap"]: Creating...
google_service_account.terraform-env-sa["proj"]: Creating...
google_service_account.terraform-env-sa["org"]: Creating...
google_service_account.terraform-env-sa["env"]: Creating...
google_service_account.terraform-env-sa["net"]: Creating...
module.seed_bootstrap.data.google_storage_project_service_account.gcs_account: Read complete after 1s [id=service-830013448499@gs-project-accounts.iam.gserviceaccount.com]
module.seed_bootstrap.module.enable_cross_project_service_account_usage.google_project_organization_policy.project_policy_boolean[0]: Creation complete after 1s [id=prj-b-seed-31ca:constraints/iam.disableCrossProjectServiceAccountUsage]
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["env"],
│   on sa.tf line 140, in resource "google_service_account" "terraform-env-sa":
│  140: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["proj"],
│   on sa.tf line 140, in resource "google_service_account" "terraform-env-sa":
│  140: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["bootstrap"],
│   on sa.tf line 140, in resource "google_service_account" "terraform-env-sa":
│  140: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["org"],
│   on sa.tf line 140, in resource "google_service_account" "terraform-env-sa":
│  140: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with google_service_account.terraform-env-sa["net"],
│   on sa.tf line 140, in resource "google_service_account" "terraform-env-sa":
│  140: resource "google_service_account" "terraform-env-sa" {
│ 
╵
╷
│ Error: Error creating KeyRing: googleapi: Error 403: Cloud Key Management Service (KMS) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "cloudkms.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ 
│   with module.seed_bootstrap.module.kms[0].google_kms_key_ring.key_ring,
│   on .terraform/modules/seed_bootstrap.kms/main.tf line 21, in resource "google_kms_key_ring" "key_ring":
│   21: resource "google_kms_key_ring" "key_ring" {
│ 
╵
╷
│ Error: error listing service accounts on project prj-b-seed-31ca: failed to list service accounts on project "prj-b-seed-31ca": googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.seed_bootstrap.module.seed_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0],
│   on .terraform/modules/seed_bootstrap.seed_project/modules/core_project_factory/main.tf line 134, in resource "google_project_default_service_accounts" "default_service_accounts":
│  134: resource "google_project_default_service_accounts" "default_service_accounts" {
│ 
╵

1237 need iam api -

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable iam.googleapis.com
Operation "operations/acat.p2-153288813308-7f675593-6ea2-4bcc-ac0c-09b4d227de62" finished successfully.

wait 5 min - then retry apply raised https://github.com/terraform-google-modules/terraform-example-foundation/issues/1140

init, plan, 
Plan: 223 to add, 0 to change, 0 to destroy.

apply
1251

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform apply bootstrap.tfplan
module.seed_bootstrap.module.seed_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creating...
google_service_account.terraform-env-sa["bootstrap"]: Creating...
module.seed_bootstrap.module.kms[0].google_kms_key_ring.key_ring: Creating...
google_service_account.terraform-env-sa["proj"]: Creating...
google_service_account.terraform-env-sa["org"]: Creating...
google_service_account.terraform-env-sa["net"]: Creating...
google_service_account.terraform-env-sa["env"]: Creating...
module.seed_bootstrap.module.seed_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creation complete after 1s [id=projects/prj-b-seed-31ca]
google_service_account.terraform-env-sa["proj"]: Creation complete after 1s [id=projects/prj-b-seed-31ca/serviceAccounts/sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
google_service_account.terraform-env-sa["org"]: Creation complete after 2s [id=projects/prj-b-seed-31ca/serviceAccounts/sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
google_service_account.terraform-env-sa["bootstrap"]: Creation complete after 2s [id=projects/prj-b-seed-31ca/serviceAccounts/sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_service_account.terraform-env-sa["env"]: Creation complete after 2s [id=projects/prj-b-seed-31ca/serviceAccounts/sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
google_service_account.terraform-env-sa["net"]: Creation complete after 2s [id=projects/prj-b-seed-31ca/serviceAccounts/sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Creating...
google_billing_account_iam_member.tf_billing_user["proj"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creating...
google_billing_account_iam_member.tf_billing_user["net"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Creating...
google_billing_account_iam_member.tf_billing_user["env"]: Creating...
module.parent_iam_member["org"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creating...
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/logging.configWriter"]: Creating...
google_billing_account_iam_member.tf_billing_user["org"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/logging.configWriter"]: Creation complete after 5s [id=630259462753/roles/logging.configWriter/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/compute.xpnAdmin"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.xpnAdmin"]: Creation complete after 5s [id=folders/1078109772786/roles/compute.xpnAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.sourcesEditor"]: Creating...
google_billing_account_iam_member.tf_billing_user["org"]: Creation complete after 9s [id=012EDD-5AD5ED-ECFF0B/roles/billing.user/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/compute.xpnAdmin"]: Creation complete after 4s [id=630259462753/roles/compute.xpnAdmin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Creating...
google_billing_account_iam_member.tf_billing_user["env"]: Still creating... [10s elapsed]
module.parent_iam_member["org"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Still creating... [10s elapsed]
google_billing_account_iam_member.tf_billing_user["proj"]: Still creating... [10s elapsed]
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Still creating... [10s elapsed]
google_billing_account_iam_member.tf_billing_user["net"]: Still creating... [10s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.sourcesEditor"]: Still creating... [10s elapsed]
google_billing_account_iam_member.tf_billing_user["proj"]: Creation complete after 18s [id=012EDD-5AD5ED-ECFF0B/roles/billing.user/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/essentialcontacts.admin"]: Creating...
google_billing_account_iam_member.tf_billing_user["net"]: Creation complete after 18s [id=012EDD-5AD5ED-ECFF0B/roles/billing.user/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creating...
google_billing_account_iam_member.tf_billing_user["env"]: Creation complete after 18s [id=012EDD-5AD5ED-ECFF0B/roles/billing.user/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 9s [id=630259462753/roles/browser/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.notificationConfigEditor"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.sourcesEditor"]: Creation complete after 13s [id=630259462753/roles/securitycenter.sourcesEditor/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Still creating... [20s elapsed]
module.parent_iam_member["org"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [20s elapsed]
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Still creating... [20s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["org"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creation complete after 27s [id=folders/1078109772786/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderViewer"]: Creation complete after 27s [id=folders/1078109772786/roles/resourcemanager.folderViewer/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/essentialcontacts.admin"]: Still creating... [10s elapsed]
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.notificationConfigEditor"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Still creating... [10s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [10s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Still creating... [20s elapsed]
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Still creating... [30s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/essentialcontacts.admin"]: Creation complete after 13s [id=630259462753/roles/essentialcontacts.admin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/securitycenter.notificationConfigEditor"]: Creation complete after 13s [id=630259462753/roles/securitycenter.notificationConfigEditor/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Creating...
module.seed_bootstrap.google_folder_iam_binding.project_creator[0]: Creation complete after 36s [id=folders/1078109772786/roles/resourcemanager.projectCreator]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 18s [id=630259462753/roles/browser/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creation complete after 36s [id=folders/1078109772786/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/artifactregistry.admin"]: Creation complete after 27s [id=folders/1078109772786/roles/artifactregistry.admin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Creating...
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Still creating... [10s elapsed]
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Still creating... [20s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/assuredworkloads.admin"]: Creation complete after 13s [id=630259462753/roles/assuredworkloads.admin/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creating...
module.parent_iam_member["bootstrap"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creation complete after 23s [id=folders/1078109772786/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Creation complete after 14s [id=630259462753/roles/resourcemanager.tagAdmin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]

dule.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagAdmin"]: Creation complete after 14s [id=630259462753/roles/resourcemanager.tagAdmin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/dns.admin"]: Creation complete after 27s [id=folders/1078109772786/roles/dns.admin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [10s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creation complete after 17s [id=630259462753/roles/resourcemanager.organizationAdmin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [10s elapsed]
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Still creating... [20s elapsed]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Still creating... [30s elapsed]
module.parent_iam_member["env"].google_folder_iam_member.folder_parent_iam["roles/resourcemanager.folderAdmin"]: Creation complete after 31s [id=folders/1078109772786/roles/resourcemanager.folderAdmin/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityResourceAdmin"]: Creation complete after 27s [id=folders/1078109772786/roles/compute.orgSecurityResourceAdmin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/cloudasset.owner"]: Creating...
module.parent_iam_member["proj"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Creation complete after 23s [id=folders/1078109772786/roles/compute.networkAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.networkAdmin"]: Creation complete after 23s [id=folders/1078109772786/roles/compute.networkAdmin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creating...
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.securityAdmin"]: Creation complete after 23s [id=folders/1078109772786/roles/compute.securityAdmin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [20s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]

odule.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [20s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creation complete after 22s [id=630259462753/roles/resourcemanager.tagUser/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/orgpolicy.policyAdmin"]: Creating...
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 22s [id=630259462753/roles/browser/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creation complete after 22s [id=630259462753/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/cloudasset.owner"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Still creating... [10s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/orgpolicy.policyAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/cloudasset.owner"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [20s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Still creating... [20s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/orgpolicy.policyAdmin"]: Still creating... [20s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [20s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [30s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [40s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [20s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/cloudasset.owner"]: Still creating... [30s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 44s [id=630259462753/roles/browser/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [30s elapsed]
google_billing_account_iam_member.tf_billing_user["bootstrap"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Still creating... [30s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/orgpolicy.policyAdmin"]: Still creating... [30s elapsed]
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [30s elapsed]
google_billing_account_iam_member.tf_billing_user["bootstrap"]: Creation complete after 4s [id=012EDD-5AD5ED-ECFF0B/roles/billing.user/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [40s elapsed]

ntextmanager.policyAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [30s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [40s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/cloudasset.owner"]: Still creating... [40s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Still creating... [40s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [40s elapsed]
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Still creating... [40s elapsed]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Creation complete after 43s [id=630259462753/roles/serviceusage.serviceUsageConsumer/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityPolicyAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/cloudasset.owner"]: Creation complete after 43s [id=630259462753/roles/cloudasset.owner/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/browser"]: Creation complete after 42s [id=630259462753/roles/browser/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/storage.admin"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creation complete after 43s [id=630259462753/roles/resourcemanager.organizationAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["env"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationViewer"]: Creation complete after 43s [id=630259462753/roles/resourcemanager.organizationViewer/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creating...
module.org_iam_member["org"].google_organization_iam_member.org_parent_iam["roles/orgpolicy.policyAdmin"]: Creation complete after 40s [id=630259462753/roles/orgpolicy.policyAdmin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudkms.admin"]: Creating...
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Still creating... [40s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]
module.parent_iam_member["net"].google_folder_iam_member.folder_parent_iam["roles/compute.orgSecurityPolicyAdmin"]: Creation complete after 5s [id=folders/1078109772786/roles/compute.orgSecurityPolicyAdmin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creating...
module.org_iam_member["env"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.tagUser"]: Creation complete after 43s [id=630259462753/roles/resourcemanager.tagUser/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["proj"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Still creating... [10s elapsed]
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [40s elapsed]

odule.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Still creating... [40s elapsed]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudkms.admin"]: Creation complete after 7s [id=prj-b-seed-31ca/roles/cloudkms.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["net"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creating...
module.org_iam_member["proj"].google_organization_iam_member.org_parent_iam["roles/serviceusage.serviceUsageConsumer"]: Creation complete after 44s [id=630259462753/roles/serviceusage.serviceUsageConsumer/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["org"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [10s elapsed]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/storage.admin"]: Still creating... [10s elapsed]
module.seed_project_iam_member["env"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Still creating... [10s elapsed]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Still creating... [10s elapsed]
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Still creating... [20s elapsed]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/storage.admin"]: Creation complete after 13s [id=prj-b-seed-31ca/roles/storage.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["net"]: Creating...
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creation complete after 12s [id=prj-b-seed-31ca/roles/resourcemanager.projectDeleter/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["proj"]: Creating...
module.seed_project_iam_member["env"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creation complete after 12s [id=prj-b-seed-31ca/roles/storage.objectAdmin/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["org"]: Creating...
module.org_iam_member["net"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creation complete after 22s [id=630259462753/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["bootstrap"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/resourcemanager.organizationAdmin"]: Creation complete after 18s [id=630259462753/roles/resourcemanager.organizationAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["env"]: Creating...
module.org_iam_member["bootstrap"].google_organization_iam_member.org_parent_iam["roles/accesscontextmanager.policyAdmin"]: Creation complete after 15s [id=630259462753/roles/accesscontextmanager.policyAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Still creating... [10s elapsed]
module.seed_project_iam_member["proj"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["net"]: Creation complete after 4s [id=012EDD-5AD5ED-ECFF0B/roles/billing.admin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["proj"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creation complete after 12s [id=prj-b-seed-31ca/roles/storage.objectAdmin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creation complete after 12s [id=prj-b-seed-31ca/roles/iam.serviceAccountAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["org"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creation complete after 7s [id=prj-b-seed-31ca/roles/storage.objectAdmin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.seed_project_iam_member["net"].google_project_iam_member.project_parent_iam["roles/storage.objectAdmin"]: Creation complete after 10s [id=prj-b-seed-31ca/roles/storage.objectAdmin/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]

google_billing_account_iam_member.billing_admin_user["proj"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["org"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["bootstrap"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["env"]: Still creating... [10s elapsed]
google_billing_account_iam_member.billing_admin_user["proj"]: Still creating... [20s elapsed]
google_billing_account_iam_member.billing_admin_user["org"]: Still creating... [20s elapsed]
google_billing_account_iam_member.billing_admin_user["proj"]: Creation complete after 21s [id=012EDD-5AD5ED-ECFF0B/roles/billing.admin/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["org"]: Creation complete after 21s [id=012EDD-5AD5ED-ECFF0B/roles/billing.admin/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["bootstrap"]: Creation complete after 20s [id=012EDD-5AD5ED-ECFF0B/roles/billing.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_billing_account_iam_member.billing_admin_user["env"]: Creation complete after 20s [id=012EDD-5AD5ED-ECFF0B/roles/billing.admin/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
╷
│ Error: Error creating KeyRing: googleapi: Error 403: Cloud Key Management Service (KMS) API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "cloudkms.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ 
│   with module.seed_bootstrap.module.kms[0].google_kms_key_ring.key_ring,
│   on .terraform/modules/seed_bootstrap.kms/main.tf line 21, in resource "google_kms_key_ring" "key_ring":
│   21: resource "google_kms_key_ring" "key_ring" {
│

need cloudkms

For Terraform 1.3.7 upgrade https://github.com/terraform-google-modules/terraform-example-foundation/issues/1141 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1142

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable cloudkms.googleapis.com
Operation "operations/acat.p2-153288813308-f346fb9f-e5a4-4ced-ba6a-d5b82c442f68" finished successfully.

0720 rerun terraform init/plan/apply

Plan: 159 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cloud_build_peered_network_id                     = (known after apply)
  + cloud_build_private_worker_pool_id                = (known after apply)
  + cloud_build_worker_range_id                       = (known after apply)
  + cloud_builder_artifact_repo                       = (known after apply)
  + csr_repos                                         = {
      + gcp-bootstrap    = {
          + id      = (known after apply)
          + name    = "gcp-bootstrap"
          + project = "prj-b-cicd-wm4z"
          + url     = (known after apply)
        }
      + gcp-environments = {
          + id      = (known after apply)
          + name    = "gcp-environments"
          + project = "prj-b-cicd-wm4z"
          + url     = (known after apply)
        }
      + gcp-networks     = {
          + id      = (known after apply)
          + name    = "gcp-networks"
          + project = "prj-b-cicd-wm4z"
          + url     = (known after apply)
        }
      + gcp-org          = {
          + id      = (known after apply)
          + name    = "gcp-org"
          + project = "prj-b-cicd-wm4z"
          + url     = (known after apply)
        }
      + gcp-policies     = {
          + id      = (known after apply)
          + name    = "gcp-policies"
          + project = "prj-b-cicd-wm4z"
          + url     = (known after apply)
        }
      + gcp-projects     = {
          + id      = (known after apply)
          + name    = "gcp-projects"
          + project = "prj-b-cicd-wm4z"
          + url     = (known after apply)
        }
      + tf-cloudbuilder  = {
          + id      = (known after apply)
          + name    = "tf-cloudbuilder"
          + project = "prj-b-cicd-wm4z"
          + url     = (known after apply)
        }
    }
  + gcs_bucket_cloudbuild_artifacts                   = {
      + bootstrap = (known after apply)
      + env       = (known after apply)
      + net       = (known after apply)
      + org       = (known after apply)
      + proj      = (known after apply)
    }
  + gcs_bucket_cloudbuild_logs                        = {
      + bootstrap = (known after apply)
      + env       = (known after apply)
      + net       = (known after apply)
      + org       = (known after apply)
      + proj      = (known after apply)
    }

down to 159

expecting more service enablement issues

The list is in https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/main.tf#L78

activate_apis = [
    "serviceusage.googleapis.com",
    "servicenetworking.googleapis.com",
    "cloudkms.googleapis.com",
    "compute.googleapis.com",
    "logging.googleapis.com",
    "bigquery.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "cloudbilling.googleapis.com",
    "cloudbuild.googleapis.com",
    "iam.googleapis.com",
    "admin.googleapis.com",
    "appengine.googleapis.com",
    "storage-api.googleapis.com",
    "monitoring.googleapis.com",
    "pubsub.googleapis.com",
    "securitycenter.googleapis.com",
    "accesscontextmanager.googleapis.com",
    "billingbudgets.googleapis.com",
    "essentialcontacts.googleapis.com",
    "assuredworkloads.googleapis.com",
    "cloudasset.googleapis.com"
  ]

terraform apply

0726

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform apply bootstrap.tfplan

module.seed_bootstrap.module.kms[0].google_kms_key_ring.key_ring: Creating...
module.seed_bootstrap.module.kms[0].google_kms_key_ring.key_ring: Creation complete after 0s [id=projects/prj-b-seed-31ca/locations/us-central1/keyRings/prj-keyring]
module.seed_bootstrap.module.kms[0].google_kms_crypto_key.key[0]: Creating...
module.seed_bootstrap.module.kms[0].google_kms_crypto_key.key[0]: Creation complete after 1s [id=projects/prj-b-seed-31ca/locations/us-central1/keyRings/prj-keyring/cryptoKeys/prj-key]
module.seed_bootstrap.module.kms[0].google_kms_crypto_key_iam_binding.decrypters[0]: Creating...
module.seed_bootstrap.module.kms[0].google_kms_crypto_key_iam_binding.encrypters[0]: Creating...
module.seed_bootstrap.module.kms[0].google_kms_crypto_key_iam_binding.encrypters[0]: Creation complete after 8s [id=projects/prj-b-seed-31ca/locations/us-central1/keyRings/prj-keyring/cryptoKeys/prj-key/roles/cloudkms.cryptoKeyEncrypter]
module.seed_bootstrap.module.kms[0].google_kms_crypto_key_iam_binding.decrypters[0]: Creation complete after 8s [id=projects/prj-b-seed-31ca/locations/us-central1/keyRings/prj-keyring/cryptoKeys/prj-key/roles/cloudkms.cryptoKeyDecrypter]
module.seed_bootstrap.google_storage_bucket.org_terraform_state: Creating...
module.seed_bootstrap.google_storage_bucket.org_terraform_state: Creation complete after 2s [id=bkt-prj-b-seed-tfstate-cca4]
module.seed_bootstrap.google_storage_bucket_iam_member.orgadmins_state_iam[0]: Creating...
module.gcp_projects_state_bucket.google_storage_bucket.bucket: Creating...
module.gcp_projects_state_bucket.google_storage_bucket.bucket: Creation complete after 1s [id=bkt-prj-b-seed-31ca-gcp-projects-tfstate]
module.seed_bootstrap.google_storage_bucket_iam_member.orgadmins_state_iam[0]: Creation complete after 4s [id=b/bkt-prj-b-seed-tfstate-cca4/roles/storage.admin/group:gcp-organization-admins@obrienlabs.app]
module.tf_source.module.cloudbuild_project.module.project-factory.random_id.random_project_id_suffix: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.random_id.random_project_id_suffix: Creation complete after 0s [id=wBU]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Creating...

odule.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [40s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [50s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m0s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m20s elapsed]

module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m40s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [1m50s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m0s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m40s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [2m50s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [3m0s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [3m10s elapsed]

module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [3m20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Still creating... [3m30s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project.main: Creation complete after 3m34s [id=projects/prj-b-cicd-wm4z]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["dns.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.google_service_account.default_service_account[0]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["sourcerepo.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.google_service_account.default_service_account[0]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/serviceAccounts/project-service-account@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["compute.googleapis.com"]: Creation complete after 3s [id=prj-b-cicd-wm4z/compute.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Creating...

module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["dns.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["sourcerepo.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["dns.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["sourcerepo.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["sourcerepo.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/sourcerepo.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["storage-api.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/storage-api.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["serviceusage.googleapis.com"]: Creation complete after 19s [id=prj-b-cicd-wm4z/serviceusage.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["billingbudgets.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/billingbudgets.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["artifactregistry.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/artifactregistry.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["servicenetworking.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/servicenetworking.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudscheduler.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbuild.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/cloudbuild.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["dns.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/dns.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudresourcemanager.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/cloudresourcemanager.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["bigquery.googleapis.com"]: Creation complete after 21s [id=prj-b-cicd-wm4z/bigquery.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["logging.googleapis.com"]: Creation complete after 3s [id=prj-b-cicd-wm4z/logging.googleapis.com]

module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudscheduler.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Still creating... [10s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudscheduler.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Still creating... [20s elapsed]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["appengine.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/appengine.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["admin.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/admin.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudbilling.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/cloudbilling.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["workflows.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/workflows.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["iam.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/iam.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.module.project_services.google_project_service.project_services["cloudscheduler.googleapis.com"]: Creation complete after 22s [id=prj-b-cicd-wm4z/cloudscheduler.googleapis.com]
module.tf_source.module.cloudbuild_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"]: Creating...
module.tf_source.google_project_iam_member.org_admins_cloudbuild_viewer: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"]: Creating...
module.tf_source.google_project_iam_member.org_admins_source_repo_admin[0]: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"]: Creating...
module.tf_source.module.cloudbuild_project.module.project-factory.google_project_default_service_accounts.default_service_accounts[0]: Creation complete after 0s [id=projects/prj-b-cicd-wm4z]
module.tf_source.google_project_iam_member.org_admins_cloudbuild_editor: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["tf-cloudbuilder"]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/repos/tf-cloudbuilder]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-bootstrap"]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/repos/gcp-bootstrap]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-projects"]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/repos/gcp-projects]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-policies"]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-org"]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/repos/gcp-org]
module.tf_source.module.cloudbuild_bucket.google_storage_bucket.bucket: Creating...
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-networks"]: Creation complete after 2s [id=projects/prj-b-cicd-wm4z/repos/gcp-networks]
module.tf_source.module.cloudbuild_bucket.google_storage_bucket.bucket: Creation complete after 1s [id=prj-b-cicd-wm4z_cloudbuild]
module.tf_source.google_sourcerepo_repository.gcp_repo["gcp-environments"]: Creation complete after 2s [id=projects/prj-b-cicd-wm4z/repos/gcp-environments]
module.tf_source.google_storage_bucket_iam_member.cloudbuild_iam: Creating...

module.tf_source.google_storage_bucket_iam_member.cloudbuild_iam: Creation complete after 5s [id=b/prj-b-cicd-wm4z_cloudbuild/roles/storage.admin/serviceAccount:1083787941178@cloudbuild.gserviceaccount.com]
module.tf_source.google_project_iam_member.org_admins_cloudbuild_editor: Creation complete after 8s [id=prj-b-cicd-wm4z/roles/cloudbuild.builds.editor/group:gcp-organization-admins@obrienlabs.app]
module.tf_source.google_project_iam_member.org_admins_cloudbuild_viewer: Creation complete after 8s [id=prj-b-cicd-wm4z/roles/viewer/group:gcp-organization-admins@obrienlabs.app]
module.tf_source.google_project_iam_member.org_admins_source_repo_admin[0]: Creation complete after 8s [id=prj-b-cicd-wm4z/roles/source.admin/group:gcp-organization-admins@obrienlabs.app]
module.bootstrap_csr_repo.null_resource.run_destroy_command[0]: Creating...
module.tf_private_pool.module.peered_network[0].module.vpc.google_compute_network.network: Creating...
google_sourcerepo_repository_iam_member.member["proj"]: Creating...
google_sourcerepo_repository_iam_member.member["bootstrap"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/storage.admin"]: Creating...
google_sourcerepo_repository_iam_member.member["env"]: Creating...
module.tf_cloud_builder.google_service_account.workflow_sa[0]: Creating...
google_sourcerepo_repository_iam_member.member["net"]: Creating...
google_sourcerepo_repository_iam_member.member["org"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/artifactregistry.admin"]: Creating...
module.bootstrap_csr_repo.null_resource.run_destroy_command[0]: Creation complete after 0s [id=6607708089699954645]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/compute.networkAdmin"]: Creating...
module.tf_cloud_builder.google_service_account.workflow_sa[0]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/serviceAccounts/terraform-runner-workflow-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.workerPoolOwner"]: Creating...
google_sourcerepo_repository_iam_member.member["bootstrap"]: Creation complete after 4s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.workerPoolOwner"]: Creation complete after 7s [id=prj-b-cicd-wm4z/roles/cloudbuild.workerPoolOwner/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/source.admin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/storage.admin"]: Creation complete after 8s [id=prj-b-cicd-wm4z/roles/storage.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/dns.admin"]: Creating...

te.networkAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/workflows.admin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/artifactregistry.admin"]: Creation complete after 12s [id=prj-b-cicd-wm4z/roles/artifactregistry.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.builds.editor"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Still creating... [10s elapsed]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/dns.admin"]: Creation complete after 9s [id=prj-b-cicd-wm4z/roles/dns.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.workloadIdentityPoolAdmin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/source.admin"]: Creation complete after 9s [id=prj-b-cicd-wm4z/roles/source.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creating...
google_sourcerepo_repository_iam_member.member["proj"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudscheduler.admin"]: Creating...
google_sourcerepo_repository_iam_member.member["env"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
module.bootstrap_csr_repo.null_resource.run_command[0]: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0]: Provisioning with 'local-exec'...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Executing: ["/bin/sh" "-c" "PATH=/google-cloud-sdk/bin:$PATH\n./scripts/push-to-repo.sh prj-b-cicd-wm4z tf-cloudbuilder ./Dockerfile\n"]
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + '[' 3 -lt 3 ']'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + CSR_PROJECT_ID=prj-b-cicd-wm4z
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + CSR_NAME=tf-cloudbuilder
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + DOCKERFILE_PATH=./Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): ++ mktemp -d
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + tmp_dir=/tmp/tmp.iGWG1EfS69
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + gcloud source repos clone tf-cloudbuilder /tmp/tmp.iGWG1EfS69 --project prj-b-cicd-wm4z
google_sourcerepo_repository_iam_member.member["net"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.tf_private_pool.module.peered_network[0].module.vpc.google_compute_network.network: Still creating... [20s elapsed]
google_sourcerepo_repository_iam_member.member["org"]: Still creating... [20s elapsed]
google_sourcerepo_repository_iam_member.member["org"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.tf_cloud_builder.google_service_account.cb_sa[0]: Creating...
module.tf_cloud_builder.google_artifact_registry_repository.tf-image-repo: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Cloning into '/tmp/tmp.iGWG1EfS69'...
module.tf_cloud_builder.google_service_account.cb_sa[0]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/serviceAccounts/tf-cb-builder-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creation complete after 18s [id=prj-b-cicd-wm4z/roles/iam.serviceAccountAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.tf_cloud_builder.google_project_iam_member.invoke_workflow_scheduler: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/workflows.admin"]: Still creating... [10s elapsed]
module.tf_private_pool.module.peered_network[0].module.vpc.google_compute_network.network: Creation complete after 22s [id=projects/prj-b-cicd-wm4z/global/networks/vpc-b-cbpools]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.builds.editor"]: Still creating... [10s elapsed]
module.tf_cloud_builder.google_workflows_workflow.builder: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): warning: You appear to have cloned an empty repository.
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Project [prj-b-cicd-wm4z] repository [tf-cloudbuilder] was cloned to [/tmp/tmp.iGWG1EfS69].
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + cp ./Dockerfile /tmp/tmp.iGWG1EfS69
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + pushd /tmp/tmp.iGWG1EfS69
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): /tmp/tmp.iGWG1EfS69 ~/tef-olapp/github/terraform-example-foundation/0-bootstrap
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config credential.helper gcloud.sh
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config init.defaultBranch main
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config user.email terraform-robot@example.com
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config user.name 'TF Robot'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git checkout main
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): error: pathspec 'main' did not match any file(s) known to git
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git checkout -b main
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Switched to a new branch 'main'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git add Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git commit -m 'Initialize tf dockerfile repo'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): [main (root-commit) 55aa00d] Initialize tf dockerfile repo
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  1 file changed, 39 insertions(+)
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  create mode 100644 Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git push origin main -f

te.networkAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/workflows.admin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/artifactregistry.admin"]: Creation complete after 12s [id=prj-b-cicd-wm4z/roles/artifactregistry.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.builds.editor"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Still creating... [10s elapsed]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/dns.admin"]: Creation complete after 9s [id=prj-b-cicd-wm4z/roles/dns.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.workloadIdentityPoolAdmin"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/source.admin"]: Creation complete after 9s [id=prj-b-cicd-wm4z/roles/source.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/resourcemanager.projectDeleter"]: Creating...
google_sourcerepo_repository_iam_member.member["proj"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudscheduler.admin"]: Creating...
google_sourcerepo_repository_iam_member.member["env"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
module.bootstrap_csr_repo.null_resource.run_command[0]: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0]: Provisioning with 'local-exec'...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Executing: ["/bin/sh" "-c" "PATH=/google-cloud-sdk/bin:$PATH\n./scripts/push-to-repo.sh prj-b-cicd-wm4z tf-cloudbuilder ./Dockerfile\n"]
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + '[' 3 -lt 3 ']'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + CSR_PROJECT_ID=prj-b-cicd-wm4z
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + CSR_NAME=tf-cloudbuilder
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + DOCKERFILE_PATH=./Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): ++ mktemp -d
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + tmp_dir=/tmp/tmp.iGWG1EfS69
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + gcloud source repos clone tf-cloudbuilder /tmp/tmp.iGWG1EfS69 --project prj-b-cicd-wm4z
google_sourcerepo_repository_iam_member.member["net"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
module.tf_private_pool.module.peered_network[0].module.vpc.google_compute_network.network: Still creating... [20s elapsed]
google_sourcerepo_repository_iam_member.member["org"]: Still creating... [20s elapsed]
google_sourcerepo_repository_iam_member.member["org"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/repos/gcp-policies/roles/viewer/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
module.tf_cloud_builder.google_service_account.cb_sa[0]: Creating...
module.tf_cloud_builder.google_artifact_registry_repository.tf-image-repo: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Cloning into '/tmp/tmp.iGWG1EfS69'...
module.tf_cloud_builder.google_service_account.cb_sa[0]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/serviceAccounts/tf-cb-builder-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.serviceAccountAdmin"]: Creation complete after 18s [id=prj-b-cicd-wm4z/roles/iam.serviceAccountAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.tf_cloud_builder.google_project_iam_member.invoke_workflow_scheduler: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/workflows.admin"]: Still creating... [10s elapsed]
module.tf_private_pool.module.peered_network[0].module.vpc.google_compute_network.network: Creation complete after 22s [id=projects/prj-b-cicd-wm4z/global/networks/vpc-b-cbpools]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.builds.editor"]: Still creating... [10s elapsed]
module.tf_cloud_builder.google_workflows_workflow.builder: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): warning: You appear to have cloned an empty repository.
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Project [prj-b-cicd-wm4z] repository [tf-cloudbuilder] was cloned to [/tmp/tmp.iGWG1EfS69].
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + cp ./Dockerfile /tmp/tmp.iGWG1EfS69
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + pushd /tmp/tmp.iGWG1EfS69
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): /tmp/tmp.iGWG1EfS69 ~/tef-olapp/github/terraform-example-foundation/0-bootstrap
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config credential.helper gcloud.sh
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config init.defaultBranch main
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config user.email terraform-robot@example.com
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git config user.name 'TF Robot'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git checkout main
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): error: pathspec 'main' did not match any file(s) known to git
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git checkout -b main
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): Switched to a new branch 'main'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git add Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git commit -m 'Initialize tf dockerfile repo'
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): [main (root-commit) 55aa00d] Initialize tf dockerfile repo
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  1 file changed, 39 insertions(+)
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  create mode 100644 Dockerfile
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): + git push origin main -f

e.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/workflows.admin"]: Creation complete after 16s [id=prj-b-cicd-wm4z/roles/workflows.admin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
module.tf_cloud_builder.google_project_iam_member.logs_writer: Creating...
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): remote: Waiting for private key checker: 1/1 objects left
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec): To https://source.developers.google.com/p/prj-b-cicd-wm4z/r/tf-cloudbuilder
module.bootstrap_csr_repo.null_resource.run_command[0] (local-exec):  * [new branch]      main -> main
module.bootstrap_csr_repo.null_resource.run_command[0]: Creation complete after 8s [id=2083100521623893606]
module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["us-central1/sb-b-cbpools-us-central1"]: Creating...
module.tf_cloud_builder.google_cloud_scheduler_job.trigger_workflow: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/locations/us-central1/jobs/trigger-terraform-runner-workflow]
module.tf_private_pool.google_dns_policy.default_policy[0]: Creating...
module.tf_private_pool.google_dns_policy.default_policy[0]: Creation complete after 1s [id=projects/prj-b-cicd-wm4z/policies/dp-b-cbpools-default-policy]
module.tf_cloud_builder.google_storage_bucket_iam_member.member: Creating...
module.tf_cloud_builder.google_artifact_registry_repository.tf-image-repo: Still creating... [10s elapsed]
module.tf_cloud_builder.google_service_account_iam_member.use_cb_sa: Creation complete after 4s [id=projects/prj-b-cicd-wm4z/serviceAccounts/tf-cb-builder-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com/roles/iam.serviceAccountUser/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.tf_private_pool.google_compute_global_address.worker_pool_range[0]: Creating...
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Still creating... [10s elapsed]
module.tf_cloud_builder.google_artifact_registry_repository.tf-image-repo: Creation complete after 11s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.workflow_list: Creating...
module.tf_cloud_builder.google_sourcerepo_repository_iam_member.member[0]: Creation complete after 4s [id=projects/prj-b-cicd-wm4z/repos/tf-cloudbuilder/roles/viewer/serviceAccount:tf-cb-builder-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.push_images: Creating...
module.tf_cloud_builder.google_project_iam_member.invoke_workflow_scheduler: Still creating... [10s elapsed]
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/cloudbuild.builds.editor"]: Creation complete after 19s [id=prj-b-cicd-wm4z/roles/cloudbuild.builds.editor/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Creating...
module.cicd_project_iam_member["bootstrap"].google_project_iam_member.project_parent_iam["roles/iam.workloadIdentityPoolAdmin"]: Creation complete after 15s [id=prj-b-cicd-wm4z/roles/iam.workloadIdentityPoolAdmin/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["env"]: Creating...
module.tf_cloud_builder.google_storage_bucket_iam_member.member: Creation complete after 4s [id=b/bkt-prj-b-cicd-wm4z-tf-cloudbuilder-build-logs/roles/storage.admin/serviceAccount:tf-cb-builder-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Creating...
module.tf_cloud_builder.google_project_iam_member.trigger_builds: Creation complete after 16s [id=prj-b-cicd-wm4z/roles/cloudbuild.builds.editor/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["proj"]: Creating...
module.tf_cloud_builder.google_project_iam_member.invoke_workflow_scheduler: Creation complete after 15s [id=prj-b-cicd-wm4z/roles/workflows.invoker/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["org"]: Creating...
module.tf_cloud_builder.google_project_iam_member.logs_writer: Creation complete after 9s [id=prj-b-cicd-wm4z/roles/logging.logWriter/serviceAccount:tf-cb-builder-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.bootstrap_projects_remove_editor["cicd"].google_project_iam_binding.iam_remove["roles/editor"]: Creating...
module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["us-central1/sb-b-cbpools-us-central1"]: Still creating... [10s elapsed]
module.tf_private_pool.module.peered_network[0].module.subnets.google_compute_subnetwork.subnetwork["us-central1/sb-b-cbpools-us-central1"]: Creation complete after 12s [id=projects/prj-b-cicd-wm4z/regions/us-central1/subnetworks/sb-b-cbpools-us-central1]
module.bootstrap_projects_remove_editor["seed"].google_project_iam_binding.iam_remove["roles/editor"]: Creating...
module.tf_private_pool.google_compute_global_address.worker_pool_range[0]: Still creating... [10s elapsed]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.push_images: Creation complete after 9s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners/roles/artifactregistry.writer/serviceAccount:tf-cb-builder-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.tf_cloud_builder.google_artifact_registry_repository_iam_member.workflow_list: Creation complete after 10s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:terraform-runner-workflow-sa@prj-b-cicd-wm4z.iam.gserviceaccount.com]
module.tf_private_pool.google_compute_global_address.worker_pool_range[0]: Creation complete after 11s [id=projects/prj-b-cicd-wm4z/global/addresses/ga-b-cbpools-worker-pool-range]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Creating...
module.tf_private_pool.module.firewall_rules[0].google_compute_firewall.rules["fw-b-cbpools-100-i-a-all-all-all-service-networking"]: Creating...
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["env"]: Creation complete after 10s [id=proj

oogle_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["env"]: Creation complete after 10s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Still creating... [10s elapsed]
module.bootstrap_projects_remove_editor["cicd"].google_project_iam_binding.iam_remove["roles/editor"]: Creation complete after 7s [id=prj-b-cicd-wm4z/roles/editor]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Still creating... [10s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["proj"]: Still creating... [10s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["org"]: Still creating... [10s elapsed]
module.bootstrap_projects_remove_editor["seed"].google_project_iam_binding.iam_remove["roles/editor"]: Creation complete after 7s [id=prj-b-seed-31ca/roles/editor]
module.tf_private_pool.module.firewall_rules[0].google_compute_firewall.rules["fw-b-cbpools-100-i-a-all-all-all-service-networking"]: Still creating... [10s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Still creating... [20s elapsed]
module.tf_private_pool.module.firewall_rules[0].google_compute_firewall.rules["fw-b-cbpools-100-i-a-all-all-all-service-networking"]: Creation complete after 11s [id=projects/prj-b-cicd-wm4z/global/firewalls/fw-b-cbpools-100-i-a-all-all-all-service-networking]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Still creating... [20s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["bootstrap"]: Creation complete after 25s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["proj"]: Still creating... [20s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["org"]: Still creating... [20s elapsed]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["net"]: Creation complete after 23s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["proj"]: Creation complete after 20s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com]
google_artifact_registry_repository_iam_member.terraform_sa_artifact_registry_reader["org"]: Creation complete after 21s [id=projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners/roles/artifactregistry.reader/serviceAccount:sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com]
╷
│ Error: Error waiting for Create Service Networking Connection: error while retrieving operation: googleapi: Error 403: Service Networking API has not been used in project tef-olapp before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/servicenetworking.googleapis.com/overview?project=tef-olapp then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/servicenetworking.googleapis.com/overview?project=tef-olapp"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/tef-olapp",
│       "service": "servicenetworking.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0],
│   on modules/cb-private-pool/network.tf line 72, in resource "google_service_networking_connection" "worker_pool_conn":
│   72: resource "google_service_networking_connection" "worker_pool_conn" {
│

0732

current list

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services list | grep NAME
NAME: analyticshub.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigqueryconnection.googleapis.com
NAME: bigquerydatapolicy.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigqueryreservation.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudbilling.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudkms.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: dataform.googleapis.com
NAME: dataplex.googleapis.com
NAME: datastore.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: servicenetworking.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

I am going to enable all services below The list is in https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/main.tf#L78

activate_apis = [
    "serviceusage.googleapis.com",
    "servicenetworking.googleapis.com",
    "cloudkms.googleapis.com",
    "compute.googleapis.com",
    "logging.googleapis.com",
    "bigquery.googleapis.com",
    "cloudresourcemanager.googleapis.com",
    "cloudbilling.googleapis.com",
    "cloudbuild.googleapis.com",
    "iam.googleapis.com",
    "admin.googleapis.com",
    "appengine.googleapis.com",
    "storage-api.googleapis.com",
    "monitoring.googleapis.com",
    "pubsub.googleapis.com",
    "securitycenter.googleapis.com",
    "accesscontextmanager.googleapis.com",
    "billingbudgets.googleapis.com",
    "essentialcontacts.googleapis.com",
    "assuredworkloads.googleapis.com",
    "cloudasset.googleapis.com"
  ]

enabling - even though most of these are for CB project

cloudbuild.googleapis.com
appengine.googleapis.com
pubsub.googleapis.com
securitycenter.googleapis.com
accesscontextmanager.googleapis.com
billingbudgets.googleapis.com
essentialcontacts.googleapis.com
assuredworkloads.googleapis.com
cloudasset.googleapis.com

https://github.com/terraform-google-modules/terraform-example-foundation/issues/1143

more service enablements

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable cloudbuild.googleapis.com
Operation "operations/acf.p2-153288813308-9511143e-75a0-473a-b019-63c3fd280ff7" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable appengine.googleapis.com
Operation "operations/acat.p2-153288813308-787a46f6-f539-4fa5-8f60-b7ca079e6baf" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable pubsub.googleapis.com
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable securitycenter.googleapis.com
Operation "operations/acat.p2-153288813308-2e9e4ed9-3423-4a58-9709-70c31d1623c3" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable accesscontextmanager.googleapis.com
Operation "operations/acat.p2-153288813308-8bcee864-8cb1-45ab-9cbc-d10d889e75c3" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable billingbudgets.googleapis.com
Operation "operations/acat.p2-153288813308-3bfd1b6b-068a-434a-b2db-42841928c4dc" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable essentialcontacts.googleapis.com
Operation "operations/acat.p2-153288813308-e747eb4d-6c9f-48fd-8791-b96b3b4b205a" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable assuredworkloads.googleapis.com
Operation "operations/acat.p2-153288813308-a0f14a88-ae37-4d11-8ca7-e500adf89572" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ gcloud services enable cloudasset.googleapis.com
Operation "operations/acat.p2-153288813308-3a94bdb9-ca59-4b9f-8146-9150d57eb568" finished successfully.
michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$

review https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/sa.tf#L34

// Roles required to manage resources in the Seed project
  granular_sa_seed_project = {
    "bootstrap" = [
      "roles/storage.admin",
      "roles/iam.serviceAccountAdmin",
      "roles/resourcemanager.projectDeleter",
      "roles/cloudkms.admin",
    ],
    "org" = [
      "roles/storage.objectAdmin",
    ],
    "env" = [
      "roles/storage.objectAdmin"
    ],
    "net" = [
      "roles/storage.objectAdmin",
    ],
    "proj" = [
      "roles/storage.objectAdmin",
    ],
  }

  // Roles required to manage resources in the CI/CD project
  granular_sa_cicd_project = {
    "bootstrap" = [
      "roles/storage.admin",
      "roles/compute.networkAdmin",
      "roles/cloudbuild.builds.editor",
      "roles/cloudbuild.workerPoolOwner",
      "roles/artifactregistry.admin",
      "roles/source.admin",
      "roles/iam.serviceAccountAdmin",
      "roles/workflows.admin",
      "roles/cloudscheduler.admin",
      "roles/resourcemanager.projectDeleter",
      "roles/dns.admin",
      "roles/iam.workloadIdentityPoolAdmin",
    ],
  }

Terraform apply 0-bootstrap

Plan: 73 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + cloud_build_private_worker_pool_id                = (known after apply)
  + gcs_bucket_cloudbuild_artifacts                   = {
      + bootstrap = (known after apply)
      + env       = (known after apply)
      + net       = (known after apply)
      + org       = (known after apply)
      + proj      = (known after apply)
    }
  + gcs_bucket_cloudbuild_logs                        = {
      + bootstrap = (known after apply)
      + env       = (known after apply)
      + net       = (known after apply)
      + org       = (known after apply)
      + proj      = (known after apply)
    }

check terraform cloud version (still modules) jetbrains intellij 2023 (up from 2021) - find references

use $terraform-google-modules = ../modules.. IntelliJ IDEA 2023.3.4 available terraform 233 marketplace plugin

0915

ichael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform apply bootstrap.tfplan
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Creating...

module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Still creating... [10s elapsed]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Still creating... [20s elapsed]
module.tf_private_pool.google_service_networking_connection.worker_pool_conn[0]: Creation complete after 21s [id=projects%2Fprj-b-cicd-wm4z%2Fglobal%2Fnetworks%2Fvpc-b-cbpools:servicenetworking.googleapis.com]
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Creating...
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Creating...
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Still creating... [10s elapsed]
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Still creating... [10s elapsed]
module.tf_private_pool.google_compute_network_peering_routes_config.peering_routes[0]: Creation complete after 11s [id=projects/prj-b-cicd-wm4z/global/networks/vpc-b-cbpools/networkPeerings/servicenetworking-googleapis-com]
module.tf_private_pool.google_cloudbuild_worker_pool.private_pool: Still creating... [20s elapsed]

https://ccticei@dev.azure.com/ccticei/Migration/_git/TEF-GCP-LZ-HS

odule.build_terraform_image.null_resource.run_command[0] (local-exec):     timeout: 1200s
module.build_terraform_image.null_resource.run_command[0] (local-exec): name: operations/build/prj-b-cicd-wm4z/YTRmODk0MTEtYWNiZi00NDZkLTgwMTAtMThmOWFmNjhiOTAx
module.build_terraform_image.null_resource.run_command[0]: Creation complete after 3s [id=6137778600788507520]

Apply complete! Resources: 73 added, 0 changed, 0 destroyed.

Outputs:

bootstrap_step_terraform_service_account_email = "sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com"
cloud_build_peered_network_id = "projects/prj-b-cicd-wm4z/global/networks/vpc-b-cbpools"
cloud_build_private_worker_pool_id = "projects/prj-b-cicd-wm4z/locations/us-central1/workerPools/private-pool-4ika"
cloud_build_worker_peered_ip_range = "192.168.0.0/24"
cloud_build_worker_range_id = "projects/prj-b-cicd-wm4z/global/addresses/ga-b-cbpools-worker-pool-range"
cloud_builder_artifact_repo = "projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners"
cloudbuild_project_id = "prj-b-cicd-wm4z"
common_config = {
  "billing_account" = "012EDD-5AD5ED-ECFF0B"
  "bootstrap_folder_name" = "folders/865611452734"
  "default_region" = "us-central1"
  "folder_prefix" = "fldr"
  "org_id" = "630259462753"
  "parent_folder" = "1078109772786"
  "parent_id" = "folders/1078109772786"
  "project_prefix" = "prj"
}
csr_repos = {
  "gcp-bootstrap" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-bootstrap"
    "name" = "gcp-bootstrap"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-bootstrap"
  }
  "gcp-environments" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-environments"
    "name" = "gcp-environments"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-environments"
  }
  "gcp-networks" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-networks"
    "name" = "gcp-networks"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-networks"
  }
  "gcp-org" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-org"
    "name" = "gcp-org"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-org"
  }
  "gcp-policies" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-policies"
    "name" = "gcp-policies"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-policies"
  }
  "gcp-projects" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-projects"
    "name" = "gcp-projects"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-projects"
  }
  "tf-cloudbuilder" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/tf-cloudbuilder"
    "name" = "tf-cloudbuilder"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/tf-cloudbuilder"
  }
}
environment_step_terraform_service_account_email = "sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com"
gcs_bucket_cloudbuild_artifacts = {
  "bootstrap" = "bkt-prj-b-cicd-wm4z-gcp-bootstrap-build-artifacts"
  "env" = "bkt-prj-b-cicd-wm4z-gcp-environments-build-artifacts"
  "net" = "bkt-prj-b-cicd-wm4z-gcp-networks-build-artifacts"
  "org" = "bkt-prj-b-cicd-wm4z-gcp-org-build-artifacts"
  "proj" = "bkt-prj-b-cicd-wm4z-gcp-projects-build-artifacts"
}
gcs_bucket_cloudbuild_logs = {
  "bootstrap" = "bkt-prj-b-cicd-wm4z-gcp-bootstrap-build-logs"
  "env" = "bkt-prj-b-cicd-wm4z-gcp-environments-build-logs"
  "net" = "bkt-prj-b-cicd-wm4z-gcp-networks-build-logs"
  "org" = "bkt-prj-b-cicd-wm4z-gcp-org-build-logs"
  "proj" = "bkt-prj-b-cicd-wm4z-gcp-projects-build-logs"
}
gcs_bucket_tfstate = "bkt-prj-b-seed-tfstate-cca4"
group_billing_admins = "gcp-billing-admins@obrienlabs.app"
group_org_admins = "gcp-organization-admins@obrienlabs.app"
networks_step_terraform_service_account_email = "sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com"
optional_groups = {}
organization_step_terraform_service_account_email = "sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com"
projects_gcs_bucket_tfstate = "bkt-prj-b-seed-31ca-gcp-projects-tfstate"
projects_step_terraform_service_account_email = "sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com"
required_groups = {}
seed_project_id = "prj-b-seed-31ca"

0-bootstrap is up 1000

1-environments dev branch only

Fortinet

SDN connector today for fortinet meet no config for vdoms after ha cluster deployed

enable-vdom - config fails right away use jump box - no default route needed - use standard subnet route (don't need vdom) add jump box to tf

can we download the config from fortinet help with the integration of their example

1-org

0-bootstrap inventory

cicd and seed projects

Cloud Source Repositories

all empty except for tf-cloudbuilder as expected

preparing for 1-org

michael@cloudshell:~/tef-olapp/github/terraform-example-foundation/0-bootstrap (tef-olapp)$ terraform output
bootstrap_step_terraform_service_account_email = "sa-terraform-bootstrap@prj-b-seed-31ca.iam.gserviceaccount.com"
cloud_build_peered_network_id = "projects/prj-b-cicd-wm4z/global/networks/vpc-b-cbpools"
cloud_build_private_worker_pool_id = "projects/prj-b-cicd-wm4z/locations/us-central1/workerPools/private-pool-4ika"
cloud_build_worker_peered_ip_range = "192.168.0.0/24"
cloud_build_worker_range_id = "projects/prj-b-cicd-wm4z/global/addresses/ga-b-cbpools-worker-pool-range"
cloud_builder_artifact_repo = "projects/prj-b-cicd-wm4z/locations/us-central1/repositories/tf-runners"
cloudbuild_project_id = "prj-b-cicd-wm4z"
common_config = {
  "billing_account" = "012...B"
  "bootstrap_folder_name" = "folders/865611452734"
  "default_region" = "us-central1"
  "folder_prefix" = "fldr"
  "org_id" = "630259462753"
  "parent_folder" = "1078109772786"
  "parent_id" = "folders/1078109772786"
  "project_prefix" = "prj"
}
csr_repos = {
  "gcp-bootstrap" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-bootstrap"
    "name" = "gcp-bootstrap"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-bootstrap"
  }
  "gcp-environments" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-environments"
    "name" = "gcp-environments"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-environments"
  }
  "gcp-networks" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-networks"
    "name" = "gcp-networks"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-networks"
  }
  "gcp-org" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-org"
    "name" = "gcp-org"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-org"
  }
  "gcp-policies" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-policies"
    "name" = "gcp-policies"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-policies"
  }
  "gcp-projects" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/gcp-projects"
    "name" = "gcp-projects"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/gcp-projects"
  }
  "tf-cloudbuilder" = {
    "id" = "projects/prj-b-cicd-wm4z/repos/tf-cloudbuilder"
    "name" = "tf-cloudbuilder"
    "project" = "prj-b-cicd-wm4z"
    "url" = "https://source.developers.google.com/p/prj-b-cicd-wm4z/r/tf-cloudbuilder"
  }
}
environment_step_terraform_service_account_email = "sa-terraform-env@prj-b-seed-31ca.iam.gserviceaccount.com"
gcs_bucket_cloudbuild_artifacts = {
  "bootstrap" = "bkt-prj-b-cicd-wm4z-gcp-bootstrap-build-artifacts"
  "env" = "bkt-prj-b-cicd-wm4z-gcp-environments-build-artifacts"
  "net" = "bkt-prj-b-cicd-wm4z-gcp-networks-build-artifacts"
  "org" = "bkt-prj-b-cicd-wm4z-gcp-org-build-artifacts"
  "proj" = "bkt-prj-b-cicd-wm4z-gcp-projects-build-artifacts"
}
gcs_bucket_cloudbuild_logs = {
  "bootstrap" = "bkt-prj-b-cicd-wm4z-gcp-bootstrap-build-logs"
  "env" = "bkt-prj-b-cicd-wm4z-gcp-environments-build-logs"
  "net" = "bkt-prj-b-cicd-wm4z-gcp-networks-build-logs"
  "org" = "bkt-prj-b-cicd-wm4z-gcp-org-build-logs"
  "proj" = "bkt-prj-b-cicd-wm4z-gcp-projects-build-logs"
}
gcs_bucket_tfstate = "bkt-prj-b-seed-tfstate-cca4"
group_billing_admins = "gcp-billing-admins@obrienlabs.app"
group_org_admins = "gcp-organization-admins@obrienlabs.app"
networks_step_terraform_service_account_email = "sa-terraform-net@prj-b-seed-31ca.iam.gserviceaccount.com"
optional_groups = {}
organization_step_terraform_service_account_email = "sa-terraform-org@prj-b-seed-31ca.iam.gserviceaccount.com"
projects_gcs_bucket_tfstate = "bkt-prj-b-seed-31ca-gcp-projects-tfstate"
projects_step_terraform_service_account_email = "sa-terraform-proj@prj-b-seed-31ca.iam.gserviceaccount.com"
required_groups = {}
seed_project_id = "prj-b-seed-31ca"

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days

GoogleCloudPlatform / pbmm-on-gcp-onboarding

Introduce 2024 light - a secondary Landing Zone to our KCC LZ based on a minimal IaaS NGFW based landing zone based on TEF V4 #345

Structure

Architecture

Base Landing Zone

Merged with Fortigate LB sandwich cluster - re-peer with above

Work Items

Iterations

change

Prep

0-bootstrap

copy tfvars

validating

roles

Terraform version 1.7.2

0 - bootstrap - terraform init

Terraform plan - 260

Terraform vet - need a local shell

Terraform apply

terraform apply

terraform apply

terraform apply

Terraform apply 0-bootstrap

Fortinet

1-org

0-bootstrap inventory

cicd and seed projects

Cloud Source Repositories