search

GoogleCloudPlatform / pbmm-on-gcp-onboarding

GCP Canadian Public Sector Landing Zone overlay on top of the TEF via CFT modules - a secure cloud foundation
https://cloud.google.com/architecture/security-foundations
Apache License 2.0
45 stars 56 forks source link

Upstream: full group creation hangs on eventual consistency - wait 5 min to restart 0-bootstrap terraform plan/apply #443

Closed obriensystems closed 3 months ago

obriensystems commented 6 months ago

add to the TEF

michael@cloudshell:~/tef-oldev4 (clouddeploy-ol)$ mkdir _431_ssh_testing
michael@cloudshell:~/tef-oldev4 (clouddeploy-ol)$ eval `ssh-agent`
Agent pid 1357
michael@cloudshell:~/tef-oldev4 (clouddeploy-ol)$ ls ~/.ssh
config      csr_id_rsa.pub         google_compute_engine.pub   id_rsa      known_hosts
csr_id_rsa  google_compute_engine  google_compute_known_hosts  id_rsa.pub  obrienlabs_org_github
michael@cloudshell:~/tef-oldev4 (clouddeploy-ol)$ gcloud config set project tef-oldev4
Updated property [core/project].
michael@cloudshell:~/tef-oldev4 (tef-oldev4)$

branch https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/tree/gh431-csr-ssh test ssh-add

ichael@cloudshell:~/tef-oldev4 (tef-oldev4)$ git clone git@github.com:GoogleCloudPlatform/pbmm-on-gcp-onboarding.git
Cloning into 'pbmm-on-gcp-onboarding'...
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
michael@cloudshell:~/tef-oldev4 (tef-oldev4)$ ssh-add ~/.ssh/obrienlabs_org_github 
Identity added: /home/michael/.ssh/obrienlabs_org_github (michael@obrienlabs.org)
michael@cloudshell:~/tef-oldev4 (tef-oldev4)$ git clone git@github.com:GoogleCloudPlatform/pbmm-on-gcp-onboarding.git
Cloning into 'pbmm-on-gcp-onboarding'...
remote: Enumerating objects: 6783, done.
remote: Counting objects: 100% (2527/2527), done.
remote: Compressing objects: 100% (892/892), done.
remote: Total 6783 (delta 1757), reused 2181 (delta 1598), pack-reused 4256
Receiving objects: 100% (6783/6783), 31.86 MiB | 28.05 MiB/s, done.
Resolving deltas: 100% (4179/4179), done.
michael@cloudshell:~/tef-oldev4 (tef-oldev4)$

edit tfvars

org_id = "58...8" # format "000000000000"

billing_account = "0...76" # format "000000-000000-000000"

// For enabling the automatic groups creation, uncoment the
// variables and update the values with the group names
groups = {
  create_required_groups = true # Change to true to create the required_groups
  create_optional_groups = true # Change to true to create the optional_groups
  billing_project        = "te..v4"  # Fill with bootstrap project id (the one you are starting with) to create required or optional groups
  required_groups = {
    group_org_admins           = "gcp-organization-admins4@obrienlabs.dev" # example "gcp-organization-admins@example.com"
    group_billing_admins       = "gcp-billing-admins4@obrienlabs.dev" # example "gcp-billing-admins@example.com"
    billing_data_users         = "gcp-billing-data4@obrienlabs.dev" # example "gcp-billing-data@example.com"
    audit_data_users           = "gcp-audit-data4@obrienlabs.dev" # example "gcp-audit-data@example.com"
    monitoring_workspace_users = "gcp-monitoring-workspace4@obrienlabs.dev" # example "gcp-monitoring-workspace@example.com"
  }
  optional_groups = {
     gcp_security_reviewer      = "gcp_security_reviewer4@obrienlabs.dev" #"gcp_security_reviewer_local_test@example.com"
     gcp_network_viewer         = "gcp_network_viewer4@obrienlabs.dev" #"gcp_network_viewer_local_test@example.com"
     gcp_scc_admin              = "gcp_scc_admin4@obrienlabs.dev" #"gcp_scc_admin_local_test@example.com"
     gcp_global_secrets_admin   = "gcp_global_secrets_admin4@obrienlabs.dev" #"gcp_global_secrets_admin_local_test@example.com"
     gcp_kms_admin              = "gcp_kms_admin4@obrienlabs.dev" #"gcp_kms_admin_local_test@example.com"
   }
}

default_region = "northamerica-northeast1"
#default_region = "northamerica-northeast2"

# Optional - for an organization with existing projects or for development/validation.
# Uncomment this variable to place all the example foundation resources under
# the provided folder instead of the root organization.
# The variable value is the numeric folder ID
# The folder must already exist.
parent_folder = "4..37"

downgrade terraform

michael@cloudshell:~/tef-oldev4 (tef-oldev4)$ cp terraform /usr/bin/terraform 
cp: cannot create regular file '/usr/bin/terraform': Permission denied
michael@cloudshell:~/tef-oldev4 (tef-oldev4)$ sudo cp terraform /usr/bin/terraform 
michael@cloudshell:~/tef-oldev4 (tef-oldev4)$ cd pbmm-on-gcp-onboarding/0-bootstrap/
michael@cloudshell:~/tef-oldev4/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev4)$ terraform version
Terraform v1.3.10
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.8.3. You can update by downloading from https://www.terraform.io/downloads.html

terraform init and plan

Plan: 271 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + bootstrap_step_terraform_service_account_email    = (known after apply)
  + cloud_build_peered_network_id                     = (known after apply)
  + cloud_build_private_worker_pool_id                = (known after apply)
  + cloud_build_worker_peered_ip_range                = "192.168.0.0/24"
  + cloud_build_worker_range_id                       = (known after apply)
  + cloud_builder_artifact_repo                       = (known after apply)
  + cloudbuild_project_id                             = (known after apply)
  + common_config                                     = {
      + billing_account       = "019283-6F1AB5-7AD576"
      + bootstrap_folder_name = (known after apply)
      + default_region        = "northamerica-northeast1"
      + folder_prefix         = "fldr"
      + org_id                = "583675367868"
      + parent_folder         = "479872525237"
      + parent_id             = "folders/479872525237"
      + project_prefix        = "prj"
    }
  + csr_repos                                         = {
      + gcp-bootstrap    = {
          + id      = (known after apply)
          + name    = "gcp-bootstrap"
          + project = (known after apply)
          + url     = (known after apply)
        }
      + gcp-environments = {
          + id      = (known after apply)
          + name    = "gcp-environments"
          + project = (known after apply)
          + url     = (known after apply)
        }
      + gcp-networks     = {
          + id      = (known after apply)
          + name    = "gcp-networks"
          + project = (known after apply)
          + url     = (known after apply)
        }
      + gcp-org          = {
          + id      = (known after apply)
          + name    = "gcp-org"
          + project = (known after apply)
          + url     = (known after apply)
        }
      + gcp-policies     = {
          + id      = (known after apply)
          + name    = "gcp-policies"
          + project = (known after apply)
          + url     = (known after apply)
        }
      + gcp-projects     = {
          + id      = (known after apply)
          + name    = "gcp-projects"
          + project = (known after apply)
          + url     = (known after apply)
        }
      + tf-cloudbuilder  = {
          + id      = (known after apply)
          + name    = "tf-cloudbuilder"
          + project = (known after apply)
          + url     = (known after apply)
        }
    }
  + environment_step_terraform_service_account_email  = (known after apply)
  + gcs_bucket_cloudbuild_artifacts                   = {
      + bootstrap = (known after apply)
      + env       = (known after apply)
      + net       = (known after apply)
      + org       = (known after apply)
      + proj      = (known after apply)
    }
  + gcs_bucket_cloudbuild_logs                        = {
      + bootstrap = (known after apply)
      + env       = (known after apply)
      + net       = (known after apply)
      + org       = (known after apply)
      + proj      = (known after apply)
    }
  + gcs_bucket_tfstate                                = (known after apply)
  + networks_step_terraform_service_account_email     = (known after apply)
  + optional_groups                                   = {
      + "gcp_global_secrets_admin" = "gcp_global_secrets_admin4@obrienlabs.dev"
      + "gcp_kms_admin"            = "gcp_kms_admin4@obrienlabs.dev"
      + "gcp_network_viewer"       = "gcp_network_viewer4@obrienlabs.dev"
      + "gcp_scc_admin"            = "gcp_scc_admin4@obrienlabs.dev"
      + "gcp_security_reviewer"    = "gcp_security_reviewer4@obrienlabs.dev"
    }
  + organization_step_terraform_service_account_email = (known after apply)
  + projects_gcs_bucket_tfstate                       = (known after apply)
  + projects_step_terraform_service_account_email     = (known after apply)
  + required_groups                                   = {
      + "audit_data_users"           = "gcp-audit-data4@obrienlabs.dev"
      + "billing_data_users"         = "gcp-billing-data4@obrienlabs.dev"
      + "group_billing_admins"       = "gcp-billing-admins4@obrienlabs.dev"
      + "group_org_admins"           = "gcp-organization-admins4@obrienlabs.dev"
      + "monitoring_workspace_users" = "gcp-monitoring-workspace4@obrienlabs.dev"
    }
  + seed_project_id                                   = (known after apply)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: bootstrap.tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "bootstrap.tfplan"

1036

expected eventually consistent error on group creation after 1 min

michael@cloudshell:~/tef-oldev4/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev4)$ terraform apply bootstrap.tfplan
module.bootstrap_csr_repo.null_resource.run_destroy_command[0]: Creating...
module.bootstrap_csr_repo.null_resource.run_destroy_command[0]: Creation complete after 0s [id=4012333594955662454]
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Creating...
module.tf_private_pool.random_string.suffix: Creating...
module.seed_bootstrap.random_id.suffix: Creating...
random_string.suffix: Creating...
module.tf_private_pool.random_string.suffix: Creation complete after 0s [id=gdo0]
module.seed_bootstrap.module.seed_project.module.project-factory.random_id.random_project_id_suffix: Creation complete after 0s [id=Sa8]
random_string.suffix: Creation complete after 0s [id=pdn7]
module.seed_bootstrap.random_id.suffix: Creation complete after 0s [id=leg]
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creating...
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creating...
module.required_group["audit_data_users"].google_cloud_identity_group.group: Creating...
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Creating...
google_folder.bootstrap: Creating...
module.optional_group["gcp_scc_admin"].google_cloud_identity_group.group: Creating...
module.required_group["group_billing_admins"].google_cloud_identity_group.group: Creating...
module.required_group["billing_data_users"].google_cloud_identity_group.group: Creating...
module.required_group["group_org_admins"].google_cloud_identity_group.group: Creating...
module.required_group["monitoring_workspace_users"].google_cloud_identity_group.group: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_service_account_user[0]: Creation complete after 6s [id=folders/479872525237/roles/iam.serviceAccountUser/group:gcp-organization-admins4@obrienlabs.dev]
module.optional_group["gcp_security_reviewer"].google_cloud_identity_group.group: Creating...
module.required_group["audit_data_users"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/0111kx3o0i0vokv]
module.optional_group["gcp_network_viewer"].google_cloud_identity_group.group: Creating...
module.optional_group["gcp_scc_admin"].google_cloud_identity_group.group: Creation complete after 10s [id=groups/01pxezwc2yon24n]
module.optional_group["gcp_global_secrets_admin"].google_cloud_identity_group.group: Creating...
module.required_group["group_billing_admins"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/02koq6561p3tqsw]
module.optional_group["gcp_kms_admin"].google_cloud_identity_group.group: Creating...
module.required_group["billing_data_users"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/017dp8vu47j024a]
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Still creating... [10s elapsed]
google_folder.bootstrap: Still creating... [10s elapsed]
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Still creating... [10s elapsed]
module.required_group["monitoring_workspace_users"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/04bvk7pj2i03lqt]
module.seed_bootstrap.google_organization_iam_binding.billing_creator: Creation complete after 11s [id=583675367868/roles/billing.creator]
module.required_group["group_org_admins"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/04bvk7pj0j31gy7]
module.seed_bootstrap.google_organization_iam_member.org_billing_admin: Creation complete after 11s [id=583675367868/roles/billing.admin/group:gcp-billing-admins4@obrienlabs.dev]
google_folder.bootstrap: Creation complete after 12s [id=folders/976224166955]
module.optional_group["gcp_security_reviewer"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/03fwokq00h2vpb9]
module.optional_group["gcp_network_viewer"].google_cloud_identity_group.group: Creation complete after 9s [id=groups/02p2csry3d9cins]
module.optional_group["gcp_kms_admin"].google_cloud_identity_group.group: Creation complete after 8s [id=groups/00sqyw640obco6o]
module.optional_group["gcp_global_secrets_admin"].google_cloud_identity_group.group: Creation complete after 8s [id=groups/01ci93xb3cor196]
╷
│ Error: Error applying IAM policy for folder "folders/479872525237": Error setting IAM policy for folder "folders/479872525237": googleapi: Error 400: Group gcp-organization-admins4@obrienlabs.dev does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 47, in resource "google_folder_iam_member" "tmp_project_creator":
│   47: resource "google_folder_iam_member" "tmp_project_creator" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "583675367868": Error setting IAM policy for organization "583675367868": googleapi: Error 400: Group gcp-organization-admins4@obrienlabs.dev does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"],
│   on .terraform/modules/seed_bootstrap/main.tf line 184, in resource "google_organization_iam_member" "org_admins_group":
│  184: resource "google_organization_iam_member" "org_admins_group" {
│ 
╵
╷
│ Error: Error applying IAM policy for organization "583675367868": Error setting IAM policy for organization "583675367868": googleapi: Error 400: Group gcp-organization-admins4@obrienlabs.dev does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"],
│   on .terraform/modules/seed_bootstrap/main.tf line 184, in resource "google_organization_iam_member" "org_admins_group":
│  184: resource "google_organization_iam_member" "org_admins_group" {
│ 
╵
╷
│ Error: Error applying IAM policy for folder "folders/479872525237": Error setting IAM policy for folder "folders/479872525237": googleapi: Error 400: Group gcp-organization-admins4@obrienlabs.dev does not exist., badRequest
│ 
│   with module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0],
│   on .terraform/modules/seed_bootstrap/main.tf line 267, in resource "google_folder_iam_member" "org_admin_serviceusage_consumer":
│  267: resource "google_folder_iam_member" "org_admin_serviceusage_consumer" {
│ 
╵

groups are there

Screenshot 2024-05-23 at 10 41 04

restarting 1042


Plan: 252 to add, 0 to change, 0 to destroy.

michael@cloudshell:~/tef-oldev4/pbmm-on-gcp-onboarding/0-bootstrap (tef-oldev4)$ terraform apply bootstrap.tfplan
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creating...
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creating...
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Creating...
module.seed_bootstrap.google_folder_iam_member.tmp_project_creator[0]: Creation complete after 4s [id=folders/479872525237/roles/resourcemanager.projectCreator/group:gcp-organization-admins4@obrienlabs.dev]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Creating...
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/billing.user"]: Creation complete after 5s [id=583675367868/roles/billing.user/group:gcp-organization-admins4@obrienlabs.dev]
module.seed_bootstrap.google_organization_iam_member.org_admins_group["roles/resourcemanager.organizationAdmin"]: Creation complete after 9s [id=583675367868/roles/resourcemanager.organizationAdmin/group:gcp-organization-admins4@obrienlabs.dev]
module.seed_bootstrap.google_folder_iam_member.org_admin_serviceusage_consumer[0]: Creation complete after 9s [id=folders/479872525237/roles/serviceusage.serviceUsageConsumer/group:gcp-organization-admins4@obrienlabs.dev]
module.seed_bootstrap.module.seed_project.module.project-factory.google_project.main: Still creating... [10s elapsed]

good
Screenshot 2024-05-23 at 10 50 50
github-actions[bot] commented 3 months ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days