search

GoogleCloudPlatform / pbmm-on-gcp-onboarding

GCP Canadian Public Sector Landing Zone overlay on top of the TEF via CFT modules - a secure cloud foundation
https://cloud.google.com/architecture/security-foundations
Apache License 2.0
41 stars 55 forks source link

Day 0: Terraform Landing Zone install - procedure verify and re-documentation #300

Closed obriensystems closed 4 months ago

obriensystems commented 11 months ago

Full day 0 clean install on a clean org

Step: get legacy 1.0.10 version of terraform to run local boostrap only

michael@cloudshell:~/lz-oe/deployed$ wget https://releases.hashicorp.com/terraform/1.0.10/terraform_1.0.10_linux_amd64.zip
--2023-09-11 15:12:11--  https://releases.hashicorp.com/terraform/1.0.10/terraform_1.0.10_linux_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 18.64.174.36, 18.64.174.78, 18.64.174.51, ...
Connecting to releases.hashicorp.com (releases.hashicorp.com)|18.64.174.36|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32674953 (31M) [application/zip]
Saving to: ‘terraform_1.0.10_linux_amd64.zip’

terraform_1.0.10_linux_amd64.zip                            100%[========================================================================================================================================>]  31.16M  98.5MB/s    in 0.3s    

2023-09-11 15:12:11 (98.5 MB/s) - ‘terraform_1.0.10_linux_amd64.zip’ saved [32674953/32674953]

michael@cloudshell:~/lz-oe/deployed$ chmod 777 terraform_1.0.10_linux_amd64.zip 
michael@cloudshell:~/lz-oe/deployed$ unzip terraform_1.0.10_linux_amd64.zip 
Archive:  terraform_1.0.10_linux_amd64.zip
  inflating: terraform               
michael@cloudshell:~/lz-oe/deployed$ ./terraform --version
Terraform v1.0.10
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.5.7. You can update by downloading from https://www.terraform.io/downloads.html
michael@cloudshell:~/lz-oe/deployed$ terraform --version
Terraform v1.5.5
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.5.7. You can update by downloading from https://www.terraform.io/downloads.html

rerun - retrofit readme script

VER=1.0.10 wget https://releases.hashicorp.com/terraform/${VER}/terraform_${VER}_linux_amd64.zip

IAM start state

Screenshot 2023-09-11 at 11 37 57

Get IAM existing roles


michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ export USER=mi...ael@obr...eering
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ export PROJECT_ID=lzone-oe
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ export ORG_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)

michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ gcloud organizations get-iam-policy $ORG_ID --filter="bindings.members:$USER" --flatten="bindings[].members" --format="table(bindings.role)"
ROLE: roles/owner

ROLE: roles/resourcemanager.folderCreator

ROLE: roles/resourcemanager.organizationAdmin
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding (lzone-oe)$ ./writeids.sh -c fill -f 557106011343
Derived organization_id: 25..715
Derived billing_id: 01A7...AE
replacing IDs: billing: 01A7E...AAE organization: 25..15 folder: 557106011343 from placeholders
environments/bootstrap/bootstrap.auto.tfvars pass - fill:true
environments/bootstrap/organization-config.auto.tfvars pass - fill:true
environments/common/common.auto.tfvars pass - fill:true
environments/nonprod/nonp-network.auto.tfvars pass - fill:true
environments/common/perimeter-network.auto.tfvars pass - fill:true
environments/prod/prod-network.auto.tfvars pass - fill:true
environments/prod/prod-workload-network.auto.tfvars pass - fill:true

ichael@cloudshell:~/lzone-oe/lz-oe (lzone-oe)$ sudo cp ./terraform /usr/bin
michael@cloudshell:~/lzone-oe/lz-oe (lzone-oe)$ sudo chmod +x /usr/bin/terraform
michael@cloudshell:~/lzone-oe/lz-oe (lzone-oe)$ terraform --version
Terraform v1.0.10
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.5.7. You can update by downloading from https://www.terraform.io/downloads.html
michael@cloudshell:~/lzone-oe/lz-oe (lzone-oe)$ cd 
deployed/  _upsource/ 
michael@cloudshell:~/lzone-oe/lz-oe (lzone-oe)$ cd deployed/pbmm-on-gcp-onboarding/environments/bootstrap/
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ ./bootstrap.sh run
User: michael@obrienlabs.engineering
Domain: obrienlabs.engineering
Is this is user and domain of the organization you want to deploy to? (y/n) 
>  y
Existing roles for michael@obrienlabs.engineering at the organizations/259997541715 level for domain obrienlabs.engineering
Existing roles for michael@obrienlabs.engineering at the organizations/259997541715 level for domain obrienlabs.engineering
ROLE: roles/billing.projectManager

ROLE: roles/billing.viewer

ROLE: roles/identityplatform.admin

ROLE: roles/orgpolicy.policyAdmin

ROLE: roles/owner

ROLE: roles/resourcemanager.folderCreator

ROLE: roles/resourcemanager.organizationAdmin

ROLE: roles/resourcemanager.organizationViewer

ROLE: roles/resourcemanager.projectCreator
INFO - Applying roles to Organization Node
Updated IAM policy for organization [259997541715].
Updated IAM policy for organization [259997541715].
Updated IAM policy for organization [259997541715].
Updated IAM policy for organization [259997541715].
Updated IAM policy for organization [259997541715].
Updated IAM policy for organization [259997541715].
Updated IAM policy for organization [259997541715].
Roles set:
ROLE: roles/billing.projectManager

ROLE: roles/billing.viewer

ROLE: roles/identityplatform.admin

ROLE: roles/orgpolicy.policyAdmin

ROLE: roles/owner

ROLE: roles/resourcemanager.folderCreator

ROLE: roles/resourcemanager.organizationAdmin

ROLE: roles/resourcemanager.organizationViewer

ROLE: roles/resourcemanager.projectCreator
INFO - Running a plan to ensure the configuration file is correct
Initializing modules...
- landing_zone_bootstrap.project in ../../modules/project
- landing_zone_bootstrap.project.project_name in ../../modules/naming-standard/modules/gcp/project
- landing_zone_bootstrap.project.project_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
- landing_zone_bootstrap.project.project_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
- landing_zone_bootstrap.project_name in ../../modules/naming-standard/modules/gcp/project
- landing_zone_bootstrap.project_name.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
- landing_zone_bootstrap.project_name.name_generation in ../../modules/naming-standard/modules/common/name_generator
- landing_zone_bootstrap.state_bucket_names in ../../modules/naming-standard/modules/gcp/storage
- landing_zone_bootstrap.state_bucket_names.common_prefix in ../../modules/naming-standard/modules/common/gc_prefix
- landing_zone_bootstrap.state_bucket_names.name_generation in ../../modules/naming-standard/modules/common/name_generator

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/google versions matching ">= 3.50.0"...
- Finding latest version of hashicorp/null...
- Finding hashicorp/google-beta versions matching ">= 3.50.0"...
- Installing hashicorp/google v4.82.0...
- Installed hashicorp/google v4.82.0 (signed by HashiCorp)
- Installing hashicorp/null v3.2.1...
- Installed hashicorp/null v3.2.1 (signed by HashiCorp)
- Installing hashicorp/google-beta v4.82.0...
- Installed hashicorp/google-beta v4.82.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│ 
│   on terraform.tf line 20, in terraform:
│   20:   experiments = [module_variable_optional_attrs]
│ 
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│ 
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│ 
│ (and 2 more similar warnings elsewhere)
╵

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│ 
│   on terraform.tf line 20, in terraform:
│   20:   experiments = [module_variable_optional_attrs]
│ 
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│ 
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│ 
│ (and 2 more similar warnings elsewhere)
╵
╷
│ Error: Invalid value for variable
│ 
│   on ../../modules/landing-zone-bootstrap/naming.tf line 36, in module "state_bucket_names":
│   36:   user_defined_string = lower(each.value.name)
│ 
│ Must be lower-case letters or numbers with a maximum length of 59 characters. Min 3 characters.
│ 
│ This was checked by the validation rule at ../../modules/naming-standard/modules/gcp/storage/variables.tf:24,3-13.
╵
╷
│ Error: Invalid value for variable
│ 
│   on ../../modules/landing-zone-bootstrap/naming.tf line 36, in module "state_bucket_names":
│   36:   user_defined_string = lower(each.value.name)
│ 
│ Must be lower-case letters or numbers with a maximum length of 59 characters. Min 3 characters.
│ 
│ This was checked by the validation rule at ../../modules/naming-standard/modules/gcp/storage/variables.tf:24,3-13.
╵
╷
│ Error: Invalid value for variable
│ 
│   on ../../modules/landing-zone-bootstrap/naming.tf line 36, in module "state_bucket_names":
│   36:   user_defined_string = lower(each.value.name)
│ 
│ Must be lower-case letters or numbers with a maximum length of 59 characters. Min 3 characters.
│ 
│ This was checked by the validation rule at ../../modules/naming-standard/modules/gcp/storage/variables.tf:24,3-13.

fixed bucket names in common

│ 
│   on terraform.tf line 20, in terraform:
│   20:   experiments = [module_variable_optional_attrs]
│ 
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│ 
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
│ 
│ (and 2 more similar warnings elsewhere)
╵

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: launchpad.2023-09-12.0009.plan

To perform exactly these actions, run the following command to apply:
    terraform apply "launchpad.2023-09-12.0009.plan"
Please confirm that you have reviewed the plan and wish to apply it. Type 'yes' to proceed

2021

fix BillingAccountAdministrator

│ Error: failed pre-requisites: failed to check permissions on billing account "billingAccounts/01A7....2AAE": googleapi: Error 403: Cloud Billing API has not been used in project lzone-oe before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=lzone-oe then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

╷
│ Error: failed pre-requisites: failed to check permissions on billing account "billingAccounts/01A7ED-3C095A-802AAE": googleapi: Error 403: Cloud Billing API has not been used in project lzone-oe before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=lzone-oe then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {

add 2 project permissions missing

# enable billing on current project
gcloud services enable cloudbilling.googleapis.com
# enable 
gcloud services enable cloudresourcemanager.googleapis.com

INFO - Applying Terraform plan
module.landing_zone_bootstrap.module.project.google_project.project: Creating...

2051

Screenshot 2023-09-11 at 20 54 34

2056


╷
│ Error: Error creating service account: googleapi: Error 403: Identity and Access Management (IAM) API has not been used in project lzone-oe before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=lzone-oe then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.Help",
│     "links": [
│       {
│         "description": "Google developers console API activation",
│         "url": "https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=lzone-oe"
│       }
│     ]
│   },
│   {
│     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
│     "domain": "googleapis.com",
│     "metadata": {
│       "consumer": "projects/lzone-oe",
│       "service": "iam.googleapis.com"
│     },
│     "reason": "SERVICE_DISABLED"
│   }
│ ]
│ , accessNotConfigured
│ 
│   with module.landing_zone_bootstrap.google_service_account.org_terraform,
│   on ../../modules/landing-zone-bootstrap/main.tf line 33, in resource "google_service_account" "org_terraform":
│   33: resource "google_service_account" "org_terraform" {
│

add gcloud services enable iam.googleapis.com

and the TF SA is already in billing

Screenshot 2023-09-11 at 21 00 08 
rerun bootstrap in place
module.landing_zone_bootstrap.google_organization_iam_member.tf_sa_org_perms["roles/compute.networkAdmin"]: Still creating... [20s elapsed]
module.landing_zone_bootstrap.google_organization_iam_member.tf_sa_org_perms["roles/compute.admin"]: Still creating... [20s elapsed]
module.landing_zone_bootstrap.google_organization_iam_member.tf_sa_org_perms["roles/serviceusage.serviceUsageAdmin"]: Still creating... [20s elapsed]

cloud build - weird we did not get this before - as the CB job is in another project

│ Error: Error creating Trigger: googleapi: Error 403: Cloud Build API has not been used in project lzone-oe before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=lzone-oe then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

reran after
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ gcloud services list | grep cloudbuild
NAME: cloudbuild.googleapis.com

failure - will wait for propagation
retry...working

INFO - Uploading ./default.tfstate to dcpecommongcs/environments/bootstrap/default.tfstate dcpe-oe-ole
Copying file://./default.tfstate [Content-Type=application/octet-stream]...
/ [1 files][206.2 KiB/206.2 KiB]                                                
Operation completed over 1 objects/206.2 KiB.                                    
Terraform default.tfstate exists.
INFO - Create bootstrap backend
INFO - Create common backend
INFO - Create bootstrap provider
INFO - Create common provider
INFO - Create nonprod backend and provider
INFO - Create prod backend and provider
INFO - Committing code to CSR
Specify your git config email

 create mode 100644 modules/vpc-service-controls/variables.tf
 create mode 100755 writeids.sh
INFO - Check if CSR is already a git remote
INFO - CSR is not a remote, adding it
INFO - Pushing code to CSR
Screenshot 2023-09-11 at 21 08 16 Screenshot 2023-09-11 at 21 09 07
obriensystems commented 11 months ago

fixing across 3 triggers

common
Step #3 - "tf plan": │ Error: Error reading organization: googleapi: Error 400: Request contains an invalid argument., badRequest
Step #3 - "tf plan": │ 
Step #3 - "tf plan": │   with module.group_telcoadmin.data.google_organization.org[0],
Step #3 - "tf plan": │   on .terraform/modules/group_telcoadmin/main.tf line 17, in data "google_organization" "org":
Step #3 - "tf plan": │   17: data "google_organization" "org" {

fix

organization_iam_group_secadmin = [
  {
    member       = "group:secadmin@DOMAIN_NAME" # REQUIRED EDIT. group:user@google.com
    organization = "REPLACE_ORGANIZATION_ID" #Insert your Ord ID here, format ############

code
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ USER=$(gcloud config list --format json|jq .core.account | sed 's/"//g')
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ echo $USER
michael@obrienlabs.engineering
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ DOMAIN_NAME=$(echo $USER | sed 's/"//g' | cut -f2 -d@)
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ echo $DOMAIN_NAME
obrienlabs.engineering
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ DOMAIN_NAME_SEARCH=DOMAIN_NAME
michael@cloudshell:~/lzone-oe/lz-oe/deployed/pbmm-on-gcp-onboarding/environments/bootstrap (lzone-oe)$ sed -i "s/${DOMAIN_NAME_SEARCH}/${DOMAIN_NAME}/g" ../common/iam-groups.auto.tfvars 

member       = "group:secadmin@obrienlabs.engineering" # REQUIRED EDIT. group:user@google.com

expected billing quota - asking for more
https://support.google.com/code/contact/billing_quota_increase

Step #4 - "tf apply": │ Error: Error setting billing account "01A7ED-3C095A-802AAE" for project "projects/dcpe-oe-guardrailsoe": googleapi: Error 400: Precondition check failed.
Step #4 - "tf apply": │ Details:
Step #4 - "tf apply": │ [
Step #4 - "tf apply": │   {
Step #4 - "tf apply": │     "@type": "type.googleapis.com/google.rpc.QuotaFailure",
Step #4 - "tf apply": │     "violations": [
Step #4 - "tf apply": │       {
Step #4 - "tf apply": │         "description": "Cloud billing quota exceeded: https://support.google.com/code/contact/billing_quota_increase",
Step #4 - "tf apply": │         "subject": "billingAccounts/01A7ED-3C095A-802AAE"

this is expected

Error: Error creating Group: googleapi: Error 403: Error(2015): Permission denied for group resource 'opsadmin@obrienlabs.engineering'.
Step #4 - "tf apply": │ Details:
Step #4 - "tf apply": │ [
Step #4 - "tf apply": │   {
Step #4 - "tf apply": │     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
Step #4 - "tf apply": │     "description": "Error(2015): Permission denied for group resource 'opsadmin@obrienlabs.engineering'.",
Step #4 - "tf apply": │     "owner": "domain:cloudidentity.googleapis.com",
Step #4 - "tf apply": │     "resourceType": "cloudidentity.googleapis.com/Group"
Step #4 - "tf apply": │   }
Step #4 - "tf apply": │ ]

add group creation permission for the terraform service account in see https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/271#issuecomment-1661425718 and https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/308

nonprod
Step #2 - "tf validate": │ Error: Reference to undeclared input variable
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │   on dns.tf line 44, in module "private_zone":
Step #2 - "tf validate": │   44:   name        = var.private_zone_name
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │ An input variable with the name "private_zone_name" has not been declared.
Step #2 - "tf validate": │ This variable can be declared with a variable "private_zone_name" {} block.
Step #2 - "tf validate": ╵
Step #2 - "tf validate": ╷
Step #2 - "tf validate": │ Error: Reference to undeclared input variable
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │   on dns.tf line 45, in module "private_zone":
Step #2 - "tf validate": │   45:   domain      = var.private_zone_domain
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │ An input variable with the name "private_zone_domain" has not been
Step #2 - "tf validate": │ declared. This variable can be declared with a variable
Step #2 - "tf validate": │ "private_zone_domain" {} block.
Step #2 - "tf validate": ╵
Step #2 - "tf validate": ╷
Step #2 - "tf validate": │ Error: Reference to undeclared input variable
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │   on dns.tf line 46, in module "private_zone":
Step #2 - "tf validate": │   46:   labels      = var.labels
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │ An input variable with the name "labels" has not been declared. This
Step #2 - "tf validate": │ variable can be declared with a variable "labels" {} block.
Step #2 - "tf validate": ╵
Step #2 - "tf validate": ╷
Step #2 - "tf validate": │ Error: Reference to undeclared input variable
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │   on dns.tf line 51, in module "private_zone":
Step #2 - "tf validate": │   51:   private_visibility_config_networks = var.network_self_links
Step #2 - "tf validate": │ 
Step #2 - "tf validate": │ An input variable with the name "network_self_links" has not been declared.
Step #2 - "tf validate": │ This variable can be declared with a variable "network_self_links" {}

prod

│ Error: Unsupported attribute
│ 
│   on main.tf line 76, in module "net-host-prj":
│   76:   parent                         = data.terraform_remote_state.common.outputs.folders_map_2_levels.ProdNetworking.id
│     ├────────────────
│     │ data.terraform_remote_state.common.outputs is object with no attributes
│ 
│ This object does not have an attribute named "folders_map_2_levels".
obriensystems commented 11 months ago

see spawned issues https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/305 https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/306 https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/issues/307

For billing - add on both IAM and billing side - BAA - or billing association is not set

Screenshot 2023-09-11 at 21 49 23

rename

DcPe-oe-auditoe projects/dcpe-oe-auditoe  
  DcPe-oe-guardrailsoe projects/dcpe-oe-guardrailsoe  
  DcPe-oe-prdoe-perim projects/dcpe-oe-prdoe-perim
fmichaelobrien commented 4 months ago

20240406: Closing issue during retrofit/rebase of this TEF V1 based/modified repo to TEF V4 standards This issue may participate in the LZ refactor after rebase Query on all issues related to the older V1 version via the tag https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/labels/2024-pre-tef-v4