search

terraform-google-modules / terraform-example-foundation

Shows how the CFT modules can be composed to build a secure cloud foundation
https://cloud.google.com/architecture/security-foundations
Apache License 2.0
1.22k stars 714 forks source link

Add bootstrap IAM/services instructions on GCP bootstrap project (the one that is used to create the 2 prj-b-cicd and prj-b-seed projects) #1161

Closed obriensystems closed 5 months ago

obriensystems commented 6 months ago

TL;DR

add to https://github.com/CloudLandingZone/terraform-example-foundation?tab=readme-ov-file#0-bootstrap and the roles section https://github.com/terraform-google-modules/terraform-example-foundation/tree/master/0-bootstrap#prerequisites

add bootstrap project to https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/GLOSSARY.md

See service enablements on the project and IAM roles for the super admin or terraform service account being impersonated details in the list on https://github.com/terraform-google-modules/terraform-example-foundation/issues/1133#issuecomment-1980982638

Screenshot 2024-03-13 at 12 33 38

Terraform Resources

No response

Detailed design

No response

Additional information

No response

obriensystems commented 6 months ago

Testing IAM Role and services enablement on the bootstrap project see https://github.com/terraform-google-modules/terraform-example-foundation/issues/1133#issuecomment-2010026868

Continue olxyz org local and cloud shell terraform 1.3 run

Target bootstrap PR for the following

see scripting example in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/gh766-script/solutions/setup.sh

create folder then use script

Screenshot 2024-03-26 at 15 01 13

wait 2 min for propagation

Screenshot 2024-03-26 at 15 03 08

folder id in this case is 736660879367

IAM SA roles

Billing Account Administrator
Compute Shared VPC Admin
Folder Admin
Folder Creator
Organization Administrator
Organization Policy Administrator
Project Billing Manager
Project Creator
Project Deleter
Project IAM Admin
Security Center Admin
Security Center Notification Configurations Editor
Service Account Token Creator
Service Usage Admin

michael@cloudshell:~$ ROOT_FOLDER_ID=736660879367
michael@cloudshell:~$ BOOT_PROJECT_ID=tef-olxyz
#BILLING_ID=$(gcloud billing projects describe $BOOT_PROJECT_ID $BILLING_FORMAT | sed 's/.*\///')
michael@cloudshell:~$ gcloud projects create "$BOOT_PROJECT_ID" --name="${BOOT_PROJECT_ID}" --set-as-default --folder="$ROOT_FOLDER_ID"
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/tef-olxyz].
Waiting for [operations/cp.9134984346595650639] to finish...done.                                                                                                                         
Enabling service [cloudapis.googleapis.com] on project [tef-olxyz]...
Operation "operations/acat.p2-438381210056-0f410b88-37c8-4953-baf3-3af50f7d4db1" finished successfully.
Updated property [core/project] to [tef-olxyz].
michael@cloudshell:~ (tef-olxyz)$ BILLING_ID=01BCCE-4EC0EE-DC58C8

michael@cloudshell:~ (tef-olxyz)$ SUPER_ADMIN_EMAIL=$(gcloud config list --format json|jq .core.account | sed 's/"//g')
michael@cloudshell:~ (tef-olxyz)$ echo $SUPER_ADMIN_EMAIL
mich...bs.xyz
michael@cloudshell:~ (tef-olxyz)$ ORG_ID=$(gcloud projects get-ancestors $BOOT_PROJECT_ID --format='get(id)' | tail -1)
michael@cloudshell:~ (tef-olxyz)$ gcloud beta billing projects link "${BOOT_PROJECT_ID}" --billing-account "${BILLING_ID}"

gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/billing.admin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/accesscontextmanager.policyAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/billing.user --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.folderAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.organizationAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.projectCreator --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.projectDeleter --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/orgpolicy.policyAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/resourcemanager.projectIamAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/serviceusage.serviceUsageAdmin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/storage.admin --quiet > /dev/null 1>&1
gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/iam.serviceAccountTokenCreator --quiet > /dev/null 1>&1

gcloud organizations add-iam-policy-binding ${ORG_ID}  --member=user:$SUPER_ADMIN_EMAIL --role=roles/securitycenter.admin --quiet > /dev/null 1>&1

bootstrap project

https://github.com/terraform-google-modules/terraform-example-foundation/issues/1139 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1140 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1142 https://github.com/terraform-google-modules/terraform-example-foundation/issues/1143


# check services
michael@cloudshell:~ (tef-olxyz)$ gcloud services list | grep NAME
NAME: analyticshub.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigqueryconnection.googleapis.com
NAME: bigquerydatapolicy.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigqueryreservation.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: dataform.googleapis.com
NAME: dataplex.googleapis.com
NAME: datastore.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudresourcemanager.googleapis.com
Operation "operations/acat.p2-438381210056-f7e0fb71-9240-434b-918d-6e4bfe1fadfb" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudbilling.googleapis.com
Operation "operations/acat.p2-438381210056-9316b2a5-637a-40ad-bd64-981dc1af28f3" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable iam.googleapis.com
Operation "operations/acat.p2-438381210056-2344a0fc-dc11-4432-b9aa-53a89f0fbb4d" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudkms.googleapis.com
Operation "operations/acat.p2-438381210056-4379d575-1d69-40b6-9104-113ff4f48704" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable servicenetworking.googleapis.com
Operation "operations/acat.p2-438381210056-b21bf432-2188-49c1-a9b9-44ec824f8d43" finished successfully.

# existing
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable  accesscontextmanager.googleapis.com
Operation "operations/acat.p2-438381210056-66c8f99c-fe16-4481-8d64-e9d467f8fe8d" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable appengine.googleapis.com
Operation "operations/acat.p2-438381210056-4426e470-344b-4b1e-9d47-52a0f2befe21" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable artifactregistry.googleapis.com
Operation "operations/acat.p2-438381210056-0d5d0ccd-3845-4edf-a062-e3ff3448ff03" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable assuredworkloads.googleapis.com
Operation "operations/acat.p2-438381210056-f189974a-ab71-49e1-a81b-48bd896df8a0" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable billingbudgets.googleapis.com
Operation "operations/acat.p2-438381210056-7b38125e-23ec-4564-bbfc-277fc3297802" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudasset.googleapis.com
gcloud services enable cloudbuild.googleapis.comOperation "operations/acat.p2-438381210056-bafe06bf-3786-4608-b994-3c3069f6042e" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudbuild.googleapis.com
Operation "operations/acf.p2-438381210056-039eab93-9f6a-420b-943f-01297e27b339" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable cloudidentity.googleapis.com
Operation "operations/acat.p2-438381210056-7989271a-604a-4e7a-9c4e-4acd5e8ab8a5" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable containerregistry.googleapis.com
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable essentialcontacts.googleapis.com
Operation "operations/acat.p2-438381210056-8230082d-b801-4096-a74c-4977faf1a85c" finished successfully.
michael@cloudshell:~ (tef-olxyz)$ gcloud services enable securitycenter.googleapis.com
Operation "operations/acat.p2-438381210056-e1a24825-5697-465d-ad46-1496b47f1b58" finished successfully.

# post check
michael@cloudshell:~ (tef-olxyz)$ gcloud services list | grep NAME 
NAME: accesscontextmanager.googleapis.com
NAME: analyticshub.googleapis.com
NAME: appengine.googleapis.com
NAME: artifactregistry.googleapis.com
NAME: assuredworkloads.googleapis.com
NAME: bigquery.googleapis.com
NAME: bigqueryconnection.googleapis.com
NAME: bigquerydatapolicy.googleapis.com
NAME: bigquerymigration.googleapis.com
NAME: bigqueryreservation.googleapis.com
NAME: bigquerystorage.googleapis.com
NAME: billingbudgets.googleapis.com
NAME: cloudapis.googleapis.com
NAME: cloudasset.googleapis.com
NAME: cloudbilling.googleapis.com
NAME: cloudbuild.googleapis.com
NAME: cloudidentity.googleapis.com
NAME: cloudkms.googleapis.com
NAME: cloudresourcemanager.googleapis.com
NAME: cloudtrace.googleapis.com
NAME: containerregistry.googleapis.com
NAME: dataform.googleapis.com
NAME: dataplex.googleapis.com
NAME: datastore.googleapis.com
NAME: essentialcontacts.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com
NAME: logging.googleapis.com
NAME: monitoring.googleapis.com
NAME: pubsub.googleapis.com
NAME: securitycenter.googleapis.com
NAME: securitycentermanagement.googleapis.com
NAME: servicemanagement.googleapis.com
NAME: servicenetworking.googleapis.com
NAME: serviceusage.googleapis.com
NAME: sql-component.googleapis.com
NAME: storage-api.googleapis.com
NAME: storage-component.googleapis.com
NAME: storage.googleapis.com

michael@cloudshell:~ (tef-olxyz)$ gcloud services list | grep NAME | wc -l
39
fmichaelobrien commented 5 months ago

stale bot timer restart - https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/.github/workflows/stale.yml#L21

fmichaelobrien commented 5 months ago

see related https://github.com/terraform-google-modules/terraform-example-foundation/issues/1189

Note: with the service enablements and role additions (owner on the sa-terraform-bootstrap@seed account we are ok and get past 2-environments

michael@cloudshell:~/tef-olapp/github/gcp-org (tef-olapp)$ gcloud scc notifications describe "scc-notify" --organization=${ORGANIZATION_ID}
ERROR: (gcloud.scc.notifications.describe) INVALID_ARGUMENT: Security Command Center Legacy has been permanently disabled as of June 7, 2021. Migrate to Security Command Center's Standard tier or Premium tier to maintain access to Security Command Center. See https://cloud.google.com/security-command-center/docs/quickstart-security-command-center for more info.

https://github.com/terraform-google-modules/terraform-example-foundation/issues/1145

on a 2nd run after enabling roles/securitycenterAdmin on the super admin running the deployment see PR chang https://github.com/terraform-google-modules/terraform-example-foundation/pull/1175/files#diff-510bd7bb34f359f116d7029a41db09abb222834e57c39feb56e2acf6f74be8eeR86

although this sa is only used for 0-bootstrap, as sa-terraform-bootstrap@seed... is used for steps 1+

check SCC

michael@cloudshell:~/tef-olxyz/github/gcp-org (tef-olxyz)$ mv ./envs/shared/terraform.example.tfvars ./envs/shared/terraform.tfvars
michael@cloudshell:~/tef-olxyz/github/gcp-org (tef-olxyz)$ export ORGANIZATION_ID=$(terraform -chdir="../pbmm-on-gcp-onboarding/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
michael@cloudshell:~/tef-olxyz/github/gcp-org (tef-olxyz)$ gcloud scc notifications describe "scc-notify" --organization=${ORGANIZATION_ID}
ERROR: (gcloud.scc.notifications.describe) NOT_FOUND: Requested entity was not found.

Deployment of 1-org was ok

Step #2 - "tf plan": Plan: 288 to add, 0 to change, 0 to destroy.

Step #4 - "tf apply": Apply complete! Resources: 288 added, 0 changed, 0 destroyed.
Step #4 - "tf apply": 
Step #4 - "tf apply": Outputs:
Step #4 - "tf apply": 
Step #4 - "tf apply": base_net_hub_project_id = "prj-c-base-net-hub-5y8h"
Step #4 - "tf apply": billing_sink_names = {
Step #4 - "tf apply":   "prj" = "sk-c-logging-prj-billing-wh58"
Step #4 - "tf apply":   "pub" = "sk-c-logging-pub-billing-wh58"
Step #4 - "tf apply":   "sto" = "sk-c-logging-bkt-billing-wh58"
Step #4 - "tf apply": }
Step #4 - "tf apply": cai_monitoring_artifact_registry = "ar-cai-monitoring-4241"
Step #4 - "tf apply": cai_monitoring_asset_feed = "organizations/1064386348915/feeds/fd-cai-monitoring-4241"
Step #4 - "tf apply": cai_monitoring_bucket = "bkt-cai-monitoring-4241-sources-726972909649-us-central1"
Step #4 - "tf apply": cai_monitoring_topic = "top-cai-monitoring-4241-event"
Step #4 - "tf apply": common_folder_name = "folders/96486704059"
Step #4 - "tf apply": dns_hub_project_id = "prj-c-dns-hub-6f4b"
Step #4 - "tf apply": domains_to_allow = tolist([
Step #4 - "tf apply":   "obrienlabs.xyz",
Step #4 - "tf apply": ])
Step #4 - "tf apply": interconnect_project_id = "prj-c-interconnect-s2zg"
Step #4 - "tf apply": interconnect_project_number = "6710...54"
Step #4 - "tf apply": logs_export_project_linked_dataset_name = "projects/prj-c-logging-fn0h/locations/us-central1/buckets/AggregatedLogs/links/ds_c_prj_aggregated_logs_analytics"
Step #4 - "tf apply": logs_export_project_logbucket_name = "AggregatedLogs"
Step #4 - "tf apply": logs_export_pubsub_topic = "tp-org-logs-o9q2"
Step #4 - "tf apply": logs_export_storage_bucket_name = "bkt-prj-c-logging-fn0h-org-logs-o9q2"
Step #4 - "tf apply": network_folder_name = "folders/65...643"
Step #4 - "tf apply": org_audit_logs_project_id = "prj-c-logging-fn0h"
Step #4 - "tf apply": org_billing_logs_project_id = "prj-c-billing-logs-ve1w"
Step #4 - "tf apply": org_id = "10...5"
Step #4 - "tf apply": org_kms_project_id = "prj-c-kms-eeg3"
Step #4 - "tf apply": org_secrets_project_id = "prj-c-secrets-2lxo"
Step #4 - "tf apply": parent_resource_id = "73...67"
Step #4 - "tf apply": parent_resource_type = "folder"
Step #4 - "tf apply": restricted_net_hub_project_id = "prj-c-restricted-net-hub-a8d5"
Step #4 - "tf apply": restricted_net_hub_project_number = "12...62"
Step #4 - "tf apply": scc_notification_name = "scc-notify"
Step #4 - "tf apply": scc_notifications_project_id = "prj-c-scc-8zsj"
Step #4 - "tf apply": shared_vpc_projects = {
Step #4 - "tf apply":   "development" = {
Step #4 - "tf apply":     "base_shared_vpc_project_id" = "prj-d-shared-base-nlqs"
Step #4 - "tf apply":     "base_shared_vpc_project_number" = "10...505"
Step #4 - "tf apply":     "restricted_shared_vpc_project_id" = "prj-d-shared-restricted-j004"
Step #4 - "tf apply":     "restricted_shared_vpc_project_number" = "10..921"
Step #4 - "tf apply":   }
Step #4 - "tf apply":   "non-production" = {
Step #4 - "tf apply":     "base_shared_vpc_project_id" = "prj-n-shared-base-b12y"
Step #4 - "tf apply":     "base_shared_vpc_project_number" = "60..5"
Step #4 - "tf apply":     "restricted_shared_vpc_project_id" = "prj-n-shared-restricted-qnv6"
Step #4 - "tf apply":     "restricted_shared_vpc_project_number" = "85...26"
Step #4 - "tf apply":   }
Step #4 - "tf apply":   "production" = {
Step #4 - "tf apply":     "base_shared_vpc_project_id" = "prj-p-shared-base-oae0"
Step #4 - "tf apply":     "base_shared_vpc_project_number" = "17..1"
Step #4 - "tf apply":     "restricted_shared_vpc_project_id" = "prj-p-shared-restricted-2pqc"
Step #4 - "tf apply":     "restricted_shared_vpc_project_number" = "98..8"
Step #4 - "tf apply":   }
Step #4 - "tf apply": }
Step #4 - "tf apply": tags = {
Step #4 - "tf apply":   "environment_bootstrap" = "tagValues/281484537587812"
Step #4 - "tf apply":   "environment_development" = "tagValues/281483791828482"
Step #4 - "tf apply":   "environment_non-production" = "tagValues/281484388371311"
Step #4 - "tf apply":   "environment_production" = "tagValues/281483304603502"
Step #4 - "tf apply": }
Step #4 - "tf apply": policy-library/policies  doesn't match production; skipping
Step #4 - "tf apply": policy-library/lib  doesn't match production; skipping
Step #4 - "tf apply": policy-library/.git  doesn't match production; skipping
Finished Step #4 - "tf apply"
Screenshot 2024-04-11 at 11 26 08 AM
obriensystems commented 5 months ago

I'll raise in a separate PR but 3-networks-hub-and-spoke also needs the compute.orgSecurityPolicyAdmin or User or more recently compute.orgFirewallPolicyAdmin IAM role on the super admin to be able to view Hierarchical Firewall Policies created under the common branch

https://cloud.google.com/firewall/docs/firewall-policies#iam Compute Organization Firewall Policy Admin https://cloud.google.com/compute/docs/access/iam#compute.orgFirewallPolicyAdmin

As well as compute api on bootstrap project in step 11 of 3-networks-hub-and-spoke

michael@cloudshell:~/tef-olxyz/github/gcp-networks (tef-olxyz)$ gcloud services enable compute.googleapis.com
Operation "operations/acf.p2-438381210056-2415ed08-fad8-4333-8c8e-1017881efb60" finished successfully.

tested entire 3-networks-hub-and-spoke with these changes including the hierarchical firewall policy retry rename procedure

Screenshot 2024-04-13 at 19 34 33
fmichaelobrien commented 5 months ago

fixed by

PRs merged to the terraform-example-foundation upstream repo